Testing OTP and SMS Verification

An end to end test for phone verification means one run that starts at your signup form and finishes with a session token, using a real handset path for the SMS leg. Everything else (mocked providers, hardcoded 000000, a stubbed webhook) tests your own code but not the delivery chain that actually breaks in production.

The short version in six steps

1. Pick one target and stay narrow. Test one service, one country, one carrier route. Testing “SMS in general” produces results you cannot act on.
2. Get a number. Either a static test SIM you own, a provider sandbox number, or a rented number for a single activation.
3. Start the activation before you submit the form. The number needs to be live and listening when the sender fires, not five seconds after.
4. Submit the phone number in your app and record the timestamp of the request.
5. Poll for the inbound SMS until the code arrives or the activation window closes. Record the delta in seconds.
6. Feed the code back into your verify endpoint, assert on the session, then close the activation so the number is released.

Steps 3 and 6 are the ones teams skip. Skipping 3 gives you flaky tests that pass locally and fail in CI. Skipping 6 leaves activations open, which costs money and blocks the number for the next run.

Cheapest markets for smoke tests

Country	From	Services	Numbers in stock
USA	$0.01	200	79,399,375
Germany	$0.01	144	101,118,633
United Kingdom	$0.02	291	389,379,749
France	$0.02	153	131,996,906
Portugal	$0.02	143	140,112,396
Uzbekistan	$0.02	38	19,318,261
Italy	$0.03	149	412,533,735
Austria	$0.04	126	183,770,041

Three sources of test numbers compared

Source	Cost per run	Setup time	Real carrier path	Good for
Physical SIM in a drawer	Plan cost, plus a person to read the phone	Hours to days per number	Yes	Manual smoke tests, one or two countries
Provider sandbox or magic number	Free	Minutes	No, the SMS is never sent	Unit tests, CI runs of your own logic
Rented number, one activation	From $0.04, varies by service and country	Minutes, via API or app	Yes	Delivery checks, new countries, pre-launch

Sandbox numbers are correct for the 90 percent of your suite that asserts on state machines: rate limits, code expiry, retry counters, wrong-code lockout. They run in milliseconds and cost nothing. Use them everywhere you are testing your own branching.

Physical SIMs are correct when you need the same number to persist across weeks, for example an account you keep logged in as a regression fixture. The cost is human: someone has to hold the phone.

When a real rented number is the only option that works

Four cases where a mock cannot answer the question:

• You are launching in a country you have never sent to. You need to know whether your sender ID survives the route into India or the United States, and whether the body arrives intact or truncated.
• A service filters VoIP numbers. Your test number needs to be a real mobile number on a carrier the service accepts.
• You changed message templates. Some senders rewrite or block bodies that contain URLs, and only a real delivery shows you the final text.
• You need a second account for QA that is separate from your personal number, for privacy hygiene reasons rather than to get around anything.

MarioSMS rents real mobile numbers for one verification at a time, starting at $0.04 and varying by service and country, across 35+ countries and hundreds of services. The code shows up in the dashboard or app, usually within a minute. Every number carries a timer, and if no SMS arrives before it expires the activation cancels and the price returns to your balance automatically. A failed delivery then costs you nothing but the data.

What a passing test actually proves

Be precise about scope. A green run with a rented number proves five things and no more:

1. Your app accepted that number format for that country.
2. Your sender dispatched a message for that request.
3. The route delivered the body to a real handset inside the window you measured.
4. The code in the body matched the code your backend generated.
5. Your verify endpoint issued a session for that code.

It does not prove delivery for a different carrier in the same country, a different sender ID, or the same route four hours later during peak traffic. Delivery is probabilistic per route, so a single pass is one sample. Record the second count on every run, not just pass or fail. A test that passed in 8 seconds last week and 47 seconds today has told you something before it ever goes red, and that trend line is what you take to your provider when you open a ticket.

What happens between the Send code tap and the SMS arriving?

A verification test that only asserts “code appeared” covers five distinct systems at once. When it fails, you have no idea which one broke. Splitting the pipeline into components tells you where to put assertions and which timeouts belong to which hop.

What a number costs per test run

Service	Cheapest country	Price	Numbers in stock
Telegram	Canada	$0.51	422,891
Whatsapp	United Kingdom	$0.90	197,726
Google	Uzbekistan	$0.10	9,620
Discord	United Kingdom	$0.04	308,494
Uber	Portugal	$0.02	18,704
Amazon	Uzbekistan	$0.02	12,015

Source: MarioSMS catalog, 2026-09-10. Price per number, stock in brackets.

The request path from client to provider

The tap on Send code triggers a POST to your backend, usually /auth/phone/start with the number in E.164 format. Your backend does four things before any SMS exists.

Cost per service for a single verification

Service	Cheapest country	Price
Telegram	Canada	$0.51
Whatsapp	United Kingdom	$0.90
Google	Uzbekistan	$0.10
Discord	United Kingdom	$0.04
Instagram	Sweden	$0.06
TikTok	United Kingdom	$0.05
Facebook	United Kingdom	$0.06
OpenAI	United Kingdom	$0.06

1. Normalize and validate the number. A US number typed as (415) 555-0100 becomes +14155550100. Bad input dies here with a 400 and no SMS cost.
2. Check rate limits. Per number, per IP, per device fingerprint, per account. Most teams run at least three counters with different windows.
3. Generate and store the code (see below).
4. Call the SMS provider’s REST API with the number, message body and sender configuration, then return 200 to the client with a resend-after value.

The provider call is the first place where the response you get is not the outcome you care about. A 201 Created from the provider means the message was accepted into their queue, not that a handset received anything. Between accepted and delivered sit aggregators, carrier gateways and the handset itself. Any test that treats the provider’s 2xx as proof of delivery is testing your HTTP client.

How the code is generated, hashed and stored

Most implementations generate 4 to 8 digits from a cryptographically secure random source, not Math.random(). The record written to storage typically holds the number, a hash of the code, an issued-at timestamp, an expiry, an attempt counter and a status field.

Store the hash, not the code. Verification then recomputes the hash from user input and compares. This matters for your tests because it means you cannot read the expected code out of the database in an integration test, which is exactly why teams reach for a real number that receives the actual SMS. A rented number from MarioSMS delivers the code the way a user gets it, in the app or dashboard, usually within a minute, with the activation timer running.

The expiry window is a product decision, commonly 5 or 10 minutes. The attempt counter usually locks after 3 to 5 wrong entries. Both are testable without sending a single SMS if you expose a clock injection point.

Carrier routing, sender IDs and delivery receipts

Once the provider accepts the message, it picks a route. A route is a path through one or more aggregators to the destination carrier, and it carries a sender identity: a long code, a short code, an alphanumeric sender ID, or a 10DLC-registered number in the US. The sender type is not cosmetic. Some countries block alphanumeric IDs, some carriers deprioritize unregistered traffic, and some filter on message content that looks like a marketing template.

The provider then emits a delivery receipt (DLR) as a webhook to your callback URL. Common states:

DLR state	What it means	What your test should do
queued	Accepted, not yet sent to carrier	Keep waiting, do not assert failure
sent	Handed to the carrier	Start the delivery clock
delivered	Carrier confirmed handset receipt	Compare against your poll result
undelivered	Carrier rejected or dropped	Capture the error code, fail loudly
failed	Provider-side failure	Retry a different route, alert

DLRs are not universally honest. Some carriers report delivered on handoff to their internal store, so a delivered receipt with no code in the inbox is a real and common outcome worth a dedicated test case.

Where latency comes from and what a normal delivery window looks like

Latency stacks across hops rather than landing in one place.

Hop	Typical contribution
Client to your backend	Tens of milliseconds
Code generation and storage write	Single-digit milliseconds
Your backend to provider API	100 to 500 ms
Provider queue to aggregator	Variable, grows under load
Aggregator to carrier	Variable by route and country
Carrier to handset	Variable, worst at peak hours

The first three are yours and are effectively constant. Everything after the provider API call is outside your control and is where the variance lives. In practice, a healthy route puts the code on the handset in seconds, a congested one can take minutes, and a broken one never arrives. Set your test timeout from the number activation window, not from a guess, and log the elapsed seconds on every run so you have a distribution rather than a single anecdote.

Why the same code can arrive twice or out of order

SMS is a store-and-forward system with no ordering guarantee across separate messages. Three mechanisms produce duplicates and reordering.

Retries. An aggregator that gets no acknowledgment inside its window resends the same message. The handset receives two identical texts, sometimes 30 seconds apart, sometimes 6 minutes apart.

User resends. The user taps Resend before the first message lands, so two valid codes are in flight. If your backend invalidates the old code on resend, the first arrival is already dead, and a test that grabs the first SMS it sees will fail against a working system.

Route switching. A provider failing over between aggregators mid-flight can push the second attempt through a faster path, so message two beats message one to the phone.

Your polling logic needs a rule for which message wins. Sort by receipt time, take the newest, and record how many messages the number received during the activation. If a single-request test ever sees two, you have caught a retry loop worth investigating before it hits production traffic.

How do you design a test matrix for phone verification?

A test matrix converts “test the SMS flow” into a finite list of runs with predicted outcomes. Without one, teams retest the same happy path twenty times and ship a bug in the resend path. The matrix forces you to name every variable you control, pick values for each, and write down what should happen before you run anything.

Every service and country in the catalog is also an API call

The five dimensions worth varying

Five dimensions cover most real defects in phone verification:

Sharing numbers, access and budget in a team

Question	A workable answer
Who can spend?	One account, API keys per environment, a monthly cap
Who sees the codes?	The dashboard and the API; keep it out of shared chat
How do we avoid collisions?	One activation per test worker, released on teardown
How do we audit?	Activation history per key, exported monthly

1. Country and number type. A virtual phone number from the US behaves differently from one in India, both in delivery time and in the sender ID that appears. Number type matters too, since some services treat non-VoIP numbers and VoIP numbers differently at signup.
2. Input format. E.164 with a plus, national format with a leading zero, spaces, dashes, parentheses, and a country code typed twice. Each should either normalize to the same value or fail with a clear message.
3. Timing. Code entered at 5 seconds, at 60 seconds, one second before expiry, and one second after. Timing bugs hide in the gap between your stated expiry and the actual token TTL.
4. Attempt count. First code, resend, third resend, and the attempt after your rate limit should have kicked in.
5. Code correctness. Right code, wrong code, expired code, a code from a previous activation, and a code with whitespace pasted from a notification.

Multiplying all five gives hundreds of combinations. You do not run all of them. You pick a spanning set where each value of each dimension appears at least once, then add specific pairs you suspect interact, like expiry plus resend.

Happy path, edge path and abuse path cases

Sort every case into three buckets so priorities stay obvious.

Happy path cases prove the feature works. Valid number, code arrives, code entered inside the window, account created. Three or four cases here, one per major country you support.

Edge path cases prove the feature degrades correctly. No SMS arrives at all, code arrives after expiry, user changes the number mid-flow, app backgrounds for two minutes and returns, network drops between submit and response. These are where users actually get stuck.

Abuse path cases prove your limits hold. Same number requested six times in a minute, same IP requesting fifty different numbers, a code submitted 200 times with different digits. You test these against your own system with your own rented numbers, for QA purposes only, in line with the acceptable use rules.

A sample matrix table with expected outcomes

ID	Country	Input format	Timing	Attempt	Code	Expected result
H1	US	E.164	20s	1st	correct	Account created, session issued
H2	India	national, leading 0	30s	1st	correct	Normalized to E.164, account created
E1	US	E.164	never	1st	none	Activation cancels, balance refunded, UI offers retry
E2	US	E.164	expiry + 5s	1st	correct	Rejected with “code expired”, resend offered
E3	UK	E.164 with spaces	15s	2nd	1st code	Rejected, old code invalidated by resend
E4	India	E.164	45s	1st	correct + trailing space	Trimmed, accepted
A1	US	E.164	n/a	6th in 60s	n/a	Send blocked, cooldown message with seconds left
A2	US	E.164	10s	1st	10 wrong tries	Attempt lock, code burned, new request required

Every row states one expected result in observable terms. “Works fine” is not an expected result. “Session issued and 201 returned” is.

How many cases to run per release

Split by cadence rather than running everything every time.

Trigger	Case count	Real SMS needed	Rough cost
Every pull request	3 to 5	0	$0
Nightly	10 to 15	3 to 5	under $0.50
Pre-release	25 to 35	8 to 12	around $1
After auth or provider change	full matrix	15 to 20	a few dollars

At $0.04 per number as a starting price, a full pre-release pass costs less than a coffee, and unused activations refund automatically when no SMS arrives.

Which cases deserve a real SMS and which can be stubbed

Stub anything that tests your own code in isolation. Format normalization, TTL arithmetic, rate limit counters, and attempt locks all run against a fake provider in milliseconds. Those are unit tests wearing a matrix row.

Rent a real number when the case depends on something outside your process: actual delivery through a carrier, sender ID rendering, message body parsing by your regex, autofill behavior on iOS and Android, and the timing of a real number activation. Row E1 in particular needs a real number, since the only honest way to test “no SMS arrives” is to request a number and let the timer run out.

How do you get a real mobile number for a single test run?

Renting a number for one test is a five step loop that takes about two minutes end to end. The steps below assume MarioSMS, where a number is rented for one verification at a time and costs start at $0.04 depending on the service and country.

A test matrix that catches the usual failures

Case	Setup	What passes
Happy path	Fresh number, first attempt	Code arrives, account is created
Slow SMS	Poll for the whole activation window	The screen waits instead of erroring at 30 seconds
No SMS at all	Let the window expire	Retry offered, no charge, no orphaned account
Wrong code	Enter four wrong digits	Lockout message, attempts counted per number
Resend	Request a second code on the same number	Rate limit respected, both codes accepted or the newest wins
Reused number	A number that already signed up	Your duplicate handling, not a crash
Country mismatch	Number from another market	The message the user actually needs

Run each case in every launch market, not only in your own.

Step 1, pick the service and country

Open the web app at app.mariosms.com, or the iOS or Android app, and choose two things: the service you are testing against and the country the number should belong to. Stock covers 35+ countries and hundreds of services, so most test matrices can be filled from one account.

Privacy and policy limits for test numbers

Rule	What it means in practice
Test your own product	Rented numbers are for your flows and your QA accounts
No impersonation	Never verify an account in someone else’s name
No ban evasion	A blocked account stays blocked
Data hygiene	Do not keep received codes or numbers in test fixtures

The acceptable use policy is the authority here: /acceptable-use/.

Pick the country from the matrix row you are executing, not from habit. A United States number and an India number behave differently in your own code: different digit counts, different formatting when your library renders them, different sender IDs in the message body. If your row says “IN, first attempt, autofill on Android”, rent an Indian number, not a US one that happens to be cheaper.

The price shown before you confirm is the price you pay. Write it into the test run log next to the row ID so budget questions later have real numbers instead of estimates.

Step 2, rent the number and note the activation window

Confirm the rental. The number appears with a timer next to it, the activation window. That timer is how long the number stays yours and stays listening for an SMS from that service.

Note the start time. Two clocks are now running, the activation window and whatever OTP expiry your own app enforces, and most confusing test results come from those two clocks disagreeing. If your app expires codes in 5 minutes and you spend 3 minutes copying the number into a form by hand, you are testing your patience, not your flow.

Have the app under test already open on the phone number screen before you rent. The rental should be the last thing you do before pasting.

Step 3, paste the number into your app under test

Copy the number from the dashboard and paste it into the phone field. Two details worth checking at this point, because they catch real bugs before any SMS is involved:

1. Does your field accept the number in the format the dashboard gives it, with the country code and no spaces?
2. Does your normalization produce the same E.164 string the SMS provider will see?

Then trigger Send code and start a stopwatch, or log the timestamp if the run is scripted.

Step 4, read the code in the dashboard or mobile app

The code appears in the dashboard or the mobile app, usually within a minute. Refresh or watch the activation row, depending on which client you are using. For scripted runs, the REST API returns the same activation state your dashboard shows, so a test can poll instead of a human watching a screen.

Copy the code into your app and finish the flow. Record three things per run:

Field	Example value	Why it matters
Time to code	14s	Baseline for your delivery SLO
Full message body	“Your code is 481920. Do not share.”	Feeds your regex and autofill tests
Sender ID as displayed	Short code or alphanumeric	Changes autofill behavior on iOS

Your parsing regex, your iOS autofill domain hint, and your support docs all depend on the exact text of the message, not just the six digits.

Step 5, let the timer expire if nothing arrives

Sometimes no SMS arrives. That is the E1 row from your matrix, and it is a legitimate test result, not a failed test setup. Leave the activation alone and let the timer run out. When the window closes with no SMS, the activation cancels and the price returns to your balance automatically.

While the timer runs, watch what your own app does. Does the resend button become tappable at the interval you designed? Does the screen still show a spinner after 90 seconds? Does anything tell the user what to do next? Those answers are the actual value of the run.

What the refund behaviour means for test budgets

Automatic refunds on no-delivery change how you budget a suite. You pay for codes that arrive, not for attempts.

A 40 row matrix where 32 rows deliver at an average of $0.06 costs about $1.92, and the 8 rows that were designed to produce nothing cost $0. Estimate spend from expected successful deliveries, then add a small margin for retries after a genuine flake. Top up by card or crypto once per sprint rather than per run, and keep the whole thing inside acceptable use: QA and OTP testing against services you own or are authorized to test.

How do you wire a rented number into an automated test?

An OTP test has two halves that run in different systems. Your test client drives the app or API under test, and a separate client talks to the number provider. The test only passes when both halves agree: the app accepted a code that arrived on a number the test rented seconds earlier. Everything below is about keeping those two halves in sync without sleeps scattered through the suite.

The activation window is the timeout your test should respect

The request, poll, extract, assert loop

The shape is the same in every language. Five steps, in order:

Pre-launch checklist

Check	Done when
Every launch market tested with a real number	Codes arrive and accounts are created
Expiry path tested	Window runs out, retry offered, nothing charged
Retry and lockout tested	Limits behave and the copy is clear
Fallback tested	Voice or email path works if you offer one
Monitoring live	Entry rate and latency per country on a dashboard
Budget capped	A daily limit on test spend

1. Request an activation for the target service and country. You get back an activation ID and a phone number in E.164.
2. Submit that number to the app under test and trigger the send-code action.
3. Poll the activation status until a message body appears or the timer expires.
4. Extract the code from the message text with a service-specific pattern.
5. Submit the code, assert on the resulting session or error, then close the activation.

Steps 1, 3 and 5 are provider calls. Steps 2 and 4 are yours. Keep them in separate modules so a provider change touches one file. Working versions of the polling half live in the Python and Node.js guides, with the same loop in Go and PHP.

Order matters at step 2. Request the number first, then trigger the send. If you trigger a send before the number activation exists, the SMS has nowhere to land and the test fails for a reason that has nothing to do with your code.

Choosing a polling interval that does not waste quota

Codes on MarioSMS usually appear within a minute, so the useful polling window is 60 to 180 seconds depending on the service. A 1 second interval over 120 seconds is 120 requests per test. Across 40 matrix rows that is 4,800 requests to detect roughly 32 messages.

Back off instead. A workable schedule:

Elapsed	Interval	Requests in window
0-10s	no polling	0
10-40s	3s	10
40-120s	5s	16
120s to timer end	10s	6 or fewer

That is about 32 requests per test instead of 120, and the worst case adds 5 seconds of latency to a passing test. The first 10 seconds are dead time in practice, since the message has to clear the sender, the aggregator and the carrier before it can be readable.

Add jitter of 200 to 500 ms to each sleep when tests run in parallel. Twenty workers polling on an exact 5 second grid produce twenty simultaneous requests every 5 seconds, which looks like a burst to any rate limiter.

Parsing the code out of message text safely

Never take the first digit run in the body. Real messages contain other numbers: short codes, years, support line digits, an unsubscribe reference. A greedy \d+ match on “Your code is 481920, valid for 10 minutes. Reply STOP to 40404” can return 10 or 40404.

Anchor on the length and the context instead:

• Match a fixed-length group: \b(\d{6})\b for a 6 digit service, \b(\d{4})\b for a 4 digit one. Do not write one pattern for both.
• Prefer a pattern with a lead-in phrase when you know the template, then fall back to the bare length match if the phrase misses.
• Store the pattern per service in the same fixture that holds the code length, so a template change is a one line edit.
• Log the full body on a parse miss. A test that fails with “no match” and no body is unusable at 2am.
• Reject a match when two different candidates of the correct length appear. Fail loudly rather than submitting the wrong one.

Alphanumeric codes exist too. If a service can send A4K9QZ, your pattern needs a character class, not \d. Check the OTP glossary entry for the formats you are likely to meet.

Handling the no-SMS case without a hanging test

Every activation carries a timer. If no SMS arrives before it expires, the activation cancels and the price returns to your balance automatically, so the cost of a silent test is $0. Your test still needs its own deadline.

Set the test timeout below the activation timer, not above it. If the timer runs 20 minutes, cap the poll loop at 180 seconds and exit. Waiting for provider-side expiry turns a 3 minute suite into a 20 minute one.

On timeout, do three things: cancel the activation explicitly, record the country, service and elapsed seconds, and mark the result as no_delivery rather than failed. A missing SMS is a delivery signal, not a broken assertion. Suites that collapse both into red lose the ability to spot a country going quiet.

Cleaning up state between runs

Teardown runs in a finally block, always, including when an assertion throws mid-loop. Close or cancel the activation so nothing stays open against your balance.

The account side needs cleanup too. A number rented for one verification is single-use, so the account you created in run 1 is orphaned when run 2 rents a different number. Either delete the test account through your own API at teardown, or tag test accounts with a run ID and sweep them on a schedule. Store the number, activation ID and account ID together in the run artifact so a sweep can find both ends.

Never cache a number for reuse across runs. The rental covers one verification, and a stale number in a fixture produces a test that fails against the wrong hypothesis for a week before anyone checks.

Which code samples do you need for Python, Node.js, Go and PHP?

Four languages, one contract. If every client in your organization exposes the same three methods, a tester who wrote the Python suite can read the Go suite without opening the API docs.

The code arrives in the app and over the API at the same moment

A shared helper interface for all four languages

Define the interface once, then implement it per language. Three methods cover the full rental lifecycle:

One activation: number, window, code, release

Method	Input	Returns	Failure behavior
rent(service, country)	service slug, ISO country code	activation ID, phone number in E.164	raises on out of stock, no charge
wait_for_code(activation_id, timeout)	activation ID, seconds	the OTP string	raises on timeout, activation cancels and refunds
release(activation_id)	activation ID	nothing	idempotent, safe to call twice

Two rules keep the implementations honest. rent never returns a number without also returning the activation ID, because a number with no ID is unreleasable. wait_for_code polls with a deadline supplied by the caller, not a constant baked into the helper, so a slow India route and a fast US route can use the same code with different budgets.

The refund behavior shapes the error handling. A number activation that receives no SMS inside its window cancels on its own and the price returns to your balance, so a timeout in wait_for_code costs nothing. That means aggressive timeouts are cheap. Set 90 seconds, not 10 minutes.

Python with requests and pytest fixtures

Use a requests.Session with a mounted HTTPAdapter so connection reuse and retry policy live in one place. Wrap the helper in a function-scoped pytest fixture that yields the number and releases in the teardown block, so a failing assertion still returns the activation.

@pytest.fixture
def rented_number(sms_client):
    act = sms_client.rent(service="telegram", country="us")
    try:
        yield act
    finally:
        sms_client.release(act.id)

Poll with time.monotonic() rather than time.time(), since a clock adjustment mid-run will otherwise extend or truncate the window. Full request and response shapes are in the Python guide.

Node.js with fetch and Playwright

Node 18 and later ship fetch and AbortSignal.timeout(), so no HTTP library is required. Pass signal: AbortSignal.timeout(10_000) on each poll request so a hung socket fails in 10 seconds instead of hanging until the suite times out.

In Playwright, put the rental in a worker-scoped or test-scoped fixture and register cleanup with the fixture’s teardown, not test.afterEach. Playwright fixtures tear down in reverse order, so a rental created before the browser context is released after the context closes, which is what you want when the app is still holding the session. The Node.js guide shows the poll loop with setTimeout promises rather than a busy while loop.

Go with context deadlines

Go makes the timeout contract explicit. Every helper method takes ctx context.Context as the first argument and passes it into http.NewRequestWithContext. The caller sets the budget:

ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
defer cancel()
code, err := client.WaitForCode(ctx, act.ID)

Use time.NewTicker(3 * time.Second) for the poll interval and select on both ticker.C and ctx.Done(). Release with a fresh context in a defer, because the parent context is already dead by the time the deferred call runs. That single mistake causes silent release failures in most first drafts.

PHP with Guzzle and PHPUnit

Guzzle’s handler stack takes middleware, so put retry logic there once instead of at every call site. Set timeout and connect_timeout separately in the client config, since a DNS stall and a slow response need different limits.

In PHPUnit, avoid setUp for rentals. Rent inside the test method and release in a finally block, or use a data provider that supplies country codes so one test body covers United States and other markets without duplication.

Retry, backoff and error typing in each language

Three error classes, and only one is retryable:

1. Transient (HTTP 429, 502, 503, connection reset). Retry with exponential backoff starting at 1 second, cap at 8 seconds, maximum 4 attempts.
2. Terminal (out of stock for that service and country, insufficient balance, bad API key). Fail immediately and print the service and country in the message.
3. Empty poll (no SMS yet). Not an error. Keep polling until the caller’s deadline.

The common bug is treating an empty poll as transient and feeding it into the backoff loop. Backoff pushes the poll interval to 8 seconds, and a code that arrives at second 41 gets read at second 48, after the assertion already ran. Poll on a fixed 3 second cadence and reserve backoff for HTTP failures only.

How does testing differ service by service?

A test suite that passes against one service and fails against four others usually has a hardcoded assumption in it: the code is six digits, the SMS arrives in 20 seconds, the message body is one line. Each of those is true somewhere and false somewhere else. Below are the behaviours that break naive tests, grouped by category.

Markets with the most numbers in stock for load testing

Country	Numbers in stock	Services	From
United Kingdom	37,001,905	189	$0.02
Italy	24,353,864	118	$0.03
Austria	12,315,289	105	$0.04
Canada	11,766,481	37	$0.05
Indonesia	8,380,976	45	$0.04
USA	2,954,215	139	$0.01
Australia	2,255,940	109	$0.04
Germany	2,147,090	123	$0.01

Source: MarioSMS catalog, 2026-09-10. Price per number, stock in brackets.

Messaging apps and their in-app code formats

Messaging apps send the code over SMS on first registration, but many of them also send the same code through their own in-app channel if the account already exists on another device. Your test account will not have that device, so plan for the SMS path only and expect the app to fall back to it after a delay.

Pick the market you are testing, with stock and price on the row

Two parsing problems show up here. First, the code often sits inside a longer sentence with a trailing security warning, so a naive “first digit run in the body” regex can match a support phone number instead. Anchor the regex to the label used by that specific sender. Second, some messaging apps include a hidden app-hash suffix for Android SMS Retriever, which adds an 11 character token after the code. Strip it before comparing.

Resend on messaging apps is typically gated for a minute or more after the first send, and the gate is per phone number, not per session. A retry loop that taps Resend three times just extends its own wait.

Social and marketplace signup flows

Social and marketplace signups usually put phone verification behind an email step, a captcha, or a profile form, which means the SMS is not the first thing your test does. Budget the setup time separately from the SMS verification assertion, or the timeout you set for the whole flow will be spent before the code request even fires.

Marketplaces add a second wrinkle. Many of them ask for verification again at the first listing or first message, not at signup. If your test only covers registration, the second challenge is untested. Write it as a separate test with its own number rather than reusing the registration number, since the activation from the first run has already closed.

Ride hailing and delivery apps with short windows

Ride hailing and delivery apps run the tightest code expiry in common use, often 60 to 120 seconds, and several of them auto-submit the code the moment the field fills. That combination punishes slow polling. If your poller sleeps 5 seconds between checks and the driver-side app expires the code in 60, you have burned 8 percent of the window on one sleep.

Auto-submit also breaks the classic Selenium pattern of typing into the field and then clicking Submit, because the click lands after navigation. Type the code, then wait for either the success element or the Submit button, whichever appears.

Financial and wallet apps with stricter checks

Financial and wallet apps apply the strictest number checks. They tend to reject known VoIP numbers, require the number country to match the account country, and lock the account after three wrong codes instead of five. Some also bind the code to the device fingerprint, so a code pulled on one machine and entered on another fails with a generic error that looks like a wrong code.

Test these with a non-VoIP number from the same country as the account profile, and treat lockout as a first-class expected outcome with its own test rather than an accident you discover in CI.

Services that prefer a voice call fallback

Some services switch to a voice call after one or two failed SMS attempts, and a few default to voice for certain countries entirely. A test that only reads SMS sees an empty inbox and reports a delivery failure that is actually a channel switch.

Handle it in two ways. Detect the fallback in the UI (the button text changes to “Call me”) and assert on it as a valid branch. And when a service is known to prefer voice in a given country, drop that combination from the SMS matrix instead of letting it fail nightly. On MarioSMS, if no SMS arrives inside the activation window the activation cancels and the price returns to your balance automatically, so a voice-preferring service costs you time rather than money.

A service comparison table for code length and window

Code length, expiry and resend behaviour vary by service and change over time. Measure them in your own environment rather than trusting a static list, and record what you find in a table like this one.

Category	Typical code length	Expiry to verify	Resend gate	Test note
Messaging apps	6 digits	Minutes	60s or more	Strip app-hash suffix
Social networks	5 to 6 digits	Minutes	30 to 60s	Captcha before send
Marketplaces	4 to 6 digits	Minutes	60s	Second challenge at first listing
Ride hailing / delivery	4 to 6 digits	60 to 120s	30s	Auto-submit on fill
Finance / wallets	6 to 8 digits	Short	60s or more	Country match, 3 attempt lockout
Developer tools	6 digits	Minutes	30s	Often TOTP as well as SMS

Fill the expiry column with measured seconds from your own runs. Two numbers per row are enough to size the poll deadline: the shortest observed expiry and the longest observed delivery time. If the gap between them is under 10 seconds, that service needs a faster poll cadence than the rest of your suite.

How do country and carrier differences change the test plan?

A verification flow is not one system. It is your code, a messaging provider, an international route, a destination carrier and a handset locale, and four of those five change when you swap the country code. A build that delivers a code in 4 seconds to a US number can time out at 90 seconds on the same provider to an Indian one, with no error on your side.

Number formats, E.164 and local dialing quirks

Store every number in E.164 (plus sign, country code, subscriber number, no spaces or dashes) and format only at the display layer. The failures come from the gap between what a user types and what E.164 needs.

One balance, one place to cap what a test suite can spend

1. Leading trunk zeros. UK, Germany and Italy users type 07911 123456. E.164 wants +447911123456 for the UK, but Italian mobile numbers keep the leading digit, so a blanket strip-the-zero rule corrupts them.
2. Variable national length. Some countries have 8 to 11 digit mobile numbers depending on the carrier block, so a fixed length === 10 check rejects valid numbers.
3. Country code collisions. +1 covers the US, Canada and 20 or so Caribbean territories. A “US only” rule written as startsWith('+1') lets all of them through.
4. Paste artifacts. Non-breaking spaces, en dashes and Unicode digits arrive from password managers and contact lists. Normalize before you validate.

Test each rented number in three input shapes: full E.164, national format with the country picker set, and national format with the picker wrong. The third one should fail with a readable message, not a 500.

Sender ID rules and alphanumeric senders

The From field is regulated differently in every market, and your assertions probably depend on it.

Market pattern	What arrives	Test impact
Alphanumeric allowed, unregistered	Your brand name	Reply-to-stop tests are meaningless, the sender is one-way
Alphanumeric with pre-registration	Registered ID, or the message is dropped	An unregistered ID in staging fails silently
Alphanumeric blocked	A long or short numeric code	Any assertion on sender text breaks
Dynamic route selection	A different number per send	Never assert on an exact sender value

Parse the code out of the body, never out of the sender. Write the regex against message text with a bounded pattern (\b\d{6}\b) and assert the sender only as “present and non-empty”.

Carrier filtering and content that gets blocked

Destination carriers run spam filters on message content, and those filters are stricter in some markets than others. Content that commonly triggers filtering includes shortened links, all-caps words, currency symbols, more than one URL, and unregistered sender identities. The result is usually a silent drop: the provider reports accepted, the handset never rings.

Build one test that sends your exact production template and one that sends a deliberately filter-prone variant. If both arrive in country A and only the first arrives in country B, you have found a template problem, not an infrastructure problem. See /glossary/sms-verification/ for the message-shape basics.

Choosing five countries that cover most of your risk

Five is enough if you pick along different axes rather than by market size.

1. Your largest user base, whichever country that is.
2. One country with a 10-digit national format and heavy carrier filtering, such as India (/numbers/india/).
3. One +1 country other than the US, to catch startsWith('+1') bugs.
4. One EU country with a leading trunk zero, to exercise your normalizer.
5. One country where your provider uses a route you rarely touch, so slow delivery shows up before a user reports it.

The US (/numbers/united-states/) usually lands in slot one or three. Rerun the full set before each release and the single largest country on every merge.

Prices and availability as a planning input

MarioSMS prices start at $0.04 per number and vary by service and country, with 35+ countries and hundreds of services in stock. Two planning consequences. First, price ranking tells you which countries are cheap to hammer and which to sample, so put your high-volume regression on the cheap end and your five-country matrix on a fixed weekly cadence. Second, stock is per service and per country, so a suite that hard-codes one country fails when that combination is out. Read availability at run time and fall back to the next country in your list.

Each number handles one verification, has a timer, and if no SMS arrives the activation cancels and the price returns to your balance automatically. Failed delivery tests therefore cost nothing, which makes negative testing in slow markets affordable.

Time zone effects on delivery and on your test schedule

Carrier queues are busiest during local waking hours, and some markets throttle promotional and transactional traffic differently at peak. A 09:00 UTC nightly run hits India mid-afternoon and the US west coast at 02:00, so your latency numbers describe two different network conditions and average into something meaningless.

Record the local hour at the destination alongside every measurement. Then schedule two runs per country, one at local 10:00 and one at local 22:00, and compare the p95 delivery time between them. A gap over 15 seconds means your production timeout needs to be sized against the busy hour, not the quiet one.

Which failure modes should every OTP suite cover?

A verification flow has maybe six moving parts, and each one fails in a way that looks fine from the outside. The login succeeds, the dashboard loads, the test goes green. The failures below are the ones that survive a happy-path suite, so each needs a test written against the failure rather than the success.

Expired code accepted by the backend

Most teams set a TTL (5 or 10 minutes is typical) in config and never verify the check runs. The code lands in a table with an expires_at column, and if the query filters on code = ? and user_id = ? but forgets the timestamp, an old code works forever.

Write a test that requests a code, freezes or advances the clock past the TTL, then submits the code. Assert HTTP 400 or 401 and an error body of code_expired. Also assert the row is either deleted or flagged so a second call cannot succeed either. If your test framework cannot control server time, insert the code row directly with an expires_at in the past and submit it through the public endpoint.

Code reuse after successful login

A code should be single use. The common bug is a verify handler that returns a session without marking the code consumed, so the same six digits work three times in the next four minutes. That matters when the SMS sits in a notification history or a shared inbox.

Assert twice in the same test. First call returns 200 with a session token. Second call with identical input returns 400 and no token. Then assert the second call did not create a second session row, since some implementations reject the code but still issue a token from a cached path.

Brute force with no attempt cap

Six digits is a million combinations, which sounds safe until you count what an attacker actually needs. With a 10 minute window and no cap, a script at 50 requests per second covers the full space long before expiry. The fix is an attempt counter per code, usually 3 to 5 tries, plus a per-phone rate limit.

Loop your test to the configured cap with wrong codes, then submit the correct code and assert it is rejected. Locking out after 5 bad guesses is useless if the right code still works on attempt 6.

Race between resend and verify

The user taps resend at second 29, the first SMS arrives at second 30, and they type the first code. Whether that works depends on whether resend invalidates the old code or both stay live. Either policy is defensible, but the test must pin which one you chose.

Fire the resend request and the verify request for code one concurrently from two threads. Assert exactly one outcome. If old codes stay valid, assert both succeed on their own code and that a third code was never issued. If resend invalidates, assert code one returns code_superseded. Related: number activation windows on a rented number end independently of your app TTL, so a test that runs long enough to hit both needs to distinguish the two failures.

Enumeration through different error messages

Registration returns “number already in use” and login returns “no account found.” Two different strings, and now anyone with a list of numbers can sort them into customers and non-customers. Response timing leaks the same thing when the registered path runs a password hash and the unregistered path returns immediately.

Assert equal status codes and byte-identical bodies across a registered and an unregistered number. For timing, run 50 requests per branch and assert the median difference stays under 50 ms. Test with a fresh rented number for the unregistered side so the state is genuinely clean, and check United States and one non-US country since some backends branch on country code.

SIM swap and number recycling assumptions

Carriers recycle numbers after a disconnection period, and SIM swaps move a number to a new device in minutes. If your app treats a phone number as a permanent identity, both events hand an account to a stranger.

Test the recovery path rather than the signup path. Assert that a verified number alone cannot reset a password without a second factor, and that a re-verification event on an existing account writes an audit row. Assert that changing the number on file invalidates active sessions and sends a notice to the old contact method.

Silent provider failure with a 200 response

An SMS gateway accepts a message, returns 200 with a message ID, and drops it at the carrier. Your send endpoint reports success and your test passes because it only checked the API response. The gap shows up as support tickets.

Never assert on the send response alone. Assert on a code actually read from a real inbox, which is what a rented number gives you. Then assert your delivery-receipt handler records a terminal status within 120 seconds and that a missing receipt raises rather than being written as delivered.

A failure mode to assertion table

Failure mode	Trigger in test	Assertion that catches it
Expired code accepted	Advance clock past TTL	400 code_expired, row consumed
Code reuse	Verify twice with same code	Second call 400, no second session
No attempt cap	N wrong codes, then correct one	Correct code rejected after cap
Resend race	Concurrent resend and verify	Exactly one documented outcome
Enumeration	Registered vs unregistered number	Identical body, median delta under 50 ms
SIM swap	Re-verify on existing account	Audit row written, sessions invalidated
Silent provider failure	Send, then read real inbox	Code read from SMS, receipt within 120 s

Each row costs one rented number at $0.04 and up, and an activation that receives nothing refunds automatically, so a suite that runs all seven stays under a dollar per full pass.

How do you debug an SMS that never arrives?

A missing code has five plausible causes and they sit at different hops: the client never sent, the provider rejected, the carrier dropped, the destination range is bad, or the number itself is dead. Triage in order, and collect one piece of evidence at each hop before moving on. Guessing costs more than checking.

Step 1, confirm the request left the client

Before blaming anything downstream, prove the app made the call. Open the network log for the test run and find the POST to your send endpoint. You need three facts: the request fired, it carried the phone number in the format you expected, and it returned a 2xx.

The common failures here are boring. A client-side validation rule silently swallowed the submit. The number was stored as 07700 900123 and sent without the country code. A retry guard from an earlier test still held the button disabled for 60 seconds. A stale auth token turned the send into a 401 that the UI rendered as a generic spinner.

Evidence to keep: request timestamp in UTC, the exact E.164 string sent, HTTP status, response body, and the client build number.

Step 2, check the provider response and message ID

A 2xx from your own backend does not mean the provider accepted the message. Look at the provider’s response payload for your send. Two outcomes matter.

If a message ID came back, the provider queued it. Record it. That ID is the only key that ties your logs to the carrier’s, and support conversations go nowhere without it.

If an error came back instead, read the code rather than the human-readable text. Providers use distinct codes for unroutable prefix, blocked destination country, sender ID not registered for that country, insufficient credit, and rate limit. Each points to a different fix, and the message string is often identical across three of them.

Step 3, read the delivery receipt status

The receipt (DLR) is the carrier’s answer. Statuses usually resolve within 5 to 30 seconds, and a receipt still stuck at queued after 120 seconds is itself a finding.

Status	What it means	Next action
queued / accepted	Provider holds it, carrier has not confirmed	Wait 120 s, then treat as a route problem
sent	Handed to the carrier, no confirmation back	Check whether that route returns receipts at all
delivered	Carrier confirmed handset delivery	Problem is the reader or the inbox, not the send
undelivered	Carrier accepted, then dropped it	Filtering, sender ID, or content
failed	Rejected outright	Bad number, barred range, or dead SIM

A delivered receipt with no code visible means your polling loop, parser, or timing window is wrong, not the SMS pipeline. Go read the raw inbox body yourself before filing anything against delivery.

Step 4, test a second country or service

One failing route tells you nothing about scope. Send the same payload to a second destination and compare. If a US number receives in 20 seconds and India times out, you have a route or filtering issue on that corridor, not a broken template. If both fail, the problem is upstream in your own code or your provider account.

Run the same split by service. If your app’s own code arrives but a third-party service’s code does not, the service is filtering or throttling, and the fix is on their side of the fence.

Step 5, switch number and retry once

Individual numbers go bad. A SIM gets recycled, a range gets blocked by one specific sender, an operator drops A2P traffic to a block. Rent a fresh number and repeat the exact same send once.

With MarioSMS each number is rented for one verification at a time from $0.04, and if no SMS arrives inside the activation window the activation cancels and the price returns to your balance automatically, so a retry that also fails costs nothing. Two consecutive failures on two separate numbers is strong evidence for a route or service problem. One failure then one success means the first number was the fault, and you log it and move on.

Retry once, not five times. Repeated sends to the same destination inside a minute trip anti-flood rules and turn a clean signal into noise.

Logging fields that make the next incident faster

Log these at send time, not after the incident:

1. request_id from your own backend, propagated to the client.
2. provider_message_id exactly as returned.
3. msisdn_e164, hashed or masked to the last 4 digits.
4. country_iso2 and service_key.
5. sent_at_utc with millisecond precision.
6. dlr_status plus dlr_received_at_utc, updated in place.
7. activation_id from the rental provider, so the number and the send join on one row.
8. attempt_number and the ID of the previous attempt it replaces.

With those eight fields, the median SMS incident resolves by reading one row instead of correlating four systems by timestamp. The Python and Node.js examples both return the activation ID on rent, so field 7 costs one variable assignment.

How do you load test OTP flows without burning budget?

A load test that rents a real number per virtual user is a bill, not a test. At $0.04 per number, 5,000 simulated signups cost $200 per run, and if the suite runs on every merge to main you pay that four times a day. The fix is to test two different things separately, because they fail for different reasons.

The stages an OTP passes through

Stage	Owned by	Typical time	What breaks here
Request accepted	Your backend	Under 200 ms	Rate limits, validation, duplicate submits
Message queued	Your SMS provider	Under 1 s	Route selection, sender ID rejected
Carrier accepted	The operator	1 to 10 s	Filtering, blocked sender, gray route
Handset delivered	The network	2 s to minutes	Roaming, congestion, powered-off device
Code entered	The user	Seconds to minutes	Expiry, retries, autofill

Your logs usually stop at stage two, which is why real-number tests find what monitoring misses.

Split the load test from the delivery test

Throughput testing answers whether your API survives 500 send requests per second: connection pools, database write contention, queue depth, the rate limiter itself. Delivery testing answers whether a code sent to a real handset in India arrives and matches. The first needs volume and no carriers. The second needs real numbers and almost no volume.

Test type	Volume per run	Real numbers	Cost driver	Runs on
Throughput	1,000 to 100,000 requests	No	Compute only	Every merge
Rate limit	50 to 500 requests	No	Compute only	Every merge
Delivery sample	5 to 30 activations	Yes	Per activation	Nightly
Release gate	20 to 60 activations	Yes	Per activation	Before ship

Mocking the provider at the boundary

Put the seam at your own SMS client interface, not at the HTTP library. One interface with a send(to, body) method and one fetchCode(activationId) method gives you a fake that returns a fixed code in 0 ms and a real implementation that talks to the provider. Under load, the fake should still model latency and failure: sleep 200 to 900 ms with jitter, return a 429 on 2 percent of calls, return a 500 on 0.5 percent. A mock that always returns 200 in 1 ms hides the timeouts and retry storms you are load testing for.

Testing rate limits with synthetic numbers

Rate limit logic keys on the phone number string, the IP and the device fingerprint. None of those need a rentable number behind them. Generate valid E.164 strings in a range your provider will never allocate, feed 60 requests for the same number in 60 seconds, and assert the 4th one returns 429 with a Retry-After header. Do the same across 500 distinct synthetic numbers from one IP to check the IP bucket. Tag those rows with synthetic=true so nobody debugs a missing SMS for a number that was never rented.

Sampling real deliveries at low volume

Real activations belong in a nightly job, not in the per-commit suite. Pick a sample that covers the axes that actually vary: 3 to 5 countries, your top 3 services, and one repeat-attempt case. Twelve to twenty activations per night catch a broken sender ID or a country that stopped receiving from a service, which no mock can catch. Because every rented number carries an activation window and refunds the price to your balance automatically when no SMS arrives, a night where a route is dead costs close to nothing. Reuse the Python or Go polling loop from your existing suite so the nightly job is a different config, not different code.

Estimating the cost of a delivery sample

Do the arithmetic before you schedule anything. Take the number of activations per run, multiply by runs per period, multiply by the price for that service and country. Prices start at $0.04 and vary, so use the highest price in your matrix, not the lowest.

Cadence	Activations per run	Runs per month	At $0.04	At $0.25
Nightly sample	15	30	$18.00	$112.50
Weekly deep matrix	60	4	$9.60	$60.00
Per-release gate	25	8	$8.00	$50.00

Refunds on failed activations pull the real number below these figures. Budget the ceiling anyway, then set a top-up alert at 40 percent of it.

Guardrails that stop a runaway test loop

1. Cap activations per process run in code, for example 40, and hard-exit when the counter trips.
2. Cap activations per hour per CI project, checked against a shared counter, not a local variable.
3. Refuse to start if the account balance is below the run’s estimated ceiling, so a partial run does not leave half a matrix untested.
4. Set a wall-clock kill at 15 minutes, since a stuck poll loop rents numbers faster than a passing one.
5. Release every number activation in a finally block, so a thrown assertion never leaves a rental holding a window open.
6. Fail the build on the guardrail trip instead of skipping quietly, because a silent skip reads as a green delivery test that never ran.

How do you monitor OTP delivery in production?

A CI suite proves the flow worked at commit time. Production monitoring proves it still works at 2am on a Tuesday when a carrier in one country starts silently dropping traffic from one sender ID. The gap between those two is where support tickets live, and the fix is a small set of metrics with thresholds you actually page on.

A minimum test matrix per market

Dimension	Values to cover	Why
Country	Every launch market	Formats, prefixes and filtering differ
Number state	Fresh, previously used, recently released	Duplicate handling
Timing	Immediate, near-expiry, after expiry	Window logic
Attempts	First, retry, exceed the limit	Lockout behaviour
Channel	SMS, voice fallback if you offer it	Fallback path

The four metrics that matter most

Track four numbers per hour, split by country and provider. Everything else is a drill-down.

Metric	Definition	Why it moves
Send acceptance rate	API calls to the SMS provider that return a success status / total send attempts	Provider outage, expired credentials, malformed numbers
Delivery rate	Delivery receipts marked delivered / accepted sends	Carrier filtering, sender ID registration lapse, spam classification
Time to first code (p95)	Seconds from send request to receipt event	Queue backlog at the aggregator, route failover
Verification completion	Users who submit a correct code / users who requested one	Any of the above, plus UX and typo rates

Send acceptance and delivery rate answer different questions. A provider can accept 100% of your requests and deliver 60% of them, and only the delivery receipt tells you that.

Delivery rate by country and carrier

Aggregate delivery rate hides the failure that matters. If 85% of your traffic is United States numbers at 97% delivery and 6% is one Indian carrier at 40%, the global number reads 93% and nobody notices. Bucket by country, then by carrier or MCC-MNC where your provider exposes it, and alert on the bucket rather than the total.

Set a minimum volume floor before a bucket can fire an alert. Ten sends in an hour at 70% delivery is noise. Fifty sends at 70% against a 14-day baseline of 96% is a route problem. Compare each bucket to its own trailing baseline, since a country that normally sits at 88% should not page you for sitting at 88% today.

Time to first code as a percentile, not an average

The average time to first code is close to useless. Most codes land in under 10 seconds, so a handful of 90-second deliveries barely move the mean while wrecking the experience for everyone in that tail. Track p50, p95 and p99 separately.

Watch p95 against your resend button timer. If the resend button becomes available at 30 seconds and p95 climbs to 34, you have just created a wave of duplicate sends, doubled spend and a second code that invalidates the first one the user is typing. That is a delivery problem that shows up as a support ticket about “wrong code.”

Verification completion funnel

Instrument five steps and store the counts as a funnel:

1. Phone number submitted.
2. Send request accepted by the provider.
3. Delivery receipt returned.
4. Code entered by the user.
5. Code accepted by your backend.

The drop between steps 3 and 4 is attention and UX, including whether autofill works. The drop between 4 and 5 is codes that arrived too late, codes typed from an older SMS, or a clock skew in your TOTP path if you also support two-factor authentication apps. Splitting those two drops keeps the mobile team and the messaging team from arguing over one blended number.

Alert thresholds and what to page on

Page on things a human can fix in the next 30 minutes. Ticket everything else.

Signal	Threshold	Action
Send acceptance rate	Below 95% for 10 minutes	Page
Delivery rate, any country with 50+ sends/hour	15 points below its 14-day baseline for 2 windows	Page
Time to first code p95	Above the resend timer for 15 minutes	Page
Verification completion	Down 10 points week over week	Ticket
Single carrier delivery rate	Below 60% for 2 hours	Ticket
Provider webhook lag	Receipts older than 5 minutes on arrival	Ticket

Require two consecutive windows before paging on delivery rate. Carriers produce short dips that resolve on their own, and a single-window trigger trains the on-call to ignore the alert.

A weekly review ritual that takes 20 minutes

1. Open the delivery rate table sorted by volume, top 10 countries, and note any bucket that moved more than 5 points.
2. Check p95 time to first code per country against last week.
3. Read the completion funnel and identify the largest single drop.
4. Rent one number per problem country and run the flow by hand, using a temporary phone number so the check costs a few cents rather than a support escalation.
5. File one ticket for the worst bucket, with the country, carrier, sample delivery receipts and the timestamps.

Step 4 is the one teams skip. Running the real flow with a real number on the suspect route turns “delivery rate looks bad in Brazil” into “the code arrives in 45 seconds and our resend timer is 30,” which is a fixable sentence.

How do you keep OTP tests stable in CI?

An OTP suite that touches real numbers and real carriers will never be as deterministic as a unit test. The goal is not zero variance, it is a pipeline where a red build means something broke in your code, not that a carrier queue was slow at 3am. That takes splitting the suite by cost and flakiness, then reporting enough context that whoever opens the failure can act in two minutes.

Failure modes worth an explicit test

Failure	How to reproduce	Correct behaviour
No SMS at all	Let the activation window expire	Retry offered, nothing charged, no half-created account
Late SMS	Code arrives after your timeout	Either accept it or explain clearly why not
Duplicate request	Tap Send twice	One code, not two competing ones
Wrong code repeatedly	Enter four wrong codes	Lockout counted per number, message says what to do
Reused number	Sign up on a number that already exists	Your merge or reject path, never a stack trace
Country mismatch	Number from an unsupported market	A message naming the supported ones

Which tests belong in every pull request

Pull request runs should finish in under 10 minutes and cost nothing per run. Keep these on every PR:

1. Contract tests against a mocked SMS provider, asserting your code sends the right request shape and parses the right code format.
2. State machine tests for the verification session: created, code sent, code entered, verified, expired, locked.
3. Rate limit and resend timer logic with a fake clock, so a 30 second resend window is tested in 3 milliseconds.
4. Code validation edge cases: wrong code, expired code, reused code, code with whitespace, code from a previous session.
5. One recorded HTTP fixture per provider endpoint, refreshed weekly, so schema drift shows up as a diff.

None of these rent a number. If a PR touches the SMS adapter itself, add a single live smoke test behind a label so contributors can opt in.

Which tests belong in a nightly job

Live delivery tests belong on a schedule, not on every push. A nightly job that rents 6 to 10 numbers costs a few dimes at $0.04 and up per number, and gives you a daily signal on the routes you actually ship to.

Test type	Trigger	Uses real number	Typical count per run
Mocked contract	Every PR	No	40 to 200
Live smoke, one country	Merge to main	Yes	1
Live matrix, top countries	Nightly	Yes	6 to 12
Full country sweep	Weekly	Yes	30 to 60
Load and throughput	On request	Partly	Varies

Order the nightly matrix so your highest volume markets run first. If the job is cut short, you still learned whether United States and India delivery worked.

Storing API keys and balance limits safely

Put the API key in the CI secret store, never in the repo and never in a fixture file. Three rules that prevent most incidents:

1. Use a separate key for CI than the one your production monitoring uses, so revoking one does not blind the other.
2. Mask the key in log output, and scrub any response body that echoes a phone number before it reaches the build log.
3. Cap the risk with balance, not with trust. Keep the CI account topped to a small working balance and refill on a schedule, so a runaway loop stops at the balance instead of running all night.

Add a pre-flight step that checks the balance before the matrix starts and fails fast with a clear message if it is below the expected cost of the run. A build that fails in 5 seconds with “balance $0.31, run needs $0.90” is better than one that fails halfway through with 14 confusing errors.

Quarantining flaky delivery tests

Carrier delay is real and it is not your bug. Handle it with policy instead of retries everywhere:

1. Set the wait for the SMS from measured data, not from a guess. If the median arrival is 12 seconds and p95 is 48, wait 90.
2. Retry a live test at most once, and only on a timeout, never on a wrong-code assertion.
3. When an activation times out with no SMS, treat it as inconclusive rather than failed. The rented number cancels and the price returns to the balance, so a retry costs one more activation, not two.
4. Track per-test flake rate over 14 days. Anything above 10 percent moves to a quarantine job that still runs and still reports, but does not block the pipeline.
5. Review quarantine weekly. A test that sits there for a month is either a real product bug or a route you should stop supporting.

Reporting results so failures are actionable

The failure message is the product here. Every live test failure should print the service, country, activation ID, the time the number was rented, the time the wait expired, and the elapsed seconds. Attach the raw message body when one arrived but did not parse, with the digits masked. Emit results as JUnit XML so your CI groups them by country, and post one summary line per nightly run to the team channel with counts, not adjectives.

A sample pipeline layout

jobs:
  unit:        # every PR, mocked, ~4 min
  contract:    # every PR, recorded fixtures
  smoke-live:  # merge to main, 1 number, US
  matrix-live: # nightly cron, 8 numbers, 6 countries
  quarantine:  # nightly, non-blocking, reports only

Keep the live jobs in one file with a shared setup step that rents the number, polls the dashboard or API for the code, and releases the session at the end. The polling helper is the same code your app uses, so the same receive SMS in Python or Node.js example works as the CI fixture with the wait constant raised.

How should a team share numbers, access and budget?

Test numbers get messy when four engineers each keep a personal balance and nobody knows which run spent what. The fix is one shared account, keys scoped per environment, and a naming rule that ties every rented number back to the suite that asked for it.

What to log on every verification attempt

Field	Why it matters
Request id	Ties the attempt to provider logs
Country and prefix	Shows which markets fail
Provider and route	Compares delivery between vendors
Time to first code entry	The real user-visible latency
Outcome	delivered, expired, wrong code, abandoned

Aggregate these by country and by hour; a delivery problem shows as a country going quiet.

One account with per-environment keys

Create a single MarioSMS account owned by the team, not by a person who might change jobs. Then issue separate API keys for local, CI and staging. Same balance, different keys, so you can revoke one without breaking the other two.

Key	Used by	Typical volume	Rotate
local	Developer laptops	1 to 5 numbers per day	On offboarding
ci	Pipeline runner secret	10 to 40 per day	Quarterly
staging	Scheduled smoke jobs	4 to 8 per day	Quarterly
load	Throughput experiments	Bursts, then idle	After each experiment

Store keys in your secret manager, never in the repo. Local keys go in a .env file listed in .gitignore, and the CI key lives as a masked variable so it does not print in job logs.

Tracking spend per test suite

Numbers start at $0.04 and vary by service and country, so a suite that rents 30 numbers per night costs a different amount than one renting 30 in a pricier country. Log the cost at rent time, not at month end.

1. Have the rent helper write one line per activation: timestamp, suite name, service, country, price, activation ID.
2. Ship those lines to the same place your test artifacts go (a CSV in the job output is enough to start).
3. Sum by suite weekly and compare against the dashboard balance drop.
4. Investigate any suite where the count grew more than 20 percent week over week.

Refunds matter here. If no SMS arrives inside the activation window, the activation cancels and the price returns to the balance automatically, so your logged spend will overstate real cost unless you mark cancelled rows. Add a refunded column and reconcile it after each run.

Top up by card or crypto and when to do it

Balance tops up by card or crypto. Pick a floor that covers a full week of the heaviest suite, then top up when the balance crosses it. If nightly runs use 40 numbers and your average is around $0.10, a week is roughly $28, so a $50 floor gives real headroom without parking money you do not need.

Set a calendar reminder rather than waiting for a failed rent in CI at 2am. A dry balance turns into a red pipeline that looks like an app bug for the twenty minutes it takes someone to check the dashboard.

Handing off a run between QA and engineering

When QA finds a broken verification flow, engineering needs four things to reproduce it: the country, the service, the activation ID and the exact SMS text. Put those in the ticket template as required fields. The activation ID is the anchor because it maps the run to a specific rented virtual phone number and its timer.

Screenshots of the app screen help, but the raw message body helps more, since sender ID and code length differ by route and that difference is often the actual defect.

Documenting which numbers were used and why

Keep a plain table in the repo, updated by the rent helper, not by hand. Columns: date, suite, country, service, activation ID, outcome (code received, timed out, refunded). Retain it for 30 to 90 days, then delete, because it is throwaway operational data and long retention creates a privacy problem you do not need. Every number is rented for one verification at a time, so the log stays one row per attempt.

Use this only for privacy hygiene, second accounts your team legitimately owns and QA or OTP testing, per the acceptable use rules.

Apps on iOS and Android for manual runs

Automation covers the repeat cases, but exploratory testing still happens by hand. The iOS and Android apps and the web app at app.mariosms.com all show the same balance and the same activation list, so a tester on a phone rents a number, watches the code land, usually within a minute, and pastes it into the app under test without touching a terminal. Pick the country from stock across 35+ countries, tap rent, and the activation appears in the shared dashboard where the rest of the team can see it.

What privacy and legal limits apply to verification testing?

Rented numbers are a normal part of a test kit, and they carry the same responsibilities as any other credential your team handles. The limits are not complicated, but they need to be written down somewhere your QA engineers, your contractors and your new hires can read them before they rent their first number.

Metrics to alert on in production

Metric	Healthy shape	Alert when
Code entry rate	Steady by country and hour	Drops by a third against the same hour last week
Median time to entry	Tens of seconds	Doubles
Expired activations	A small steady share	Spikes in one country
Retry rate	Low	Climbs with no release

Privacy hygiene as the core use case

Most teams start renting numbers because they do not want personal handsets in test data. A QA lead who signs up for twelve staging accounts with a personal phone number ends up with that number in twelve databases, some of which will be exported, some of which will be backed up in places nobody tracks. Marketing messages follow. So do password reset attempts if any of those systems get breached.

A rented number breaks that chain. It handles one verification, the activation closes, and the number goes back to the pool. Nothing links the test account to a human’s contact record. The same logic applies to a founder testing a competitor’s onboarding flow or a support engineer reproducing a customer bug: they need a working number, not their own.

Second accounts for legitimate reasons

Plenty of legitimate work needs more than one account. A social media manager runs the brand account and a personal one. A developer keeps a sandbox account separate from the production account on the same platform. A support team needs a customer-side account to see what users see. A localisation tester needs an account registered in the market they are testing, which usually means a number from that country, such as an Indian number or a US number.

Check the terms of the service you are registering with. Some allow multiple accounts outright, some allow them with a business relationship, some restrict them. The number source does not change what those terms say.

QA and OTP testing on systems you own or are authorised to test

The cleanest case is testing your own product. You control the sender, the code format, the retry policy, and the account lifecycle, so nothing about the test touches a third party except the carrier path. Write your test accounts to a staging database, tag them, and delete them on a schedule.

When you test somebody else’s system, get authorisation in writing first. A signed penetration test scope, a client contract that names the systems, or a bug bounty program with a published policy all count. Verbal permission from a person who does not own the system does not. Rate limits and abuse detection on OTP endpoints exist for a reason, and hammering an endpoint you were not invited to test is a problem regardless of where the numbers came from.

What is never acceptable

Two things are always out of bounds. The first is getting back into an account you were banned from, whether the ban was for spam, fraud, harassment or a terms violation. The ban is the platform’s decision about you, not about your phone number. The second is pretending to be someone else: registering under a real person’s name, taking over an identity, or creating accounts designed to look like a company or individual you are not.

Fraud, spam campaigns and anything targeting another person also sit outside acceptable use. MarioSMS suspends accounts for this. The full policy lives at /acceptable-use/.

Data retention for test artifacts

OTP test runs generate artifacts: logs with numbers in them, screenshots of verification screens, CI job output, database rows for test accounts. Treat all of it as personal data even though the numbers are rented.

Artifact	Suggested retention	Handling
CI logs with activation IDs	14 to 30 days	Mask the last 4 digits of numbers
Screenshots from failed runs	7 days	Auto-delete from the artifact bucket
Test account rows in staging	Until the sprint closes	Delete by tag, not by hand
Received code bodies	Do not store	Assert on the extracted code, then drop the text

Codes themselves are short-lived, often 5 to 10 minutes, but a stored SMS body sitting in a log for a year is still a record of a message sent to a real number.

Pointing teammates to the acceptable use page

Put the link in three places: the onboarding doc for new QA hires, the README of the test repo that calls the API, and the pinned message in whatever channel your team uses to request budget top-ups. When somebody asks whether a use case is allowed, answer from the policy text rather than from memory, and if the case is genuinely unclear, ask support before the rental rather than after the account gets created.

Frequently asked questions about testing OTP and SMS verification

How fast does a test code usually arrive?

Most codes land in the app or dashboard within a minute of the send request. The variance comes from the route, not the rental. A domestic route to a major carrier often delivers in 5 to 15 seconds. A cross-border route through several hops can take 40 seconds or more. Set your polling timeout at 120 seconds and treat anything faster as a bonus, not a guarantee.

Keeping the suite stable in CI

Problem	Fix
Flaky timeouts	Set the test timeout longer than the activation window, not shorter
Parallel runs colliding	One number per worker, released in teardown
Budget creep	Cap runs per day, use the cheapest market for smoke tests
Secrets in logs	Redact numbers and codes in test output

What happens if no SMS arrives at all?

Every number carries an activation window. If the window closes without an SMS, the activation cancels and the price returns to your balance automatically. You do not file a ticket or wait for a manual review. For a test suite this matters more than it sounds, because a broken build that fires 200 failed activations costs nothing beyond the wasted minutes. Log the cancellation as a test failure anyway, since a missing code is a real signal about the sender.

Can one number receive two codes?

No. MarioSMS rents a number for one verification at a time. A resend inside the same activation window sometimes arrives on the same rental, but do not design around it. If your test case covers “user taps resend twice and uses the second code,” treat the retry path as a separate scenario and check whether the flow needs a fresh rental. Assume one number, one code, and your test stays deterministic.

How much does a single test number cost?

Prices start at $0.04 and vary by service and country. A cheap service in a high-supply country sits near the floor. A number for a service with tight sender rules in a smaller market costs more. Price your suite by multiplying the per-run rental count by the highest number in your matrix, not the lowest, then compare that to the cost of an engineer chasing a flaky manual test.

Do I need a different number per test case?

Yes for anything that creates an account, and yes for anything that checks the “this number is already registered” branch. Reuse invites cross-test contamination where case B passes only because case A left state behind. Pure delivery checks (does an SMS reach this route at all) can share a rental within one activation, but that is the only safe overlap.

How do I test expiry without waiting?

Three options, in order of preference:

1. Move the clock in your backend. Set the OTP record’s created_at backwards through a test-only endpoint, then submit the code.
2. Shorten the TTL in the test environment. A 30 second expiry gives you a real wall-clock test in under a minute.
3. Rent a number, receive the code, and hold it past the real window. Slow and expensive, but it is the only variant that exercises the production TTL exactly as shipped.

Use option 1 for CI and option 3 once per release.

Can I automate the whole flow through an API?

Yes. The REST API covers rental, code retrieval and cancellation, with examples in Python, Node.js, Go and PHP. A working automated case is four calls: request a number, trigger your app’s send, poll for the code, release or let the activation expire. Wrap those in a fixture and the test reads like any other integration test.

How many countries are available?

35+ countries and hundreds of services are in stock, with availability shifting as supply moves. Check stock at rental time rather than hardcoding a country into a test config. If your matrix requires a specific market, add a skip condition so the suite reports “no stock for XX” instead of a red failure that engineers waste an hour debugging.

Do emulators and simulators receive SMS?

Not real ones. An iOS Simulator has no radio and no SIM. An Android emulator accepts an injected SMS through the console (sms send on port 5554), which tests your parsing and autofill logic but tells you nothing about carrier delivery. Split the concerns: injected messages for UI behavior, a rented real number for the delivery path.

How do I test voice call fallback?

Fallback usually triggers after the SMS attempt fails or the user taps “call me instead.” To reach that branch reliably, let the SMS attempt time out rather than trying to force an error. Confirm your app shows the fallback control at the right moment, then verify separately that the voice provider is configured. Numbers rented for SMS receive text, so plan voice checks against your own provider dashboard.

Should I test with my own personal number?

No, for four reasons: your number accumulates registration state that skews results, you cannot test the “new user” path twice, teammates cannot reproduce your run, and your personal number ends up in shared CI logs. Use rented numbers and keep the personal one out of the repo.

What belongs in a bug report for a failed verification?

Field	Example
Timestamp (UTC)	2026-09-10 14:22:05
Country and service	India, messaging app
Number (last 4 only)	…4471
Activation ID	act_8812f
Send request status	HTTP 200, sender accepted
Time to timeout	118 s
Sender ID seen (if any)	none
App build	4.7.1 (2211)

Eight fields, and support or your SMS provider can trace the route without a follow-up thread.

How do I test rate limiting without getting blocked?

Point the limiter tests at your own backend with a stub SMS sender. Counting requests is your logic, so it does not need a real message. Fire 10 sends in 60 seconds against the stub, assert the 429 and the retry-after header, then run one real rented number through the happy path to confirm the limiter is not blocking legitimate traffic.

Can I reuse a number next week?

Assume no. Each rental covers one activation, and the same number will not be waiting for you later. Design tests to be identity-free: no fixture that expects a specific phone string, no golden file with a number in it, no account that must survive between runs. Generate the identity at test time and tear it down at the end.

Run through the pre-launch checklist before you ship phone verification

Print this, walk it top to bottom, and tick items only after you have seen the behavior in a real environment. Every item below is something a team has shipped broken at least once.

Where the money goes in a test plan

Run	Numbers used	Rough cost
Smoke test per deploy	1 to 2	Cents
Full matrix per market	6 to 10	Under a dollar
Weekly regression, 8 markets	60 to 80	A few dollars
Load test, 1,000 sign-ups	1,000	Tens of dollars, at catalog prices

Catalog prices, 2026-09-10; failed activations are refunded, so the real bill is lower.

Code generation and storage checks

1. Codes come from a cryptographically secure random source, not Math.random() or a seeded PRNG.
2. Code length and alphabet are fixed and documented (6 digits is the common default for OTP flows).
3. The code is stored hashed, with the phone number and purpose bound into the record, so a code minted for login cannot be replayed against password reset.
4. Expiry is stored as an absolute timestamp, checked server side, and enforced even when the client sends a stale value.
5. Verification is single use. The record is marked consumed inside the same transaction that issues the session.
6. Comparison is constant time, so response timing does not leak how many leading digits matched.
7. Phone numbers are normalized to E.164 before storage and before lookup, so +1 415 555 0142 and 4155550142 resolve to one record.

Delivery and fallback checks

1. The send call has a timeout (5 to 10 seconds) and a retry policy that does not double-send on a slow but successful response.
2. Provider errors are mapped to user-facing states: invalid number, unsupported country, carrier rejected, provider down.
3. A resend button exists, is disabled for the first 30 to 60 seconds, and reuses the same code rather than minting a new one on every tap.
4. At least one fallback path is wired (second provider, voice call, or email) and you have triggered it manually, not just read the code.
5. Long message bodies, non-Latin sender IDs and unicode in the template have been sent to a real handset in each launch country.
6. Delivery receipts are consumed and stored, so “sent” and “delivered” are separate states in your data.

Abuse and rate limit checks

1. Limits exist per phone number, per IP, per account and per device fingerprint, each with its own window.
2. The enumeration response is identical whether or not the number is registered.
3. Cost ceilings are set per hour and per day, with a hard stop rather than an alert only.
4. High-cost destinations are allow-listed or blocked, since traffic pumping targets exactly the ranges you do not sell to.
5. Failed verification attempts are capped (5 is typical) and the record is invalidated after the cap, not just rejected.

Monitoring and alerting checks

Signal	Alert when	Why it matters
Send success rate	Drops below your 7-day baseline by 5 points	Provider or account issue
Delivery rate per country	Any country falls 10 points in an hour	Carrier filtering
Median code entry time	Rises above 90 seconds	Slow delivery upstream
Verification completion rate	Drops sharply on one platform	Client bug, not delivery
Spend per hour	Exceeds your ceiling	Pumping or a retry loop

Each alert needs an owner and a page-or-ticket decision made before launch, not during the first incident.

Documentation and runbook checks

1. A runbook page names the provider, the account owner, the support channel and the expected response time.
2. Steps for switching providers are written as commands or config keys, not prose.
3. The support team has a script for “I never got the code” that covers country, carrier, roaming, DND registries and handset filters.
4. Test credentials, sandbox numbers and the rented-number workflow are documented so a new engineer can run the suite on day one.
5. Data retention for phone numbers and code records is written down with a number of days, and the deletion job is scheduled.
6. Your acceptable use position is clear internally: second accounts, QA and privacy hygiene are supported, ban evasion and impersonation are not.

The five-minute smoke test after every deploy

1. Rent one number from MarioSMS in your largest market, for example the United States, for $0.04 and up depending on the service.
2. Trigger a send from the deployed build and start a timer.
3. Confirm the code appears in the dashboard within the activation window (usually under a minute; if nothing arrives, the activation cancels and the balance is refunded).
4. Enter the code, confirm the session issues, then re-submit the same code and confirm it is rejected.
5. Trigger one deliberate failure (expired code) and confirm the error string matches what support expects to hear.