Who Answered and Who Did Not: What a 19 Percent Patient Experience Survey Sample Can Honestly Represent at an Outpatient Surgery Center
Student Name
American College of Education
RES5303: Research Methods and Applied Statistics in Healthcare
Module 2 Assignment
Instructor Name
September 13, 2027
The Claim and Its Sample
An outpatient surgery center, a composite created for this assignment, performs about 7,200 procedures a year in orthopedics, ophthalmology, gastroenterology and general surgery. Its marketing materials state that 94 percent of patients would recommend the center to friends and family. The figure comes from a patient experience survey mailed to every patient within a week of their procedure. In the most recent year, 1,368 of the 7,200 surveys were returned, a response rate of 19 percent, and 1,286 of the returned surveys answered the recommendation question positively.
The claim is about all patients: 94 percent of them would recommend the center. The evidence behind it comes from the 19 percent who chose to respond. Whether the claim is justified depends on how those respondents relate to everyone else, and that is a question about sampling. A sample does not represent a population simply because it came from it; it represents the population only to the extent that the process of getting into the sample did not depend on the thing being measured.
Population, Frame and Method
The target population, the group about which the center wants to make its claim, is all patients who had a procedure at the center during the year. The sampling frame, the list from which the survey was actually drawn, is the scheduling system's list of completed procedures with a valid mailing address. The frame is close to the population but not identical: about 4 percent of patients had no usable address, including patients with unstable housing and some who gave a relative's address, so they could not be surveyed at all. That is a small coverage gap, but not a random one.
Technically, the center did not draw a sample; it attempted a census, surveying every patient on the frame. The sample was produced by patients themselves, through the decision to return the survey. That makes it a self-selected sample, a form of nonprobability sampling in which each patient's chance of inclusion depends on their own choice and cannot be known in advance. Polit and Beck (2021) note that nonprobability samples can be informative, but that their results cannot be generalized to a population with the confidence that probability sampling allows, because the selection process may be related to the outcome.
Who Responded
Comparing respondents with the full frame on data the center already holds shows that the respondents were not a miniature of the patient population. Among respondents, 58 percent were aged 65 or older, compared with 39 percent of all patients. Medicaid and self-pay patients made up 21 percent of all patients but 9 percent of respondents. Ophthalmology patients, most of them older adults having cataract surgery, were 30 percent of patients and 44 percent of respondents, while general surgery patients were underrepresented.
The same pattern appears in published work. Tyser et al. (2016) examined responses to a widely used commercial patient satisfaction survey among orthopedic patients and found that only 16.5 percent responded, that older age strongly increased the odds of responding and that male sex, Medicaid or self-pay insurance and trauma subspecialty care decreased them. The center's respondents resemble that profile: older, more often privately insured or on Medicare and weighted toward elective, predictable procedures. If those patients also tend to report more positive experiences, the 94 percent figure would overstate the experience of the whole population.
Response Rate Is Not the Same as Response Bias
It is tempting to conclude that a 19 percent response rate makes the survey worthless. That conclusion is too quick. Groves (2006) reviewed studies that measured both response rates and nonresponse bias and found that the response rate alone was a poor predictor of bias. What matters is whether the likelihood of responding is related to the variable being measured. A survey with a low response rate can be nearly unbiased if people respond for reasons unrelated to the question, and a survey with a high response rate can be badly biased if the missing few are systematically different.
For this survey, the question is whether patients who had a worse experience were less likely to respond. Two findings suggest they may have been. Groups that responded less, younger patients, Medicaid and self-pay patients and general surgery patients, are also groups that in the center's own complaint log report more problems with wait times and billing. And survey mode matters: Elliott et al. (2009), in an experiment with a national hospital patient experience survey, found that patients randomized to telephone and interactive voice modes gave more positive evaluations than those randomized to mail, and that mode and patient mix adjustments were needed for fair comparisons. The low response rate is a warning; the pattern of who did not respond is the evidence.
What the Sample Can Honestly Represent
The sample can support a narrower claim than the one the center makes. It can honestly say that 94 percent of patients who returned the survey would recommend the center, and it can report that figure for subgroups where respondents are numerous, such as ophthalmology patients aged 65 and older. It cannot support a claim about all patients, and it cannot be used to compare service lines fairly, because the service lines differ so much in who responded.
The sample is still useful for improvement. Comments from 1,368 patients identify specific problems, and trends over time within the same survey method can show whether a change made a difference among the kinds of patients who respond. What the sample cannot do is stand in for the patients who did not answer, who are younger, less often privately insured and more often in the service lines with the most complaints.
A Design That Matches the Claim
If the center wants to make claims about all its patients, it needs a probability sample with active follow-up. The proposed design stratifies the patient population by service line and insurance type, creating eight strata, and draws a random sample of 80 patients from each stratum every quarter, or 640 patients a quarter. Each sampled patient receives the mailed survey, followed by a text message with a web link after five days and a telephone call after ten days for those who have not responded. Mixing modes this way tends to raise response among groups less likely to return mail, and the design records the mode of each response so that mode effects can be adjusted for in analysis, as Elliott et al. (2009) recommend.
Results would be weighted back to the population using the known size of each stratum, so that underrepresented groups count in proportion to their share of patients. With 640 patients sampled a quarter and a target response rate of 45 percent, the center would have about 290 responses a quarter, enough to estimate the overall recommendation rate within about 6 percentage points with 95 percent confidence. That is a smaller sample than the current census, and a far more honest one.
Conclusion
The center's 94 percent figure rests on a self-selected sample of 19 percent of patients who are older, more often privately insured and concentrated in one service line. The low response rate alone does not prove bias, but the pattern of nonresponse and the evidence on survey mode suggest the figure overstates the experience of patients as a whole. The existing survey can honestly describe its respondents and track trends among them. A stratified probability sample with mixed-mode follow-up and weighting would allow the claim the center wants to make, supported by a sample designed to make it.
References
Elliott, M. N., Zaslavsky, A. M., Goldstein, E., Lehrman, W., Hambarsoomians, K., Beckett, M. K., & Giordano, L. (2009). Effects of survey mode, patient mix, and nonresponse on CAHPS hospital survey scores. Health Services Research, 44(2, Pt. 1), 501-518. https://doi.org/10.1111/j.1475-6773.2008.00914.x
Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70(5), 646-675. https://doi.org/10.1093/poq/nfl033
Polit, D. F., & Beck, C. T. (2021). Nursing research: Generating and assessing evidence for nursing practice (11th ed.). Wolters Kluwer.
Tyser, A. R., Abtahi, A. M., McFadden, M., & Presson, A. P. (2016). Evidence of non-response bias in the Press-Ganey patient satisfaction survey. BMC Health Services Research, 16, Article 350. https://doi.org/10.1186/s12913-016-1595-z
How this RES 5303 Module 2 example is structured
RES 5303 Module 2 typically covers sampling and what a drawn sample can honestly represent; your classroom's instructions decide whether the paper critiques an existing sample or designs a new one. This example does both. It starts with the claim being made, describes the population, frame and method behind it, then examines who responded and why that matters, using published evidence on nonresponse. A separate section distinguishes response rate from response bias, a point students often confuse. The proposed redesign ends the paper with a sample that matches the claim the center wants to make.
RES5303 Module 2 questions, answered
What does RES5303 Module 2 usually ask for?
RES5303 Module 2 typically covers sampling: the difference between a population, a sampling frame and a sample, the main probability and nonprobability methods, and what a given sample can represent. Many sections ask students to critique a sample or design one for their own question. Your classroom's instructions decide which.
Does a low response rate always mean a survey is biased?
No. Bias depends on whether the likelihood of responding is related to what the survey measures. A low response rate raises the risk, but the key evidence is how respondents differ from nonrespondents on characteristics linked to the outcome. Compare the two groups on data you already hold before drawing a conclusion.
What is stratified random sampling?
Stratified random sampling divides the population into groups, or strata, such as service lines or insurance types, and draws a random sample from each. It ensures that important subgroups are represented and allows results to be weighted back to the population. It is especially useful when some groups are small or tend to respond less.
Write yours, or have the desk draft it
This paper is an original model document written by our desk, not a submitted student paper and not an official American College of Education document. Read it for the moves, then write your own to the instructions in your classroom. If you want one built to your exact prompt and rubric, the first custom sample is free and arrives in 24 to 48 hours.