A four-clinician family practice preparing for PCMH recognition had a patient survey. It was a comment card at checkout with five smiley faces and a line for suggestions. Over a year they had collected 212 cards, 190 of them the happiest face, and the practice manager honestly believed that was their patient experience data. When we asked what the practice would change based on the cards, nobody could name anything. That is the difference between collecting opinions and measuring experience, and NCQA's standards are built around the second one.
A patient experience survey for PCMH has a specific job. NCQA's Performance Measurement and Quality Improvement concept, criterion QI 04, is a core requirement: the practice monitors patient experience through a survey covering at least three of access, communication, coordination and whole-person care (which NCQA also describes as self-management support and comprehensiveness), and through qualitative feedback such as a patient advisory council or interviews. Elsewhere in the QI concept, the practice has to set a goal on at least one patient experience measure and act on it. A smiley face card meets none of that.
Two definitions for readers new to this. CAHPS stands for Consumer Assessment of Healthcare Providers and Systems, the family of standardized patient survey instruments developed and maintained by the Agency for Healthcare Research and Quality (AHRQ). The Clinician and Group survey, CG-CAHPS, is the version built for office practices, and the CAHPS PCMH item set is a supplemental group of questions written for medical homes. NCQA recommends the CAHPS PCMH survey but does not require it; any instrument that covers the required dimensions will do for QI 04.
Key takeaways
- The survey must cover at least three of four dimensions (access, communication, coordination, whole-person care), so pick questions by dimension first and instrument second.
- Use standardized CAHPS-style wording and the Never/Sometimes/Usually/Always scale so your results can be compared to published benchmarks and to yourselves next year.
- AHRQ guidance for CG-CAHPS runs from 50 to 300 completed surveys depending on how many clinicians you report on, and you need to send several times that many.
- Score with the top-box method (percent answering Always, or rating 9 or 10) rather than averages, because that is how benchmarks are published and it is harder to fool.
- Every survey cycle should end with one documented change tied to one low-scoring item; that is the evidence NCQA actually reads.
Building the patient experience survey for PCMH: questions by dimension
Start with the four dimensions and put at least two questions under each of the three or four you choose. The CG-CAHPS 3.0 core survey organizes its questions into composites that map cleanly: Getting Timely Appointments, Care and Information (access); How Well Providers Communicate With Patients (communication); Providers' Use of Information to Coordinate Patient Care (coordination); plus an overall provider rating from 0 to 10 and a composite on helpful, courteous and respectful office staff. The CAHPS PCMH item set adds questions on self-management support, shared decision making, comprehensiveness (asking about mental or emotional health) and after-hours access, which is where the whole-person care dimension comes from.
Here is a compact set of ten items we often recommend for a small practice, all in CAHPS-style wording with a six-month look-back. For access: when you phoned this provider's office to get an appointment for care you needed right away, how often did you get an appointment as soon as you needed? When you phoned during regular office hours, how often did you get an answer to your medical question that same day? For communication: how often did this provider explain things in a way that was easy to understand? Listen carefully to you? For coordination: when this provider ordered a blood test, x-ray or other test, how often did someone from the office follow up to give you the results? For whole-person care: did anyone in this office talk with you about specific goals for your health? Did anyone ask whether there are things that make it hard for you to take care of your health?
Add the 0 to 10 provider rating and two open-text questions (what could we do better; what should we keep doing) for the qualitative side. Resist adding questions about parking and decor; they are not in the dimensions and they produce feedback that consumes a staff meeting without changing care.
How many responses you need
The AHRQ guidelines for the CG-CAHPS survey give a table of recommended completed surveys that ranges from about 50 to 300 depending on how many clinicians you want to report on. Report at the practice level; comparing individual clinicians fairly needs far more completes per clinician than most small practices can collect.
Working backward from completes is the step that gets skipped. AHRQ's own example: to end up with 300 completed surveys at a 40 percent response rate and a 15 percent ineligible rate, you need a starting sample of 882 patients. Small practices using text message or email links see lower response rates than that in our experience, often 15 to 25 percent, so the starting sample has to be larger still.
| Target completed surveys | Expected response rate | Ineligible rate | Starting sample needed | Realistic for a practice with |
|---|---|---|---|---|
| 50 | 25% | 10% | 223 | Any size; enough for a practice-level directional read |
| 100 | 25% | 10% | 445 | 1,500 or more active patients |
| 150 | 20% | 10% | 834 | 3,000 or more active patients |
| 300 | 40% | 15% | 882 | Mixed mail and phone modes, larger practices |
The formula is simple: starting sample equals target completes divided by (response rate times (1 minus ineligible rate)). For the 150-complete row: 150 divided by (0.20 times 0.90) is 833.3, so 834. Draw the sample from patients with a visit in the last six months, one survey per patient per year, randomly rather than by asking staff to pick, and exclude nobody for being unhappy.
Scoring: top-box, not averages
CAHPS results are reported as top-box scores: the percentage of respondents who chose the most positive answer, Always on the frequency items, Yes on the yes/no items, and 9 or 10 on the 0 to 10 rating. Averages hide the distribution; a mean of 3.4 on a four-point scale tells a physician nothing, while "61 percent of patients said we Always followed up on test results" is a sentence a team can act on. Composites are the average of the top-box percentages of their member items.
A worked example for the four-clinician practice in the opening, first real survey, 118 completes.
| Composite or item | Top-box score | Where practices we work with typically land | Read |
|---|---|---|---|
| Getting timely appointments, care and information | 58% | Mid 60s to low 70s | Below range; the same-day phone answer item was 44% |
| How well providers communicate | 84% | Low to mid 80s | In range |
| Coordination: results follow-up | 61% | Low to mid 70s | Below range; the single lowest item after phone access |
| Whole-person care: asked about health goals | 39% | Varies widely; PCMH item | Baseline for next year |
| Provider rating 9 or 10 | 79% | Around 80 | In range |
The third column is our experience, not a published standard. Real benchmarks come from the AHRQ CAHPS Database, which publishes CG-CAHPS results submitted by participating organizations, and from your own prior years. Treat them as ranges, not targets; a practice serving a population with heavy phone demand and one receptionist will sit below the access benchmark for structural reasons, and the survey just told you that.
Turning a low score into a change
The results table above contains exactly two items worth acting on this year: same-day phone answers and test result follow-up. That is the whole list. Practices that try to improve every item improve none of them.
Take result follow-up at 61 percent. The practice pulls 20 recent lab orders and traces what happened: 14 were released to the portal with no message, 4 were called, 2 were never reviewed. The cause is not attitude; it is that "release to portal" counts as follow-up in the workflow and does not count as follow-up to the patient. The change is a rule: every result gets a portal message from the ordering clinician or nurse in plain words within two business days, with a phone call for abnormal results and for patients not enrolled in the portal. That is a PDSA cycle (plan, do, study, act, the quality improvement method NCQA expects): a measurable aim (raise the item to 75 percent on the next survey), a change, a way to measure it monthly (percentage of results with a message within two days, pulled from the EHR), and a review date.
Document it as NCQA wants to see it: the baseline score, the goal, the change made and when, the interim process measure, and the re-measured score. A one-page summary per improvement, dated, is enough. If your practice is heading toward recognition or an annual review, this document is what the reviewer opens first.
Qualitative feedback: the half everyone forgets
QI 04 also asks for qualitative feedback, and this is cheaper than the survey. Three formats work in small practices. A patient and family advisory council, five to eight patients meeting quarterly for an hour with the practice manager and one clinician, with an agenda built from the survey results. Brief structured interviews, ten patients a quarter, five questions, done by a staff member in the waiting room or by phone. And reading the open-text comments, coded into themes and counted.
The advisory council produces the insights the survey cannot. In one practice we supported, the survey said phone access was poor; the council explained that the phone tree's second menu had no option for "I am calling back about a message you left me," so those calls went to the appointment queue. A 30-minute phone tree edit, and the next survey showed it. You cannot get that from a top-box score.
Mistakes that make the survey worthless
Handing surveys to patients at checkout, where staff choose who gets one and patients answer in front of the desk. Surveying only patients who use the portal. Rewriting the questions each year so trends cannot be compared. Running the survey and holding no meeting about it. Treating a low score as a staff discipline matter rather than a workflow problem. And the most common one: doing all this in the six weeks before the NCQA submission and never again. NCQA's expectation is ongoing monitoring, and our transformation team spends much of its time helping practices make the survey a calendar item rather than a project.
Questions we hear
Do we have to use the official CAHPS PCMH survey?
No. NCQA recommends it and offers credit for using it, but QI 04 accepts any survey that covers at least three of the four dimensions. We suggest CAHPS wording for the questions you do use, because it is validated and benchmarkable, even if you do not field the full instrument.
Can we use the survey our EHR vendor sends automatically after visits?
Often, with two checks. Map its questions to the four dimensions and confirm at least three are covered with more than one item each. Then confirm it is sent to all patients or a random sample, not only portal users, and that you can export item-level results. If it fails either check, supplement it rather than replace it.
How often should we survey?
Continuous sampling, with a small number of surveys going out every week and results reviewed quarterly, is better than one annual burst. It produces enough completes over a year for a small practice, and gives the improvement team a fresh number every quarter to check whether the change worked.
What to do this week
- Write down which three or four dimensions you will measure and list at least two CAHPS-style questions under each.
- Count active patients with a visit in the last six months and calculate your starting sample from the target completes, response rate and ineligible rate.
- Decide the mode (text link, email, mail, or a mix), the random draw method and who owns the weekly send.
- Build the results template with top-box scoring by item and composite, and a column for last year's score.
- Schedule the first quarterly review meeting now, with the rule that it ends with one improvement aim and one owner.
- Recruit five to eight patients for an advisory council, or schedule ten brief interviews, so the qualitative half of QI 04 is covered from the start.
