What the HHFA measures and what it does not: a critical reading
This chapter records the conclusions reached while extracting the four modules, comparing them with the Emergency Obstetric and Newborn Care (EmONC) assessment and the Primary Health Care (PHC) Measurement Framework, and reading a published national report. The frame is Donabedian’s distinction between structure (resources and systems), process (what is done to and for the patient) and outcome (what happens to the patient).
1. What the HHFA measures well, and where process and outcome enter
The four modules do not sit at the same point on the Donabedian scale. Modules 1 and 2 are almost entirely structure. Modules 3 and 4 add process and outcome, at a smaller scale and with different methods.
Structure: the HHFA’s strength
Modules 1 and 2 measure structure with a breadth and precision that few facility instruments match.
- Building structure per service area, by direct observation, with materials and condition recorded separately (Module 1).
- 1,441 tracer items in the readiness core questionnaire, each with an availability status (observed / reported, not seen / not available) and a functioning status (Module 2).
- For medicines, expiry status and any stock-out in the past three months.
- Standardized readiness domains and indices (basic amenities, basic equipment, infection prevention, diagnostics, essential medicines) that are comparable across facilities, over time and across countries.
- Service-specific readiness computed only among facilities that offer the service.
For a costed investment plan this is the core input: an inventory of gaps that can be priced.
Process and outcome: present in Modules 3 and 4
Module 3 measures process directly. The record review checks, in the client’s own record, whether the steps required by the clinical standard were documented, for six tracer conditions.
Module 4 carries process and outcome in three forms.
Practices that were actually carried out. Beyond asking whether a system exists, many questions ask when it last ran or what it produced: the last external supervision visit, the last data-review meeting, whether a case review or death review took place in the past three months, whether an audit report can be observed, what percentage of the recurrent budget was received.
Outcome rates. Section 16 collects bed occupancy, average length of stay, case fatality, post-operative infection, 30-day mortality, readmissions and avoidable admissions. The values are as reported by the facility and each block is behind a gate question, so the data describe facilities that monitor these rates, not all facilities.
Interpreted external scores. Accreditation status, the Infection Prevention and Control Assessment Framework (IPCAF) level and the Hand Hygiene Self-Assessment Framework (HHSAF) level.
The quality-monitoring block of the combined core questionnaire (June 2023, Section 7, printed pages 13–16) shows how these sit together. Most questions ask about the system: does the facility take part in external assessment (701), which certification processes (702), is there a quality committee and how often does it meet (707–708), is there a feedback system (730). A smaller set asks for a result or an event: year of the last accreditation (703), date of the last committee meeting (709), documentation that quality information was reviewed (710), a case review or death review in the past three months (718, 721), the interpreted IPCAF and hand-hygiene scores (736, 740). Section 11 follows the same layout for data quality (last external verification 1114, data-review meeting frequency 1117, use of data for planning 1118) and for morbidity and mortality coding (1120–1134).
Summary by module
| Module | Structure | Process | Outcome |
|---|---|---|---|
| 1. Availability / building structure | Almost all | — | — |
| 2. Readiness | Almost all | Stock-out in past three months | — |
| 3. Quality of care | — | Documented care steps per client, six tracers | — |
| 4. Management and finance | Most questions | Practices carried out and their dates; budget received | Section 16 rates; interpreted external scores |
Reading
The HHFA leans on structure by design, and that is where it is most reliable. Process and outcome are present, mainly in Modules 3 and 4, and should be used. Two limits apply to them: outcome rates are self-reported and gated, and the process measurement in Module 3 rests on a small sample. Neither limit makes the data unusable; both mean the results should be read alongside the routine health information system rather than as a substitute for it.
2. Module 3 is the whole of the HHFA’s direct process measurement
The record review is the only place where care delivered to a named client is observed. Its scope is small: five records per condition per facility, six tracer conditions, a six-month lookback, with replacement allowed for any record that cannot be found. Replacement is itself a data quality signal that the survey records but does not score.
In the crosswalk against the PHC Measurement Framework, Module 3 produces one indicator: adherence to clinical standards (indicator 77), for six tracers.
This has a practical consequence. When a country drops Module 3, it drops all process measurement. That is what happened in Guiné-Bissau (section 7 below).
3. Readiness is a snapshot
Module 2 records what is present and functioning on the day of the visit. The three-month stock-out column for medicines is the only backward-looking item. A facility can score high on readiness while its supply chain is failing, and the survey cannot tell the two apart without routine logistics data. For an investment plan, the question is which of those two problems is being funded.
The observed / reported-not-seen / not available scale is the instrument’s verification device. It allows a strict readiness index (observed only) and a permissive one (observed plus reported) from the same data. Both should be reported when results are contested.
4. Two mechanical traps in country adaptation
Indicators attached to items. In Module 2, 379 indicators attach only to tracer items and never to a numbered question. Turning off one grid question in CSPro removes its whole item list and every indicator on it. Any proposed cut must be tested against the question-indicator map at item level.
Skip logic that empties blocks. In Module 4, the inpatient gate (Q2100) and the surgical gate (Q2113) remove case reviews, death reviews, adverse-event and post-operative infection monitoring. Most of the quality-monitoring content in section 8.3 is hospital-only, so primary care facilities produce almost no data there. The mortality gate (Q4200) removes cause-of-death and ICD mortality coding for any facility with no deaths in the past 12 months, which changes the denominator of those indicators from all facilities to facilities with deaths. Q4110 skips general ICD morbidity coding whenever the facility has no cancer services.
5. Coverage of the PHC Measurement Framework
The 96 indicators of the WHO–UNICEF PHC Measurement Framework were taken as the specification, and each instrument was coded per indicator as F (produces it), P (partial or proxy), A (facility analogue that resembles the indicator without producing it), or blank.
Only 46 of the 96 indicators are measurable at facility level by design. The other 50 are national or subnational: policy existence, legislation, national health accounts, workforce accounts, civil registration, population surveys. No facility instrument can reach them.
| F | P | A | Not covered | |
|---|---|---|---|---|
| All 96 indicators | 26 | 24 | 12 | 34 |
| The 46 facility-measurable | 24 | 11 | 4 | 7 |
Against 96, coverage is 27%. Against 46, it is 52% produced and 76% at least partially covered. The instruments are close to their structural ceiling.
Per instrument:
| Instrument | F | P | A | Indicators touched |
|---|---|---|---|---|
| HHFA Module 1 (building structure) | 0 | 7 | 0 | 7 |
| HHFA Module 2 (service readiness) | 10 | 9 | 1 | 20 |
| HHFA Module 3 (record review) | 1 | 3 | 0 | 4 |
| HHFA Module 4 (management and finance) | 11 | 22 | 13 | 46 |
| EmONC Angola | 12 | 22 | 3 | 37 |
Module 4 touches the most indicators and carries the most analogues. It alone produces indicators 35, 44, 52, 54, 55, 56, 60, 72 and 85 (patient records, electronic records, management capability, supportive supervision, facility budgets, community linkages, quality systems, discharges, bed occupancy). Without it, the framework’s Models of Care and Systems for Quality domains have no facility source.
The analogue trap: Module 4 Q1303–Q1304 asks whether the facility’s plan names quality as a priority. That is not indicator 4, which is the existence of a national quality policy. Reporting analogues as coverage is the most likely technical error in a survey report.
Domains with no coverage: Governance (0 of 9) and Financing (0 of 8). These are not gaps to be closed by adding questions to a facility survey. They are the space the Health Financing Progress Matrix (HFPM) and the public financial management assessment occupy. This is the reason the three HIIP components are complementary rather than redundant.
Quality Care at 3 of 14 is the more uncomfortable result, because it is partly facility-measurable. Seven facility-level indicators are produced by no instrument: 81 (admissions for ambulatory care sensitive conditions), 78, 79, 80 (30-day case fatality, avoidable diabetes complications, readmissions), and 83, 87 (antipsychotic prescribing in over-65s, waiting time to elective surgery). The first four require coded inpatient records linked by a persistent patient identifier. Module 4 Q3600–Q3604 tells you whether that linkage is possible; it never yields the rate.
6. HHFA and EmONC are complements, not duplicates
EmONC outperforms the HHFA on nine indicators. Referral protocols (49): EmONC asks whether written criteria were observed, whether staff were trained, whether the register is in routine use and whether staff call ahead; the HHFA asks seven questions in section 10.3. Patient-reported experience (74): the 30-item person-centred maternity care exit interview is the only patient voice in the whole set; the HHFA asks whether a client-opinion system exists. Multidisciplinary team delivery (53): EmONC maps which of 13 cadres performs each of 38 functions; the HHFA cannot produce that. EmONC also produces caesarean rate, institutional mortality by cause, and bed density by obstetric, labour, delivery, postpartum and neonatal intensive care. It is the only instrument touching perceived barriers to access (63), with the caveat that exit interviews reach only those who arrived.
EmONC carries no question on individual patient charts. The HHFA holds that content (Module 4 section 13.2, Q3600–Q3624, plus three service-specific questions in Module 2 and 21 sampling-audit fields in Module 3).
The crosswalk is biased against the HHFA on obstetric and newborn availability because the Availability Core and Core+Additional questionnaires, which hold that content, were not extracted.
7. Empanelment is absent everywhere
Indicator 47 of the PHC Measurement Framework (empanelment, meaning patient registration lists) appears in none of the five instruments. HHFA Module 4 measures catchment area, which is a geographic concept, not a list of registered persons. The framework itself sources indicator 47 to key-informant review at national or subnational level. A single-question addition at facility level has been proposed and not yet implemented.
8. What a national report actually publishes: Guiné-Bissau 2023
The Guiné-Bissau HHFA 2023 national report (250 pages, Ministry of Health and WHO) was compared with the four questionnaires, using the level-3 subsection as the unit (218 subsections, 1,624 numbered questions; 1,567 without the repeated cover page).
- The report published 128 of the 224 analytical tables listed in its own index. The other 96 (43%) are listed and marked “NA”.
- The cut is clean: tables 1–137 (availability and clinical readiness) were published; tables 138–180 and 186–238 (surgery, emergency, special needs, laboratory, and the whole management, finance and information systems block) were not.
- Module 3 was not applied. The report’s limitations section says so explicitly.
- Module 1 building structure data was collected but not reported: no table, no figure, no mention of accessibility for persons with mobility limitations.
- From Module 4, only governance, support services and community linkages reached the report. Budget, expenditure, user fees, insurance, audit and health information systems were left out entirely.
The parts of the HHFA most relevant to a PFM and health financing reading are the parts most likely to be dropped between collection and publication. For HIIP, the report outline and tabulation plan should be agreed before data collection, and the Module 4 Section 12 tables should be named in it.
9. Two standing caveats
Version. Modules 1, 2 and 4 were extracted from the December 2021 generation; Module 3 from June 2023 version 2.1; the process-measurement search used the June 2023 combined core questionnaire. Section and question numbers differ between generations and between stand-alone and combined formats. Any number quoted in a country document must be checked against the file the country will actually use.
Paper versus CSPro. The paper questionnaire is the adaptation document. The executed instrument is the CSPro application, which manages skips and question activation automatically and may suppress duplicated fields. Indicator counts from the paper file are an upper bound on what a given country configuration will collect.
10. Implications for HIIP
- The HHFA supplies the supply-side picture: what facilities have and what systems exist. The HFPM and the PFM assessment supply the financing and budget-execution picture. The costed PHC investment plan is where they are combined. Neither survey substitutes for the other.
- Module 4 Section 12 (Q3008, Q3015 on budget received) is the one place the two assessments touch the same fact from opposite ends. Those questions should be protected in country adaptation and their consent-gate missingness reported.
- Process and outcome must come from elsewhere: Module 3 if applied, the routine health information system, and a separate data quality review with the WHO DQA toolkit.
- Before agreeing any cut to the questionnaire, test it at item level against the indicator map.
- Decide the tabulation plan first, name the tables, and hold the report to it.
Open items
- Extract the Availability Core and Core+Additional questionnaires to complete Module 1 and remove the obstetric bias in the EmONC crosswalk.
- Implement the single-question facility-level addition for indicator 47.
- Check all question numbers cited here against the June 2023 stand-alone files.
Sources
Instruments and frameworks:
- HHFA questionnaires (June 2023). WHO
- HHFA Comprehensive guide. WHO · SharePoint
- PHC Measurement Framework and Indicators (WHO–UNICEF, 2022). SharePoint
- Operational Framework for Primary Health Care (WHO–UNICEF, 2020). SharePoint
- Guiné-Bissau, Relatório do HHFA 2023 (MINSAP/WHO). SharePoint
- HHFA global archive of national reports. WHO
Instruments in spreadsheet form:
- HHFA module workbooks (extraction). Module 1 SharePoint · Module 2 SharePoint · Module 3 SharePoint · Module 4 SharePoint
- Annex1_EmONC_Angola_Questionnaires.xlsx (source annex). SharePoint
- EmONC_Angola_questionnaires_HHFA_format.xlsx (EmONC reformatted to the HHFA extraction schema; 619 questions, 884 sub-items). SharePoint · Chat
- WHO_UNICEF_PHC_Measurement_Framework_Indicators.xlsx (96 indicators). SharePoint
Analytical outputs produced in this project. Downloadable from the chat where each was built; SharePoint links to be added once uploaded.