Patient-Reported Outcomes

Interpretation of PRO Results

It is important to understand what domain the PROM is measuring and how it is scored in order to be able to utilize it appropriately, especially if your institution will share the results with the patients. Additionally, it essential to be aware of how clinically significant change in scores between patients or within patients over time is defined and analyzed.

In PRO data reporting, minimally important differences (MIDs) represent a specific approach to measure clinical significance. They are defined as “the smallest difference in score in the outcome of interest that informed patients or informed proxies perceive as important.” Although this statistical method may be necessary to utilize when reporting scores, it may vary by population and by context and some PROMs may already have known thresholds or meaningful cut points. Further review of these statistical concepts may be necessary by participants as they utilize PRO data. AJRR's goal was to make participants aware of the issues of PRO data collection and usage. As AJRR’s PRO dataset develops, AJRR will be able to provide more insight on how to report and interpret PROM scores.

Therefore, the three main concepts to think about when interpreting PROM results are:

1. Knowing the domain that is being measured and labeling it correctly (e.g., degree pain interferes with physical activity, global physical health)
2. Understanding the score and the meaning of the score (e.g., provide the mean and indicate the reference population, describe the direction of score)
3. In addition to MIDs, there may also be known thresholds or meaningful cut points (e.g., mild/moderate/severe impairment)

How do I Develop a Workflow for Data Capture

Once you have decided the purpose of PRO data collection, which PROM(s) to collect, and your timeline for data capture, you and your colleagues should develop the protocol for data collection. If electronic data capture is an option for you, then you will need to assess whether the AJRR platform or another system meets your needs. Again, questions to discuss:

 What does our current EHR platform provide? Is the infrastructure sufficient for data capture? Do our clinics or affiliated practice groups share the same EHR?
 Do the AJRR tools help us meet those aims?
 What do other orthopaedic charting vendors offer in regards to PROs?
 What are your patient population characteristics? Will they have access to complete the forms electronically?
 Are PROMs in paper form suitable for our aims for data collection? Do we have a process for integrating that information into AJRR?
 What follow-up time points will be the most manageable to capture in clinic? Or do we only want to collect off site via a secure patient portal? Or both?
 Will there be adequate time set aside for clinical staff to manage the surveys, track patient enrollment, monitor data completion quality and rates, and follow up on problems?
 How do we train our staff on the data collection processes?
 How do we frame the PRO discussion with patients and when? At time of first visit, after TJR procedure has been scheduled, at the time of the pre-operative TJR class?
 Will the care team want to review the PROM scores with patients at clinic visit?

AJRR has developed two simple workflow diagrams (one for paper format collection and one for electronic capture) to help visualize the whole process from data collection to utilization of data. Although most institutions would prefer to collect PROMs via electronic surveys, this may not be an option for some participants. Understanding your patient flow from pre-op to post-op follow-up will help guide your institution on how to collect PROMs at defined time points. As shown by the bold arrows in the following diagrams, the most streamlined method for real time data submission and access is with the use of the AJRR platform.

AJRR Digital

7 Things to Consider When Choosing PROM Timelines

Measurement timeline

PROMs guidelines from groups such as the International Consortium for Health Outcome Measurement (ICHOM) have recommended pre-operative (baseline) and one-year follow-up as appropriate time points for data collection to provide meaningful data for comparing outcomes across providers. The CJR Final Rule also advises that post-operative surveys be collected between 270 and 365 days post-surgery.

Each time point will have a window for data collection that follows CMS’s accepted time-point intervals. AJRR’s platform will allow for other time points (e.g., three-month, six-month, etc.) to be submitted and stored in AJRR’s database should you wish. However, national benchmarks will only be reported for pre-operative and one-year outcomes.

Further questions to discuss with your colleagues:

 How often would we like to collect a PROM post-surgery?
 At what time points do our patients consistently come back for a follow-up visit? Is this the best time to capture the PROMs? (If patient doesn’t complete via Internet prior to visit, we have a chance to collect at the clinic visit or provide a reminder to complete it after the visit.)
 For what time point has our chosen PROM been validated to detect change (six months vs. one year)?
 What time frame is needed to address our primary aims for PRO data collection (e.g., data at each clinic visit if used to inform clinical encounter vs. pre-operative and one year data to meet minimum reporting requirements for payers)?
 How many time points for data collection can be managed by our staff?
 How frequently are patients willing to complete surveys? Should a random subgroup be utilized for more frequent assessment (this may be an option for research driven initiatives)?
 What is our hospital status regarding federal quality initiatives? Are we in one of the CJR geographic areas? Are we in the Bundled Payments for Care Improvements (BPCI) initiative?

What PRO Measures are Appropriate for Me

Determine the Appropriate PRO Measure

Your reasons for implementing a PRO program will direct your team to a certain instrument(s) that will allow you to meet your objectives. For example, if your institution has decided to collect PROMs for research initiatives, you may choose an instrument that measures multiple domains, like the HOOS and KOOS. These instruments are joint-specific instruments that have five subscales measuring function, pain, hip/knee symptoms, sports and recreation (high level activities of daily living) and quality of life, which would provide a depth of data to address many research hypotheses. However, if you are focused on measuring patients’ outcomes as part of your clinical care, a shorter instrument focused on physical function or critical symptoms (e.g., pain) will suffice as long as it is able to detect change. Other factors to review when assessing PRO instruments are the levels of validity, reliability, and responsiveness of the PROM. Each PROM that is available through AJRR’s platform has been validated, however it will be up to your institution to determine which instrument(s) meets the standards or needs for your PRO program. You will also need to consider the cost to use a specific PROM. Your hospital and/or practice group will be responsible for any PROM licensing cost if you do not use AJRR’s PRO platform to collect a PROM that requires a license. However, only three forms (SF-12, EQ-5D, and WOMAC) require licenses.

For psychometric and validation data pertaining to each of these measures, please visit the following sites:

Health-related quality of life measures 

 Disease-specific measures 

   Other measures 

A major factor to address in determining which PROM to utilize is patient and clinician burden. Although longer instruments may provide a more comprehensive or sensitive measure, compliance may be low due to the time it takes a patient to complete the survey (or for a clinician to administer the survey) and may be difficult to analyze if all items are not answered. AJRR, in conjunction with the AAHKS Patient-Reported Outcome Summit, has identified patient burden to complete long instruments as a major barrier in data collection on a national level. AJRR’s recommended PROMs are short instruments (12 items or less) that have been validated to detect measurable change over time. Longer instruments may require clinicians or clinic staff taking a more active role in monitoring and checking completion of the forms. With electronic capture, short forms should increase compliance rates and reduce missing data points.

Comparability across populations is another factor to discuss. Instruments that are commonly used and have been developed for across population comparison are useful when comparing diverse communities. AJRR’s HRQL measures have been validated for this process. Also, a cross walk has been developed for the VR-12 and PROMIS-10 Global Health to facilitate linkage between these instruments.12 In regards to joint-specific measures, outcome measurement groups are in the process of facilitating linkage between these types of instruments as well as among other commonly utilized PROMs.

When considering a PROM, you should take into account your patient population. Age, education, socioeconomic status, and patient literacy (reading, health, and computer) can all be potential barriers for patients to participate with the data collection protocol. Language is another consideration – do you need a measure that has been validated and translated into other languages? At this time, AJRR platform only provides English versions of the accepted PROMs. If AJRR participants begin to request non-English versions, AJRR will review and assess whether each PROM has a valid non-English version appropriate for use and analysis.

Finally, although total joint PRO surveys may not be considered controversial in content (i.e., they do not ask personal, sensitive questions), they may make a patient feel insecure about their functional capabilities or patients may not be sure how to answer a question if they do not perform a specific activity described in the survey (e.g., they do not engage in vigorous activities).  Evaluating these types of factors will be essential when discussing the type and method to collect PROMs and how you will frame this additional assessment to your patients, especially if your organization plans to share results with patients. For example, including explicit language indicating the purpose of the survey such as: a) we are asking you to complete this survey because we want to know more about how you are doing, b) indicate what the health care team plans on doing with this information (e.g., share only with care team), c) there are no right or wrong answers – some things may be more or less important or relevant to you. Please try to answer all questions.

There are many factors to consider when choosing a PROM to meet your PRO objectives. Again, this link provides additional resources from leading outcome experts that offer more in-depth discussion on the above topics. However, as in each section, below are basic questions to start the discussion.

What does the measure assess – quality of life or function?
   -Do we want to understand both or just one aspect?
What is the associated patient burden?
   -What is the length of form? How easy/hard is the form to complete?
What is the associated staff burden (full-time employee time)? Who will be assisting patients with forms? Whose job is it to follow up with patients?
What is the cost (licensing fee) to use the form?
Do we have non-English speakers in our patient population? Will we need translated versions of our preferred PROM(s)?

How do I start a Patient-reported Outcomes Program at my Institution

Developing a PRO program may be an involved process for your institution. It may take months to design and implement, therefore taking time to discuss each category below will hopefully streamline the process and guide you to a successful program.

Define PRO team

Most likely, within your institution a core group of colleagues (i.e., orthopaedic group, quality department, or hospital administration) has decided that including PROs in your practice of care for total joint arthroplasty may provide important information to improve patient outcomes. Beyond this core group, you may need to bring in other disciplines to help drive this initiative and development. Below is a list of potential key stakeholders who may assist in developing and capturing PROMs for your institution:

• Orthopaedic Department
• Orthopaedic Practice Groups and Clinics (not only physicians and nurses, but front desk staff may need to be involved)
• Quality Department
• Information Technology
• Orthopaedic Service Line
• Research
• Rehabilitation
• Hospital Administration
• Patient Advocate/Patient Representative
• Institutional Review Board (guidance/review)

As you begin to design your protocol for data collection, the appropriate stakeholders will become apparent and engaging these groups will help build collaboration and buy-in on why a PRO program is beneficial and sustainable.

Define your institution’s goals for implementing a PRO program

If you are interested in a PRO program, you and your colleagues will need to determine the reasons you would like to launch a PRO program.

You may consider asking yourselves questions such as:

 Are we launching a research initiative with specific aims?
 Are we seeking comparative benchmarks to our peers?
 Are we wanting to quantify our outcomes from our patients’ perspective?
 Do we want to measure if patients have improved function or reduced pain?
 Are there other areas of self-reported health that are critical to assess?
 Do we want to screen for referrals for pain management?
 Do we want to screen for patients at risk for poor outcomes?
 Do we want to measure whether our patients’ overall health has improved?
 Are we wanting patient provided data to be utilized during the clinical encounter?
 Do we want to utilize PROs for evaluating quality improvement initiatives?
 Do we want to allow for analyses to compare procedures or surgical protocols?
 What are the requirements of the payer-specific program for which we are hoping to qualify?

Determine the appropriate PRO measure

Your reasons for implementing a PRO program will direct your team to a certain instrument(s) that will allow you to meet your objectives. For example, if your institution has decided to collect PROMs for research initiatives, you may choose an instrument that measures multiple domains, like the HOOS and KOOS. These instruments are joint-specific instruments that have five subscales measuring function, pain, hip/knee symptoms, sports and recreation (high level activities of daily living) and quality of life, which would provide a depth of data to address many research hypotheses. However, if you are focused on measuring patients’ outcomes as part of your clinical care, a shorter instrument focused on physical function or critical symptoms (e.g., pain) will suffice as long as it is able to detect change. Other factors to review when assessing PRO instruments are the levels of validity, reliability, and responsiveness of the PROM. Each PROM that is available through AJRR’s platform has been validated, however it will be up to your institution to determine which instrument(s) meets the standards or needs for your PRO program. You will also need to consider the cost to use a specific PROM. Your hospital and/or practice group will be responsible for any PROM licensing cost if you do not use AJRR’s PRO platform to collect a PROM that requires a license. However, only three forms (SF-12, EQ-5D, and WOMAC) require licenses.

For more information about PRO measures, visit here.

What are My Options to Submit PROMs to AJRR

You have three options to choose from to submit PROMs to the AJRR:

1. If a participant utilizes the AJRR RegistryInsights platform to collect PROMs, all PROM data is automatically saved once the patient completes the survey and submits their responses. Each item response and final scores are saved in real time so that clinicians are able to access the data on-demand. Hence, results may be used to facilitate discussion during a patient encounter. The system also provides reports for tracking compliance, completed PRO scores for further analysis, and dashboards for recommended PROMs. AJRR will provide a Level III PRO platform User Guide and also provide training webinars for new users.

2. If a participant already has an internal system to collect PROMs either electronically or by paper, they can submit PROM scores via a .csv or .xls file to AJRR’s SFTP server. A PROM data specification file will be provided to participants so that data meet AJRR standards for submission. If hospitals are also subscribing to the AJRR RegistryInsights platform, they will have access to the national dashboards and be able to pull PRO reports for further analysis. Additionally, if a subscribing participant is collecting PROMs via paper administration, the system allows for manual data entry of form responses.

3. If a participant employs an orthopaedic charting or other PROMs vendor to capture PROMs data and they are part of the AJRR Authorized Vendor program, the vendor is able to submit data directly to AJRR on behalf of a participating institution. See a full list of those organizations who are part of the AJRR Authorized Vendor program.

What PRO Measures Do AJRR Collect

The table below lists all the patient-reported outcome measures (PROMs) that AJRR will accept and store in the Registry. AJRR participants have the choice to collect the PROMs that best fit their needs, however PRO national benchmarks will only be available for AJRR’s recommended PROMs (as shown in gold in the table).

 AJRR offers HRQL measures and joint-specific measures. Each type of measure assesses different domains of health from the patient perspective. An HRQL yields a global summary of well-being, which can be further delineated into mental and physical health, but is not limited to these domains. For example, some measures, like the SF-36, can also be scored into more specific domains. Beyond the SF-36’s Physical and Mental Summary Component Scores, it has eight subscales (physical functioning, role – physical, bodily pain, general health, vitality, social functioning, role – emotional, and mental health). A disease-specific measure has been designed to address a specific health condition. The majority of disease-specific (or joint-specific) PROMs utilized in orthopaedics were primarily developed for osteoarthritis or rheumatoid arthritis to evaluate treatments for these diseases and to be more sensitive to specific symptoms of these diseases. 6-10 Joint-specific PRO surveys may measure a single domain or multiple domains, such as functional status (ability to perform specific activities), pain, symptoms, or patient experience (i.e., patient satisfaction or patient expectation), summaries of which can be found in the AAOS Registry Program Patient-Reported Outcome Measures Guide, Table 2 on page 10. These measures were developed for each domain subscale to be reported as an individual score for the PROM.

What Does AJRR Provide for a PRO System

There are two ways AJRR can help you. One is through a patient-reported outcome (PRO) platform that is inlcuded with your license subscription and the other is offering patient-reported outcome measure (PROM) guidance.

RegistryInsights PRO module

One of AJRR’s goals is to provide the orthopaedic community with national comparative PRO data. To assist AJRR facilities in PRO data capture and deliver a service to store and have on-demand access to the data, AJRR developed a PRO platform within our RegistryInsights module which is available to subscribing partipants as part of their licensing fee. The platform has many features for clinical staff to access their patient data while having the ability to manage and assign PRO surveys electronically via a secure assessment tool. AJRR’s secure assessment tool allows patients to access their surveys by means of the Internet at home or in the clinic to complete the surveys in a convenient manner. 

Please note: Currently, AJRR’s PRO platform can be accessed using a PC or tablets and only includes English versions of PRO measures (PROMs). In the future, AJRR may offer non-English versions of measures (if participants express the need for other language surveys).

AJRR’s data system also has the capacity to accept final PROM scores for participating hospitals and practice groups who are collecting PROMs through another method (e.g., PRO collection by their electronic health records (EHR) system or an orthopaedic charting vendor). For example, Epic and other EHR vendors offer PRO platforms in their systems for those who wish to utilize a platform tailored to their individual needs. AJRR collaborates with a number of these vendors to facilitate seamless transfer of PRO data. Check out this licnk for more information on our vendor partners.

If a participant would like to use the AJRR platform for PROM data collection, the participant will need to subscribe to the AJRR RegistryInsights platform. AJRR will provide a Level III PRO platform User Guide and also provide training webinars for new users. The participant will be responsible for managing PRO data collection at their site(s).

PROM Guidance

Beyond providing a system to capture PRO measures or accessing PRO dashboards, AJRR provides guidance regarding what PROMs your organization may wish to collect, especially for hospitals and practice groups just starting a PRO program. As the National Registry that works closely with the orthopaedic societies and associations, hospitals and surgeons are looking to AJRR for guidance regarding PRO collection. AJRR’s Data Committee, in collaboration with orthopaedic specialty organizations, identified the specific measures that AJRR recommends for national benchmarks.

The recommendation includes utilization of both an HRQL measure (either VR-12 or PROMIS-10 Global Health) and a joint-specific measure (HOOS, JR. and KOOS, JR.). The decision to recommend the instruments was based on AJRR participation in the American Association of Hip and Knee Surgeons’ (AAHKS) Patient-Reported Outcome Summit for Total Joint Arthroplasty that convened in August 2015. Representatives from orthopaedic organizations (AAHKS, AJRR, American Academy of Orthopaedic Surgeons (AAOS), The Hip Society, and The Knee Society), CMS, Yale-New Haven Health Services Corporation Center for Outcomes Research and Evaluation (YNHHSC/CORE), private payers, and other stakeholders participated in the Summit. The Summit’s goal was to obtain a consensus regarding the PRO suitable for total hip and knee arthroplasty performance measures. Additionally, our recommendations are also consistent with the CJR Final Rule.

Although AJRR has recommended PROMs for national benchmarking, AJRR provides a large list of PROMs for hospitals interested in collecting measures other than AJRR’s recommendations. AJRR understands institutions may have in place a long-standing PRO data collection process with specific PRO instruments. As these groups may wish to continue utilizing their preferred PROM, AJRR will provide a repository to warehouse all levels of orthopaedic data in a centralized system. Additionally, a hospital or practice may prefer to use another measure in order to conduct comprehensive analysis at a more granular level. RegistryInsights will allow for aggregated site specific reports detailing the patients and summary results for each PROM supported on the AJRR system, even though national benchmarks and dashboards will not be available for these other measures.

Can I Start a PRO Program at my ASC

Currently, AJRR enrolls mainly hospitals into the Registry because total joint arthroplasties are primarily performed in hospital settings, and therefore data regarding the procedure is collected and warehoused in the hospital’s EHR system or other hospital systems (e.g., operating room or service line). As total joint arthroplasty procedures are increasingly performed in other settings, such as ambulatory surgical centers (ASCs), the Registry has begun to enroll ASCs. AJRR also enrolls private practice groups, which will provide opportunities to capture pre- and post-op data elements not contained in hospitals’ EHRs. AJRR assumes that most PRO data collection will happen at the clinic level, which could be owned by the hospital or a private practice group. Hence, it will be important for practice groups and hospitals or ASCs to have dialogue regarding this type of initiative and how they may collaborate on this effort. 

This dialogue is critical to successful implementation of the CJR bundled payment initiative as well. All providers (i.e., hospitals, ASCs, and surgeons) who participate in the Registry will have access to their own specific outcome data and de-identified, aggregate national benchmarks.

Why Should I Include a PRO Program at My Institution

Patient-reported outcomes (PROs) can be valuable tools in guiding physicians and patients on understanding a patient’s health status, in the decision-making process regarding patient care – for example, severity of joint disease based on patient-reported outcome measures (PROMs) may be a good indicator for surgery, and for evaluating the effectiveness of quality improvement initiatives. In short, there are three major reasons why providers may be interested in collecting PROs:

1. Expanding the criteria for how to evaluate care by including outcomes based on patient’s viewpoint with other clinical measures

As the California Joint Replacement Registry (CJRR), an organization that recently merged with AJRR, was developing their PRO platform in 2010, they discussed the rationale for capturing PROs in this manner:

“Longitudinally tracking patient assessment of pain and function can provide insights into the effectiveness of hip and knee arthroplasty across a much broader patient population than the relatively small number of patients that suffer implant failures and require surgery. Perhaps most importantly, PRO data reflect the patient’s perspective on the outcome of the surgery – described as ‘the truest end result of our care as physicians,’ by one orthopedic surgeon.” 

PRO data provide meaningful information in conjunction with other clinical measures, and therefore, provide the opportunity to have a fuller picture of the impact of care and may be used by the clinician during patient encounters. These surveys can be considered an important step toward engaging patients in their own health care and informing medical decision making.

2. Federal initiatives

PRO data will be critical as the Centers for Medicare & Medicaid Services (CMS) and other payers move toward defining quality measures to evaluate health care providers’ performance for value-based reimbursement of care. For example, CMS emphasizes the use of PROs by defining PRO requirements within their Medicare reimbursement programs such as the Comprehensive Care for Joint Replacement (CJR) model and the Physician Quality Reporting System (PQRS). Eligible professionals will need to include PRO measures in their course of clinical care and submit their results to meet these new standards for reimbursement without incurring penalties. Specifically, CMS intends to have 90% of payments tied to quality outcomes by 2018.

For the CJR initiative, hospitals are not required to submit PRO data, however PRO data will be linked to the hospital’s reconciliation payment. CMS is using a composite score methodology to link quality outcomes to payment. Specifically, a hospital’s score will be determined in part by performance and improvement on two quality measures: a) the THA/TKA Complications measure (NQF #1550) and b) the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) Survey (NQF #0166) as well as voluntary submission of THA/TKA of PRO and limited risk variable data. Those institutions voluntarily submitting PRO data will have to submit data on ≥50% or ≥50 eligible procedures. Successful submission of PRO and risk variable data will add two additional points to a participant hospital’s composite quality score. For more information, please view any of the documents posted under the “Additional Information” section of the CJR website at https://innovation.cms.gov/initiatives/cjr.

3. Comparative benchmarks

 AJRR provides hospitals and practice groups with PRO benchmarks to compare their results to the national experience. Knowing how surgeons compare with their peers, as well as against the nation can be very beneficial for practice improvement efforts. Comparative data may facilitate and provide evidence for the need for quality improvement work at both the hospital and surgeon level. Additionally, hospital and practice groups can use that data to publicly report their own results, should they wish. Having the data from a robust National Registry will enable institutions to make informed decisions based on clinical facts and figures.  A national database will be able to analyze and report national total hip and knee arthroplasty measures and offer opportunities for further investigation at both local and national levels. AJRR understands that researchers, institutions, and organizations may want to have opportunities to access the data in the Registry to conduct further analyses to address specific hypotheses, such as replacement outcomes and implant performance, beyond the national benchmarks.

What is the difference between a PRO and a PROM

Definition: Patient-Reported Outcome (PRO)

A PRO is defined as any information on the outcomes of health care obtained directly from patients without modification by clinicians or other health care professionals.

Definition: Patient-Reported Outcome Measure (PROM)

A PROM is a survey that captures a patient’s self-assessment of health including status (mental, physical), function, symptoms, and health related quality of life (HRQL). It is an important research tool used in PRO programs.

Both PROs and PROMs are included in AJRR's Level III data specifications. Click here to see AJRR's full data specifications list.

The Register

Sign Up for AJRR's Particpant E-newsletter Registry Insider

Subscribe

AAOS AJRR Sept2018 yellow background

The Register

Sign Up for AJRR's Particpant E-newsletter Registry Insider

Subscribe