Institutional approaches to evaluative practice

Gary Allen, Mark Israel and Colin Thomson

In 2001, the NHMRC published its policy document When does Quality Assurance in Health Care Require Independent Ethical Review? The document was rescinded in 2007 and is no longer available since the update to the NHMRC website in 2018. Several changes led to the rescinding of the 2001 policy document:

1. 1. The release of the 2007 edition of the National Statement provided a mechanism for exempting work with de-identified data where the work involved no more than negligible risk.
   2. The 2007 edition of the National Statement established clear criteria for determining whether research could be reviewed outside of an HREC (e.g. a project cannot involve any greater than a low risk of harm and cannot involve matters the National Statement specifies as requiring HREC review). [1]

3. The ‘pressure to publish’ has meant a significant amount of work that used to be conducted as an evaluation or for quality assurance is now being submitted for publication to refereed journals.

4. 

   Stakeholders and funders require services and expenditure to be based on robust evidence and analysis.

As a result,the distinctions between research, evaluative practice and quality assurance have become blurred to the point of no longer being helpful and the research ethics review mechanisms for exemption, proportional review and mandated HREC review in specified circumstances, might be sufficient for the appropriate handling of evaluative practice.

Nevertheless, submissions to the NHMRC’s Australian Health Ethics Committee prompted the release in 2014 of Ethical Considerations in Quality Assurance and Evaluation Activities.

Ethical Considerations in Quality Assurance and Evaluation Activities describes whether quality assurance and evaluation work requires research ethics review and the most appropriate way to approach that review. It:

1. discusses how such activities can be conducted over a spectrum of work, which may change over time and the divide between evaluative practice and human research can be porous,
2. concedes HREC review is often not helpful when the primary purpose of the activity is to inform and improve an organisation’s practice (rather than contribute to the wider body of knowledge),
3. describes four matters [2] to which the design and conduct of evaluative practice must adhere,
4. describes four criteria to identify where oversight but not review is required, [3]
5. directs institutions to establish policies with regard to these matters, [4]  and
6. at provision (e) describes circumstances where consideration of the need for review is required [5] and, where it is, offers guidance at provision (f) to appropriate levels of review.

The 2014 policy document therefore provides criteria to determine whether an institution’s evaluative practice/quality assurance activity requires:

1. only administrative consideration [6] to confirm that the institution’s policies relating to the use of those data to assess its services are being/have been adhered to in the design of the work,
2. a special research review process [7] within the organisation to test whether HREC review is required and to confirm the institution’s policies relating to evaluative practice have been adhered to in the design of the work, or
3. ethics review by an HREC or another review body.

Given many staff will want to publish the outcome of evaluations and there will be academic interest in matters related to evaluations/quality assurance, the institution’s arrangements must not preclude academic publication.

The following approach is recommended:

1. The institution needs to know what evaluative practice is being conducted and so must have a mandatory process for review that is similar to a scientific research review committee, but is also an institutional policy process (its review feedback should provide advice to facilitate the planned activity and relate to how to conduct the work ethically and successfully).
2. There is a mechanism for the research review of evaluative practice/quality assurance, with some such reviews being delegated to a special panel or administrative review.
3. The review pathway being initially determined by the responses to a sequence of yes/no questions.
4. The responses to the questions being reviewed by an ethics officer(in the case of clinical audits this the person conducts the initial assessment will need relevant clinical expertise) who confirms the review pathway.
5. Every 12 months a small proportion of evaluative projects are randomly selected for audit, not to revisit the decision but to confirm the process is working correctly.
6. Policy and guidance material informs the ethical design, conduct, reporting and publication of evaluative practice/quality assurance, as well as its ethics review.
7. There are briefing sheets (two double-sided A4 pages)
8. For researchers: summarising the institution’s arrangements for evaluative practice/quality assurance (including a summary of their responsibilities).
9. For heads of school/department: discussing their role in the review of services/procedures/teaching and learning in their area.
10. For editors/publishers: explaining the institution’s arrangements, for provision by researchers if they are asked to provide a copy of the HREC approval.
11. If the institution has a network of collegiate Research Ethics Advisers (see our earlier blog post about REAs) some advisers should be experienced in the conduct of evaluative practice.

Available for USD10/month to subscribers, AHRECS has created notes to inform the in-house development of A, B and C. A USD15/month subscription provides access to our growing library of materials.

AHRECS would be delighted to discuss an arrangement where we provide feedback on the materials you produce in-house or us producing the materials for you.

Email us at Pa****@****cs.com with any question about our Patreon page or becoming a patron, or Ev********@****cs.com to discuss how we could assist you with regard to the ethics of evaluative practice.

[1] Because of design factors such as the deception of participants, and participant factors such as research with Aboriginal and Torres Strait Islander people.

[2] What really matters is that:

• participants in QA/evaluation are afforded appropriate protections and respect
• QA and/or evaluation is undertaken to generate outcomes that are used to assess and/or improve service provision
• those who undertake QA and/or evaluation adhere to relevant ethical principles and State, Territory and Commonwealth legislation
• organisations provide guidance and oversight to ensure activities are conducted ethically including a pathway to address concerns.

[3] In many situations, oversight of the activity is required, but an ethics review is not necessary. These include situations where:

• The data being collected and analysed is coincidental to standard operating procedures with standard equipment and/or protocols
• The data is being collected and analysed expressly for the purpose of maintaining standards or identifying areas for improvement in the environment from which the data was obtained;
• The data being collected and analysed is not linked to individuals; and
• None of the triggers for consideration of ethics review (listed below) are present.

[4]  ‘Organisations should develop policies on QA/evaluation which provide guidance for oversight of QA or evaluation activities. It is recommended that such policies address the following issues’.

[5] Triggers for consideration of ethics review include:

• Where the activity potentially infringes the privacy or professional reputation of participants, providers or organisations.
• Secondary use of data – using data or analysis from QA or evaluation activities for another purpose.
• Gathering information about the participant beyond that which is collected routinely. Information may include biospecimens or additional investigations.
• Testing of non-standard (innovative) protocols or equipment.
• Comparison of cohorts.

[6] Preferably prior to data collection commencing, but in the case of data collected prior to the adoption of this document, the check must occur prior to any use of the collected data.

[7] A Panel should be created for the purpose of conducting these reviews. Even though there are undeniable advantages from this panel being comprised in full (or at least primarily) of members of the institution’s HREC, it must be stressed to Panel members there are important and valid differences between academic research and evaluative practice. For this reason, a proportion of the Panel members should not be drawn from the HREC and should be experienced in the conduct of evaluative practice.

[colored_box]

 

Contributors
Dr Gary Allen, Senior consultant AHRECS | Profile | ga********@****cs.com
Dr Mark Israel, Senior consultant AHRECS | Profile | ma*********@****cs.com
Prof., Colin Thomson AM, Senior consultant AHRECS | Profile | co***********@****cs.com

This post may be cited as:
Allen, G., Israel, M. and Thomson, C. (18 March 2019) Institutional approaches to evaluative practice. Research Ethics Monthly. Retrieved from: https://ahrecs.com/human-research-ethics/institutional-approaches-to-evaluative-practice