Feedback on a radiology report, from radiologists and referring clinicians

Radiology Report Feedback lets Health NZ radiologists and referring clinicians give feedback on a radiology report: a learning opportunity, or a great call. It covers reports from external teleradiology providers and reports written in Health NZ. Each learning opportunity is checked, reviewed, passed to the provider or the local department, answered and closed, and the person who raised it is told the outcome.

Part of Radiology LIFT In beta testing, synthetic data only No account needed for referring clinicians

Open Radiology Report Feedback (beta)

What it is

It is feedback between colleagues, for learning. A radiologist or a referring clinician reads a report that prompts a comment and records it in about a minute. The tool makes sure each comment reaches someone who can act on it, and that the person who raised it hears what happened.

Learning opportunity

Something in a report that could have been better: a missed finding, a misinterpretation, a report that was unclear or did not answer the clinical question, a critical finding that was not passed on. It is checked and reviewed, then goes to the provider or the local department for a response.

Great call

A report worth acknowledging: it picked up an important finding, was clear and actionable, answered the clinical question, compared carefully with prior imaging or came back quickly. A great call is sent straight to the provider or the local department with the sender's name. It is never reviewed.

Who gives feedback

  • Health NZ radiologists, signed in with their My HNZ Radiology account. A radiologist can give feedback on a report from an external provider or from Health NZ.
  • Referring clinicians, from a ward or clinic. They need no account and give a Health NZ work email when they save or send.

What it does not cover

  • Patient harm still goes through the local incident system. The form says so when harm is recorded and asks for the incident reference.
  • It does not rank or discipline individual radiologists.
  • It is not a channel for complaints about a provider.
  • For a Health NZ report it adds to local feedback pathways and does not replace them.
beta.reportfeedback · Front page
The Radiology Report Feedback front
          page, titled Feedback on a Radiology Report, asking which are you, with two cards:
          I'm a Health NZ radiologist, sign in with your My HNZ Radiology account; and I'm a
          referring clinician, no account needed. A yellow banner says test data only.
Someone who is not signed in chooses radiologist or referring clinician. A radiologist already signed in at My HNZ Radiology goes straight to the form.

Giving feedback

Everyone starts on the same page, "Feedback on a Radiology Report", and chooses "Flag a learning opportunity" or "Acknowledge a great call". The form then asks only what the case needs, and only once an earlier answer makes it relevant.

Radiologists

Paste the report

Radiologists · signed in

A radiologist can paste the report instead of typing the details. The browser reads it and fills in what it is sure of: accession, provider, reporting radiologist, modality, body region and dates. A value it is less sure of is offered as a button to click, and anything it cannot find is marked so. Each filled value can be removed.

The pasted report stays in the browser and is never saved or sent. Only the details the radiologist keeps, any sentence they select from the report, and a one-way fingerprint of the signature block (used to recognise the provider next time) leave it.
beta.reportfeedback · Flag a learning opportunity
A synthetic report pasted on
            the left, marked Not saved or sent, with the accession, provider, radiologist,
            modality and dates highlighted in colour. On the right, What the report gave lists
            each field with the value found, each with a cross to remove it, and a button to
            continue with the eight confirmed details.
A synthetic report pasted in. Each value found is highlighted in the report and listed on the right in the same colour.

The questions that follow:

  • What went wrong, on a Report tab and a Service tab. More than one issue can be ticked.
  • For a missed finding or a misinterpretation, how serious it was, asked in plain words. For anything else, the impact on the patient.
  • The report details needed to sign: provider (searched by name, "Not sure" allowed), reporting radiologist ("Not known" allowed), modality, body region and exam date. Everything else is optional, on an Add context tab.

A "Your draft" panel shows the feedback as it will be sent and what is still needed. The radiologist can sign at once or save a draft, which is emailed to them as a link to finish later. If the study has already been flagged, the form shows the date it was flagged and offers to add a note to that case instead of opening a second one.

Clinicians

Four steps

Referring clinicians · no account needed

A referring clinician's form is walked through in four steps, with Continue and Go back between them:

  1. The report: the accession number, and who reported it (a Health NZ radiologist, an external teleradiology provider, or not sure).
  2. What's wrong (it's wrong, it's unclear, it doesn't answer what I asked, or something else in their own words) and the impact on the patient.
  3. Case complexity: general, subspecialty or uncertain, each with guidance.
  4. Tell us more, in free text.

A clinician is never asked to grade severity. That judgement belongs to the radiologist who checks the case.

beta.reportfeedback · Clinician form, step 2
Step 2 of 4 of the
            clinician's form. On the left, What's wrong with the report with It's unclear
            chosen, and Patient impact with Potential chosen, then Go back and Continue to case
            complexity buttons. On the right, the Your draft panel lists the answers so far, with
            Save draft and Cancel.
Step 2 of the clinician's form, with the draft so far on the right.
Everyone

A great call

Radiologists and clinicians

A great call has its own short form: the accession, who reported it, what made it great (any of the five reasons above, ticked), the provider and reporting radiologist for an external report, and an optional note. Signing closes it at once.

What happens next

Every signed learning opportunity ends with a recorded outcome, and the person who raised it is told what it was.

1

Checked

A clinician's learning opportunity is first checked by a radiologist reviewer. It either joins the review queue, or is closed with a short reason the clinician can see. A radiologist's goes straight to the queue.

2

Reviewed

A reviewer for the case's region takes it on. They record the agreed issue, severity and where the problem started, and fill in details the person who raised it did not know. The original answers are kept beside the reviewer's. Nobody reviews a case they raised. If the problem started before the report (missing priors, thin clinical details, the wrong protocol), the case closes there and is counted with other upstream causes.

3

Sent on

For an external report, the reviewer raises it with the provider through the existing outsourcing channels and marks it sent. For a Health NZ report, it goes to the region's local department contact. The provider is never told who raised it.

4

Answered and closed

The reviewer records the response and closes the case with an outcome. The person who raised it gets an email with the outcome and a link to the case, and a monthly summary of what happened to their cases.

beta.reportfeedback · Review
A reviewer's case page titled
          Learning opportunity: CT chest, in review with 25 days left to close. On the left,
          what was flagged: accession, issue, severity, provider, modality, body region and
          exam date, with a note not to pass the submitter's name to the provider. On the right,
          What's next, with the agreed issue, agreed severity and where it started.
A synthetic case in review: what was flagged on the left, the reviewer's agreed values on the right.

Time limits

Each stage has a limit, counted in working days with New Zealand public holidays left out. A case past its limit is raised with the reviewer, then with the region's Regional Service Delivery Lead.

StageLimit
A clinician's case checked5 working days
Review started5 working days
Response from the provider or department10 working days
Signed to closed30 calendar days
A learning opportunity that records actual patient harm, a critical finding not passed on, or the wrong patient or side, is sent to the region's Regional Service Delivery Leads as soon as a radiologist has signed or checked it. The email carries labels and dates and a link that needs a sign-in.
Until a reviewer takes a case on, the person who raised it can withdraw it. After that they can add an addendum, as with a report.

Learning and CPD

Two things follow from a closed case: a learning point can be shared with radiologists in every region, and taking part is recorded for each person.

Learning

Learning points

Signed-in radiologists, every region

When closing a case, the reviewer can share a short learning point in their own words. Shared points appear on a Learning page for radiologists in every region, newest first, each labelled only with the issue, modality and body region. Nothing on the page names a patient, provider, radiologist or hospital. The reviewer can change or withdraw a point later. Learning points are never put in an email; the monthly summary says how many are new.

beta.reportfeedback · Learning
The Learning page listing two
            synthetic learning points: Check the lung apices on every CT of the neck, labelled
            Missed finding, CT neck; and Say in the report who was phoned, and when, labelled
            Critical finding not communicated, MRI.
Two synthetic learning points, each with the kind of case it came from.
Record

My CPD

Anyone who gives or reviews feedback

A My CPD page shows, month by month, the learning opportunities a person flagged, the great calls they sent, and the cases they checked and reviewed, and prints a year at a time. It gives counts only and claims no hours. How the activity counts toward CPD is a question for RANZCR or the person's own college.

Metrics

Metrics show what is coming in, where it is concentrated, and whether it is being dealt with. They count learning opportunities once a radiologist has signed or checked them, using the reviewer's agreed values where there are some, with great calls beside them.

TabShows
OverviewLearning opportunities signed each week, open cases, ageing, median days to close, great calls
ProvidersCounts for each provider by issue, the share upheld after review, response times, and a weekly control chart
ERP ReportersCounts for each reporting radiologist at a provider, in and after hours
Case mixModality, body region, patient setting, after hours
UpstreamCauses that started before the report, by facility
PipelineDrafts, the checking queue and overdue cases

Who sees what

  • Viewers, such as clinical directors, see Metrics for their regions and no individual case. A count of 1 or 2 shows as "<3".
  • Reviewers, outsourcing managers and admins see exact counts and the control chart.
  • ERP Reporters is open only to reviewers and outsourcing managers.

Reading the provider figures

The figures are counts. A provider that reads more studies will show more feedback, and the Providers tab says so. Rates per thousand studies will follow once outsourced volumes are available. Feedback sent in when something stood out is a biased sample, so great calls are shown beside learning opportunities.

beta.reportfeedback · Metrics
The Metrics page for Central with
          filters for dates, region, provider, modality and how the case was found, tabs for
          Overview, Providers, ERP Reporters, Case mix, Upstream and Pipeline, five headline
          figures, and a bar chart of learning opportunities signed each week with a 4-week
          average line.
The Overview tab with synthetic figures. Every tab has a table under its chart and downloads as CSV.

Privacy

Nothing that identifies a patient goes into the tool's text, email, links or logs.

What is stored

  • The accession number identifies the study. No patient name, NHI number or date of birth is asked for.
  • Every free-text box is checked, in the browser and again on the server, for an NHI number, a date of birth or a title followed by a name, and refuses them.
  • Fields that could identify a case are stored encrypted, and every read of them is logged.

What is sent

  • Emails about learning opportunities carry labels, dates and a link, never case text.
  • A great call's email includes the accession, so the provider or department can find the report.
  • A pasted report never leaves the browser.

Who can do what

WhoWhat they can do
RadiologistGive feedback, finish drafts, sign, add addenda, read Learning points and My CPD
Referring clinicianGive feedback with no account; see their own cases after confirming their work email
ReviewerCheck and review cases in their regions, record the response and outcome, share learning points
Outsourcing managerKeep the list of providers and ERP Reporters, read Metrics for their regions
ViewerRead Metrics for their regions, with small counts hidden
AdminContacts, escalation rules and exports. Admin alone does not open cases

Where to find it

Radiology Report Feedback is in beta testing at beta.reportfeedback.hnzradiology.nz, with synthetic data only. A banner on every page says not to enter real patient or report details. Radiologists sign in with their My HNZ Radiology account; referring clinicians need no account. Review, Metrics and admin access are set by an HNZ Rad Tools administrator.

Where it stands

Real cases wait on a privacy gate: approval to host the data, a privacy impact assessment, declaration as a protected quality assurance activity under the Health Practitioners Competence Assurance Act, custody of the encryption keys, and a security review. After that, one region runs a pilot for 8 weeks before the tool opens nationally. Logins for providers, so each can see and answer its own cases in the tool, are planned after the pilot.

Open Radiology Report Feedback (beta)