Comparison of MRI response evaluation methods in rectal cancer: a multicentre and multireader validation study

Objectives: To compare four previously published methods for rectal tumor response evaluation after chemoradiotherapy on MRI. Methods: Twenty-two radiologists (5 rectal MRI experts, 17 general/abdominal radiologists) retrospectively reviewed the post-chemoradiotherapy MRIs of 90 patients, scanned at 10 centers (with non-standardized protocols). They applied four response methods; two based on T2W-MRI only (MRI tumor regression grade (mrTRG); split-scar sign), and two based on T2W-MRI+DWI (modified-mrTRG; DWI-patterns). Image quality was graded using a 0-6-point score (including slice thickness and in-plane resolution; sequence angulation; DWI b-values, signal-to-noise, and artefacts); scores < 4 were classified below average. Mixed model linear regression was used to calculate average sensitivity/specificity/accuracy to predict a complete response (versus residual tumor) and assess the impact of reader experience and image quality. Group interobserver agreement (IOA) was calculated using Krippendorff's alpha. Readers were asked to indicate their preferred scoring method(s). Results: Average sensitivity/specificity/accuracy was 57%/64%/62% (mrTRG), 36%/79%/66% (split-scar), 40%/79%/67% (modified-mrTRG), and 37%/82%/68% (DWI-patterns); mrTRG showed higher sensitivity but lower specificity and accuracy (p < 0.001) compared to the other methods. IOA was lower for the split scar method (0.18 vs. 0.39-0.43). Higher reader experience had a significant positive effect on diagnostic performance and IOA (except for the split scar sign); below-average imaging quality had a significant negative effect on diagnostic performance. DWI pattern was selected as the preferred method by 73% of readers. Conclusions: Methods incorporating DWI showed the most favorable results when combining diagnostic performance, IOA, and reader preference. Reader experience and image quality clearly impacted diagnostic performance emphasizing the need for state-of-the-art imaging and dedicated radiologist training. Key points: • In a multireader study comparing 4 MRI methods for rectal tumor response evaluation, those incorporating DWI showed the best results when combining diagnostic performance, IOA, and reader preference. • The most preferred method (by 73% of readers) was the "DWI patterns" approach with an accuracy of 68%, high specificity of 82%, and group IOA of 0.43. • Reader experience level and MRI quality had an evident effect on diagnostic performance and IOA.