Can knowledge-based DVH predictions be used for automated, individualized quality assurance of radiotherapy treatment plans?
© Tol et al. 2015
Received: 6 August 2015
Accepted: 11 November 2015
Published: 19 November 2015
Treatment plan quality assurance (QA) is important for clinical studies and for institutions aiming to generate near-optimal individualized treatment plans. However, determining how good a given plan is for that particular patient (individualized patient/plan QA, in contrast to running through a checklist of generic QA parameters applied to all patients) is difficult, time consuming and operator-dependent. We therefore evaluated the potential of RapidPlan, a commercial knowledge-based planning solution, to automate this process, by predicting achievable OAR doses for individual patients based on a model library consisting of historical plans with a range of organ-at-risk (OAR) to planning target volume (PTV) geometries and dosimetries.
A 90-plan RapidPlan model, generated using previously created automatic interactively optimized (AIO) plans, was used to predict achievable OAR dose-volume histograms (DVHs) for the parotid glands, submandibular glands, individual swallowing muscles and oral cavities of 20 head and neck cancer (HNC) patients using a volumetric modulated (RapidArc) simultaneous integrated boost technique. Predicted mean OAR doses were compared with mean doses achieved when RapidPlan was used to make a new plan. Differences between the achieved and predicted DVH-lines were analyzed. Finally, RapidPlan predictions were used to evaluate achieved OAR sparing of AIO and manual interactively optimized plans.
For all OARs, strong linear correlations (R2 = 0.94–0.99) were found between predicted and achieved mean doses. RapidPlan generally overestimated the amount of achievable sparing for OARs with a large degree of OAR-PTV overlap. RapidPlan QA using predicted doses alone identified that for 50 % (10/20) of the manually optimized plans, sparing of the composite salivary glands, oral cavity or composite swallowing muscles could be improved by at least 3 Gy, 5 Gy or 7 Gy, respectively, while this was the case for 20 % (4/20) AIO plans. These predicted gains were validated by replanning the identified patients using RapidPlan.
Strong correlations between predicted and achieved mean doses indicate that RapidPlan could accurately predict achievable mean doses. This shows the feasibility of using RapidPlan DVH prediction alone for automated individualized head and neck plan QA. This has applications in individual centers and clinical trials.
KeywordsTreatment planning quality assurance OAR sparing Knowledge-based planning
The increasing complexity of radiotherapy treatment planning, particularly due to the attempt to spare more individual organs-at-risk (OARs), has made it challenging to efficiently produce consistent, high quality radiotherapy treatment plans [1, 2]. This has led to considerable interest in (semi) automated planning strategies [3–9]. Plan quality assurance (QA) is another step in the treatment preparation workflow that might benefit from increased automation, since often, insufficient attention is given to evaluating whether a given plan can be improved. The time consuming , difficult and subjective nature of this process QA makes it hard to be confident that good plan quality is being obtained for individual patients. Furthermore, sub-optimal plans submitted to clinical trials have been correlated with worse clinical outcomes , suggesting that high quality plans are important for maximizing treatment outcomes. However, the fact that sub-optimal plans were accepted to the trial identifies a clear need for a robust and efficient plan QA tool to determine in near real-time whether plans meet an acceptable quality standard for individual patients . Although Moore et al.  have recently published an analysis in which the number of patients at unnecessary risk for normal tissue complication probability was determined by comparing the predicted and achieved dosimetry, they used an in-house developed knowledge-based planning solution .
In contrast we have investigated whether OAR dose-volume histograms (DVH) predicted by RapidPlan™, a commercial knowledge-based planning solution (Varian Medical Systems, Palo Alto, USA), are accurate enough to serve as a plan QA tool that could be used in clinical departments or for clinical trials. RapidPlan utilizes a library of plans to construct a model [14–17]. This model uses the geometrical features and associated dosimetry of the plans included in the library to predict a range of achievable DVH-lines for OARs of new patients. This range consists of the mean estimated DVH-line ± one standard deviation. The accuracy of the DVH prediction is therefore influenced by the consistency of the plans and the range of patient geometries in the library. The DVH-lines obtained during treatment planning for a patient can be compared with the DVHs predicted by RapidPlan. If the RapidPlan predictions are accurate, this comparison can be used to determine whether the evaluated plan has achieved adequate OAR sparing, as judged against the model library. This approach does not require the creation of additional plans and it would therefore present a fast and straightforward solution for plan QA. Although there have been reports of improvements in OAR sparing using RapidPlan [5–7], these do not necessarily imply that there was a close correspondence between the achieved and predicted DVH-lines, nor has this relationship been evaluated.
Our work differs from previous publications [14, 18] in several ways, including (i) using a commercial program to predict achievable DVHs, (ii) use of complex head and neck cancer (HNC) treatment plans, involving sparing of the parotid glands, submandibular glands, oral cavity and individual swallowing muscles, along with two PTVs and a simultaneous integrated boost, (iii) detailed evaluation of the relationship between predicted and achieved dosimetry along the entire DVH-curve, and (iv) demonstration of QA application by using DVH predictions generated by RapidPlan to benchmark previously created plans.
General description of treatment planning
All HNC plans were created with a simultaneous integrated boost (SIB) technique using 6MV photons and 2 full RapidArc™ (Varian Medical Systems, Palo Alto, USA) arcs. In 35 fractions, plans aimed to deliver 95 % of the prescribed dose of 54.25Gy/70.0Gy to 98 %/99 % of the elective/boost PTV (PTVE/PTVB), while limiting the volume of each PTV receiving >107 % of the prescribed dose. A 5mm transition zone (PTVT) was created between PTVE and PTVB to facilitate dose fall-off between them. The optimization goals and included OARs have been outlined in detail previously [7, 19–21].
Summary of the RapidPlan model characteristics for each individual organ-at-risk (OAR)
Number of structures included in model
Suggested number in model
Model fit (R2)
Regression model’s average χ2
Upper Esophageal Spincter
Replanning of dosimetric outliers
The RapidPlan model configuration window provides detailed information regarding geometric and dosimetric outliers that are present in the created model. Although AIO provides an automated and consistent approach to OAR dose reduction during the optimization process, some OARs could still be identified by the model as providing insufficient sparing. This could for example be because a trade-off with sparing of other OARs or strict PTV dose homogeneity criteria prevented better sparing. The consistency of the plans included in the model was therefore improved by two iterations of replanning such dosimetric outliers. In this process, the included OAR DVHs were compared against the range of achievable DVHs predicted by the model. A patient was replanned using RapidPlan if, subjectively judged, a meaningful improvement in sparing was predicted for at least one OAR. The RapidPlan model configuration window assists the user in this process by allowing them to visualize the predicted and achieved DVHs for all OARs included in the model. Cleaning the model in this fashion is intended to improve the consistency of the included plans and lead to more accurate predictions of achievable OAR DVHs for new patients.
Evaluation group for plan quality checking
- (1).For each OAR spared in the original clinical plan, the mean dose predicted by the RapidPlan model was derived by creating a “mid-prediction” DVH-line running through the middle of the DVH prediction range (Fig. 4). To determine the accuracy of the predicted DVH, this mean dose was compared against the mean dose that was obtained when a plan was made using the RapidPlan model.
Because this comparison of mean doses does not reflect possible differences between the mid-prediction DVH-line and the achieved DVH-line, this difference was determined over the entire dose range.
The mid-prediction DVH-line was used to evaluate the quality of AIO plans and manually optimized clinical plans for the 20 patients in the evaluation group. The manually optimized plans were created before AIO was introduced in our clinic, and based on prior experience , were generally expected to provide less OAR sparing than the AIO plans. For illustrative purposes, the plans were considered acceptable if the mean dose to the composite (volume weighted) salivary glands, oral cavity and composite swallowing muscles was no more than 3 Gy, 5 Gy and 7 Gy higher, respectively, than the mean dose of the mid-prediction DVH-line. These arbitrary thresholds were chosen to represent clinically meaningful values. A higher threshold value was used for the oral cavity because institutional experience suggest that this structure is subject to more contouring and geometric variability. The threshold was highest for the swallowing muscles because these structures are relatively small and large dose differences are more easily obtained when replanning. Since the OAR DVH prediction by the RapidPlan model takes the geometrical features of the evaluated patient into account, this provides a patient specific approach to plan quality assurance.
This study assessed the potential to use OAR DVH predictions generated by a RapidPlan model to evaluate the dosimetric quality of plans that were not included in the model. For all OARs, strong correlations were found between mean OAR doses predicted by the model and mean doses achieved after the model was used to guide the creation of a new treatment plan, with linear fit slopes close to one. This means that in general, achievable OAR mean doses could be determined in advance by using the model solely to predict DVHs, without requiring the creation of an actual treatment plan. Although previous investigations showed the potential of RapidPlan to improve plan quality [5–7], the present study is the first to evaluate the accuracy of the dosimetric predictions. This is relevant because improved plan quality does not necessarily imply that accurate predictions were generated, whilst accurately modeling achievable OAR doses is an important prerequisite for using RapidPlan as a plan QA tool. The prediction accuracy was determined for mean OAR doses because such doses have been shown to correlate to late toxicity for HNC patients [22–26].
Since the created RapidPlan model was found able to accurately predict the achievable mean OAR doses (Fig. 6), it could be used to evaluate the quality of previously created (manually optimized) clinical plans. Under the test conditions, this suggested that 50 % of the evaluated plans contained a composite OAR for which the sparing could be improved. These predicted gains in plan quality were not unexpected since the RapidPlan model was made using an AIO plan library, and AIO plans have been previously shown to provide improved OAR sparing over manual interactively optimized plans, along with being optimized in a more consistent fashion (9). Consistent with this, only 4/20 AIO plans were rejected because oral cavity sparing could have been improved, while salivary gland and swallowing muscle sparing was acceptable in all plans. The predicted improvements in plan quality were validated by replanning the patient using the RapidPlan model. The agreement was consistent with the high R2 values between the predicted and achieved mean OAR doses, and thus demonstrated successful application of RapidPlan-predicted DVH metrics as a QA tool. It is important to note that the high-level correlation between predicted and achieved dosimetry also allows the RapidPlan model to be used to evaluate the plans included in the model library. Replanning sub-optimal plans would allow for the continuous and consistent improvement of the model library and the resulting plan quality.
Although on average, RapidPlan could accurately predict mean doses, predictions could deviate in the high-dose regions for OARs with both low (<20 Gy) and high (>40 Gy) mean doses (Fig. 7). For the latter, RapidPlan overestimated the amount of achievable OAR sparing, probably because RapidPlan placed the line objective horizontally in the region of OAR-PTV overlap. In plans with multiple dose levels, OARs that only overlap with the lower dose PTV get no optimization objectives to restrict doses above the prescribed dose for this PTV (Fig. 1). Although this may not be as important in organs where toxicity is correlated with mean OAR dose, it could be an issue when high dose volumes are considered predictors of toxicity. If in future releases of RapidPlan, line objectives are also modeled for the part of the OAR that overlaps with the PTV, correlations between predicted and achievable OAR DVHs should improve.
Accurate DVH predictions made by RapidPlan models could similarly be used to benchmark the quality of plans that were created using alternative delivery techniques, such as proton therapy , without requiring the creation of photon plans. A comparable study was recently performed that evaluated the potential of an IMRT model from one institute and made using one IMRT technique to aid IMRT planning of another institute and predict achievable DVHs with another IMRT technique . They found that a fixed gantry model could accurately predict the median dose of the parotid gland in Tomotherapy plans, which indicated that it was spared similarly by both institutions. In addition, predictions of median dose reductions to some OARs in the Tomotherapy plans could be achieved after replanning.
Potential limitations of the current study include the following. RapidPlan indicated that for some swallowing muscles, more training cases containing these structures were required to improve the model (Table 1). This may have influenced the predicted and obtained mean doses. All plans included in the model were created using the same number of arcs, and similar collimator angles and field sizes for most plans. If plans created using different field settings are evaluated, RapidPlan may predict OAR doses that are only realizable after changing the field settings. Additionally, different optimization and dose calculation algorithms were used to create the plans contained in the RapidPlan model library (PRO and AAA v10.0.28) and the RapidPlan plans (photon optimizer [PO] and AAA v13.5.33). Small improvements in OAR sparing when using newer algorithms were found previously . Part of the gains achieved by RapidPlan may therefore be attributed to the use of the newer optimization and dose calculation algorithms, and in effect, the RapidPlan model predictions slightly underestimated the gains that could be achieved. Although our study showed good correlations between achieved and predicted OAR doses, determining the optimality of the plans included in the model library or of the resulting plans was beyond the scope of the present study. Future studies could incorporate retrospective RapidPlan QA of completed clinical trials. If a RapidPlan model were to be used as a planning QA tool for clinical trials, time and effort would need to be invested to ensure appropriate model composition and quality. Finally, if changes in treatment planning techniques lead to improvements in plan quality, these improved plans would need to be incorporated into updated RapidPlan libraries to ensure that the QA model remained state of the art, and of sufficiently high quality.
The present study showed that RapidPlan models can accurately predict achievable mean doses for most OARs, enabling them to be used to benchmark the quality of existing plans. However, the predictions may be less accurate for OARs that overlap substantially with the PTVs and may not accurately reflect the shape of the DVH-line. Addressing these limitations should improve the prediction accuracy and the potential of RapidPlan for plan quality assessment.
This research was supported by a grant from Varian Medical Systems.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Nelms BE, Robinson G, Markham J, Velasco K, Boyd S, Narayan S, et al. Variation in external beam treatment plan quality: An inter-institutional study of planners and planning systems. Pract Radiat Oncol. 2012;2:296–305.View ArticlePubMedGoogle Scholar
- Das IJ, Cheng C-W, Chopra KL, Mitra RK, Srivastava SP, Glatstein E. Intensity-modulated radiation therapy dose prescription, recording, and delivery: patterns of variability among institutions and treatment planning systems. J Natl Cancer Inst. 2008;100:300–7.View ArticlePubMedGoogle Scholar
- Voet PWJ, Dirkx MLP, Breedveld S, Fransen D, Levendag PC, Heijmen BJM. Toward fully automated multicriterial plan generation: a prospective clinical study. Int J Radiat Oncol Biol Phys. 2013;85:866–72.View ArticlePubMedGoogle Scholar
- Breedveld S, Storchi PRM, Voet PWJ, Heijmen BJM. iCycle: Integrated, multicriterial beam angle, and profile optimization for generation of coplanar and noncoplanar IMRT plans. Med Phys. 2012;39:951–63.View ArticlePubMedGoogle Scholar
- Fogliata A, Wang P-M, Belosi F, Clivio A, Nicolini G, Vanetti E, et al. Assessment of a model based optimization engine for volumetric modulated arc therapy for patients with advanced hepatocellular cancer. Radiat Oncol. 2014;9:236.PubMed CentralView ArticlePubMedGoogle Scholar
- Fogliata A, Belosi F, Clivio A, Navarria P, Nicolini G, Scorsetti M, et al. On the pre-clinical validation of a commercial model-based optimisation engine: Application to volumetric modulated arc therapy for patients with lung or prostate cancer. Radiother Oncol. 2014;113:385–91.View ArticlePubMedGoogle Scholar
- Tol JP, Delaney AR, Dahele M, Slotman BJ, Verbakel WFAR. Evaluation of a knowledge-based planning solution for head and neck cancer. Int J Radiat Oncol Biol Phys. 2015;91:612–20.View ArticlePubMedGoogle Scholar
- Wu B, McNutt T, Zahurak M, Simari P, Pang D, Taylor R, et al. Fully automated simultaneous integrated boosted-intensity modulated radiation therapy treatment planning is feasible for head-and-neck cancer: a prospective clinical study. Int J Radiat Oncol Biol Phys. 2012;84:e647–53.View ArticlePubMedGoogle Scholar
- Tol JP, Dahele M, Peltola J, Nord J, Slotman BJ, Verbakel WF. Automatic interactive optimization for volumetric modulated arc therapy planning. Radiat Oncol. 2015;10:75.PubMed CentralView ArticlePubMedGoogle Scholar
- Miles EA, Clark CH, Urbano MTG, Bidmead M, Dearnaley DP, Harrington KJ, et al. The impact of introducing intensity modulated radiotherapy into routine clinical practice. Radiother Oncol. 2005;77:241–6.View ArticlePubMedGoogle Scholar
- Peters LJ, O’Sullivan B, Giralt J, Fitzgerald TJ, Trotti A, Bernier J, et al. Critical impact of radiotherapy protocol compliance and quality in the treatment of advanced head and neck cancer: results from TROG 02.02. J Clin Oncol. 2010;28:2996–3001.View ArticlePubMedGoogle Scholar
- Wang J, Chen W, Studenski M, Cui Y, Lee AJ, Xiao Y. A semi-automated tool for treatment plan-quality evaluation and clinical trial quality assurance. Phys Med Biol. 2013;58:N181–7.View ArticlePubMedGoogle Scholar
- Moore KL, Schmidt R, Moiseenko V, Olsen LA, Tan J, Xiao Y, et al. Quantifying Unnecessary Normal Tissue Complication Risks due to Suboptimal Planning: A Secondary Study of RTOG 0126. Int J Radiat Oncol Biol Phys. 2015;92:228–35.View ArticlePubMedGoogle Scholar
- Appenzoller LM, Michalski JM, Thorstad WL, Mutic S, Moore KL. Predicting dose-volume histograms for organs-at-risk in IMRT planning. Med Phys. 2012;39:7446–61.View ArticlePubMedGoogle Scholar
- Yuan L, Ge Y, Lee WR, Yin FF, Kirkpatrick JP, Wu QJ. Quantitative analysis of the factors which affect the interpatient organ-at-risk dose sparing variation in IMRT plans. Med Phys. 2012;39:6868–78.View ArticlePubMedGoogle Scholar
- Zhu X, Ge Y, Li T, Thongphiew D, Yin F-F, Wu QJ. A planning quality evaluation tool for prostate adaptive IMRT based on machine learning. Med Phys. 2011;38:719.View ArticlePubMedGoogle Scholar
- Moore KL, Brame RS, Low DA, Mutic S. Experience-based quality control of clinical intensity-modulated radiotherapy planning. Int J Radiat Oncol Biol Phys. 2011;81:545–51.View ArticlePubMedGoogle Scholar
- Lian J, Yuan L, Ge Y, Chera BS, Yoo DP, Chang S, et al. Modeling the dosimetry of organ-at-risk in head and neck IMRT planning: An intertechnique and interinstitutional study. Med Phys. 2013;40:121704.PubMed CentralView ArticlePubMedGoogle Scholar
- Tol JP, Dahele M, Doornaert P, Slotman BJ, Verbakel WF. Toward optimal organ at risk sparing in complex volumetric modulated arc therapy: An exponential trade-off with target volume dose homogeneity. Med Phys. 2014;41:021722.View ArticlePubMedGoogle Scholar
- Doornaert P, Verbakel WF, Bieker M, Slotman BJ, Senan S. RapidArc planning and delivery in patients with locally advanced head-and-neck cancer undergoing chemoradiotherapy. Int J Radiat Oncol Biol Phys. 2011;79:429–35.View ArticlePubMedGoogle Scholar
- Doornaert P, Verbakel WF, Rietveld DHF, Slotman BJ, Senan S. Sparing the contralateral submandibular gland without compromising PTV coverage by using volumetric modulated arc therapy. Radiat Oncol. 2011;6:74.PubMed CentralView ArticlePubMedGoogle Scholar
- Beetz I, Schilstra C, van der Schaaf A, van den Heuvel ER, Doornaert P, van Luijk P, et al. NTCP models for patient-rated xerostomia and sticky saliva after treatment with intensity modulated radiotherapy for head and neck cancer: the role of dosimetric and clinical factors. Radiother Oncol. 2012;105:101–6.View ArticlePubMedGoogle Scholar
- Dijkema T, Raaijmakers CPJ, Ten Haken RK, Roesink JM, Braam PM, Houweling AC, et al. Parotid gland function after radiotherapy: the combined michigan and utrecht experience. Int J Radiat Oncol Biol Phys. 2010;78:449–53.PubMed CentralView ArticlePubMedGoogle Scholar
- Deasy JO, Moiseenko V, Marks L, Chao KSC, Nam J, Eisbruch A. Radiotherapy dose-volume effects on salivary gland function. Int J Radiat Oncol Biol Phys. 2010;76(3 Suppl):S58–63.PubMed CentralView ArticlePubMedGoogle Scholar
- Christianen MEMC, Schilstra C, Beetz I, Muijs CT, Chouvalova O, Burlage FR, et al. Predictive modelling for swallowing dysfunction after primary (chemo)radiation: results of a prospective observational study. Radiother Oncol. 2012;105:107–14.View ArticlePubMedGoogle Scholar
- Murdoch-Kinch C-A, Kim HM, Vineberg KA, Ship JA, Eisbruch A. Dose-effect relationships for the submandibular salivary glands and implications for their sparing by intensity modulated radiotherapy. Int J Radiat Oncol Biol Phys. 2008;72:373–82.PubMed CentralView ArticlePubMedGoogle Scholar
- Weber DC, Hurkmans CW, Melidis C, Budach W, Langendijk JH, Peters LJ, et al. Outcome impact and cost-effectiveness of quality assurance for radiotherapy planned for the EORTC 22071-24071 prospective study for head and neck cancer. Radiother Oncol. 2014;111:393–9.View ArticlePubMedGoogle Scholar
- Barten DLJ, Tol JP, Dahele M, Slotman BJ, Verbakel WF. Comparison of organ-at-risk sparing and plan robustness for spot-scanning proton therapy and volumetric modulated arc photon therapy in head-and-neck cancer. Med Phys. 2015;42:6589–98.View ArticlePubMedGoogle Scholar