Project “Dame”

Usability Engineering

Together with COSMONiO IMAGING B.V., UMC Groningen (UMCG), Radiologie Westmünsterland and Carl-von-Ossietzky University Oldenburg, Use-Lab participated in the DAME project (Deep-Learning Algorithm for Medical Imaging Evaluation) within the INTERREG program. The central task of Use-Lab was to perform the usability engineering for this deep learning algorithm.

Literature research as part of the requirements analysis

In order to provide the researchers with a more comprehensive knowledge of medical imaging, its foundations, and any differences between Germany and the Netherlands, Use-Lab first conducted an extensive literature review. For this purpose, various sources were consulted, such as national statistics, laws and guidelines, scientific journals, manufacturer websites, or, if available, reports and documentation of existing systems.

After final analysis of the collected data and Use-Lab’s clinical expertise and the application of general guidelines, an overview of the current industry standards in computer-aided diagnostics as well as the regional differences between Germany and the Netherlands emerged.

In order to provide the researchers with a more comprehensive knowledge of medical imaging, its foundations, and any differences between Germany and the Netherlands, Use-Lab first conducted an extensive literature review. For this purpose, various sources were consulted, such as national statistics, laws and guidelines, scientific journals, manufacturer websites, or, if available, reports and documentation of existing systems.

After final analysis of the collected data and Use-Lab’s clinical expertise and the application of general guidelines, an overview of the current industry standards in computer-aided diagnostics as well as the regional differences between Germany and the Netherlands emerged.

Field study and semi-structured interview within the requirements analysis

A field study is the obvious choice for collecting data from real users as part of their normal workflow. Here, the user’s activities are observed as unobtrusively as possible and then discussed.

A semi-structured interview is a research method used to collect focused, qualitative data. This method can uncover extensive descriptive data from the perspective of (potential) end users. In such an open interview, the users are given the opportunity to elaborate on the things that are most important to them.

For the present study, this offered comprehensive insights into the workflow of medical imaging, with a focus on exploring the requirements and wishes of the interviewees.

A field study is the obvious choice for collecting data from real users as part of their normal workflow. Here, the user’s activities are observed as unobtrusively as possible and then discussed.

A semi-structured interview is a research method used to collect focused, qualitative data. This method can uncover extensive descriptive data from the perspective of (potential) end users. In such an open interview, the users are given the opportunity to elaborate on the things that are most important to them.

For the present study, this offered comprehensive insights into the workflow of medical imaging, with a focus on exploring the requirements and wishes of the interviewees.

Summary of the results

A total of 3 sites in the Netherlands and Germany were visited. Workflows, image generation, image acquisition, reporting, workflow in and after radiology, software experience of the experts and DICOM data structures were investigated.

Animation film workflow of radiology

Following the requirements analysis, Use-Lab collaborated with UMCG to create an animated film that would explain the vision of DAME in a simple and understandable way. You can find the result here:

Formative study

The project scope also included a formative study with the aim of identifying potential improvements in terms of usability and safety of the graphical user interface (GUI) for controlling a “deep-learning” algorithm for the evaluation of medical imaging by potential users (radiologists, medical physicists and techs). The algorithm was developed by COSMONiO BV of Groningen. The AI platform behind it is called NOUS.

NOUS is an AI platform that can be taught to recognize specific patterns in 2D image data. As part of DAME, it is being further developed for an expansion of applications for radiologists as well as medical physicists.

The formative study was conducted as a expert interviews by Use-Lab’s interdisciplinary human factors expert team of application specialists, engineers, usability experts and designers. The GUI was presented to the team and discussed and critiqued in a collective brainstorming session. The expertise was extended by interviewing a radiologist and two medical physicists.

The goal of the study was to highlight strengths and weaknesses of the GUI, focusing on the usability of the GUI for training the algorithm to detect specific tissue in CT scans.

Based on the expertise and feedback from external experts, recommendations for the design of the GUI were derived, on the basis of which a completely revised GUI was developed.

The project scope also included a formative study with the aim of identifying potential improvements in terms of usability and safety of the graphical user interface (GUI) for controlling a “deep-learning” algorithm for the evaluation of medical imaging by potential users (radiologists, medical physicists and techs). The algorithm was developed by COSMONiO BV of Groningen. The AI platform behind it is called NOUS.

NOUS is an AI platform that can be taught to recognize specific patterns in 2D image data. As part of DAME, it is being further developed for an expansion of applications for radiologists as well as medical physicists.

The formative study was conducted as a expert interviews by Use-Lab’s interdisciplinary human factors expert team of application specialists, engineers, usability experts and designers. The GUI was presented to the team and discussed and critiqued in a collective brainstorming session. The expertise was extended by interviewing a radiologist and two medical physicists.

The goal of the study was to highlight strengths and weaknesses of the GUI, focusing on the usability of the GUI for training the algorithm to detect specific tissue in CT scans.

Based on the expertise and feedback from external experts, recommendations for the design of the GUI were derived, on the basis of which a completely revised GUI was developed.

Webinars & Presentations

To inform project partners and the public about Use-Lab’s findings in the DAME project, Use-Lab developed a mini-webinar on trust in automation (based on Deep Learning algorithms). The webinar is available on YouTube via the channel of the UMCG’s DASH platform in English:

Webinar “Strategies towards appropriate reliance on Deep-Learning Algorithms”:
Cookies are set when the video is played on YouTube. For more information, see the Cookie-Policy.

In addition, Use-Lab gave a presentation at the DAME Symposium on the results of the research. The results were supported by Use Labs’ expertise in the field of usability. The presentation is available in English on YouTube via the UMCG’s DASH platform channel:

Symposium Talk “Usability & AI”:
Cookies are set when the video is played on YouTube. For more information, see the Cookie-Policy.

Eye Tracking Study

As part of the project, an eye tracking study was conducted. The aim of this study was to investigate the impact of AI support on annotation, i.e. training of a Deep Neural Network (DNN).

The study was conducted with a total of 10 medical imaging professionals. In a first round, the professionals trained a DNN using NOUS (see above) to recognize healthy organs in CT image data without automatic analysis. In a second round, the professionals performed the same task using automatic analysis by a pre-trained algorithm. Various eye tracking measurements were taken to investigate what effect automatic analysis had on the performance of the professionals.

Several contrasting effects emerged in the study. For example, in some cases the introduction of AI support was associated with an improvement in performance for all types of tasks, but in other cases it was associated with a decrease. In some cases, the introduction of AI support improved performance on the most difficult tasks, while in other cases it produced difficult tasks. Third, sometimes peak performance improved after the introduction of AI support, while in other cases it deteriorated. Fourth, participants with more work experience performed better manually than with AI support, while this was reversed for less experienced participants. Their performance could be improved with AI support. Overall, it was found that AI support can have both positive and negative effects on the work of medical imaging professionals, although the reasons for these effects could vary.

As part of the project, an eye tracking study was further conducted. The aim of this study was to investigate the impact of AI support on annotation, i.e. training of a Deep Neural Network (DNN).

The study was conducted with a total of 10 medical imaging professionals. In a first round, the professionals trained a DNN using NOUS (see above) to recognize healthy organs in CT image data without automatic analysis. In a second round, the professionals performed the same task using automatic analysis by a pre-trained algorithm. Various eye tracking measurements were taken to investigate what effect automatic analysis had on the performance of the professionals.

Several contrasting effects emerged in the study. For example, in some cases the introduction of AI support was associated with an improvement in performance for all types of tasks, but in other cases it was associated with a decrease. In some cases, the introduction of AI support improved performance on the most difficult tasks, while in other cases it produced difficult tasks. Third, sometimes peak performance improved after the introduction of AI support, while in other cases it deteriorated. Fourth, participants with more work experience performed better manually than with AI support, while this was reversed for less experienced participants. Their performance could be improved with AI support. All in all, it could be concluded that AI support can have some, positive or negative, impact on the work of medical imaging professionals, although the reasons for these effects could vary.

This project was financially supported by the European Union and INTERREG partners under the INTERREG program.

www.deutschland-nederland.eu

Funding Partners

Project Partners

Nice to see you

In a personal conversation, we can determine your needs and tailor our services to your requirements.

We look forward to meeting you.