Streamline Scientific Reviews with Confidence throughout the Product Lifecycle

Speakers: Dimitriya Bozukova, Diane Legere, Romney Adams, and Ulla Gertig

[Download the Guide to Medical Literature Review]

Index

[Download Slides]

Purpose and Process of Literature Reviews 

(03:44 - 13:47) 

• Begin in R&D to assess competition and support design 
• Essential for clinical evaluation, PMS, and PMCF 
• “State of the art” reflects accepted practices 
• Effective process includes defined search plans, validated terms, and reusable templates 

Search Strategy and Tools 

(13:48 - 22:21) 

• Searches address conditions, interventions, benchmark and similar devices 
• Tools: PubMed, Embase, EndNote, Zotero, Citavi, DistillerSR 
• DistillerSR enables team collaboration, AI tagging, reuse, and PDF handling 

Key Takeaways and Implementation 

(22:22 - 30:09) 

• Start reviews early and update regularly post-market 
• Use validated templates and automate when possible 
• Notified bodies expect annual updates at minimum  

Panel Insights on Effectiveness and AI 

(30:10 - 48:42) 

• Effectiveness gauged by saturation, specificity, and consistency 
• DistillerSR praised for automation, integration, and data reuse 
• AI aids screening, extraction, and ongoing updates 
• Validation and evolving data present challenges for AI adoption 

Common Pitfalls and Final Q&A 

(48:43 - 58:30) 

• Main deficiency: lack of synthesis in reports 
• When benchmarks are missing, rely on risk assessments and generic device groups 
• Literature reviews must be dynamic and ongoing 

Full Transcript

Ulla Gertig - 03:43
Hello everyone, and thank you for joining today's webinar on Mastering Literature Review on Medical Devices. It is a pleasure to be here, and before I begin, I want to thank our guests again for joining this presentation and also for sharing the insights later on. So I will start with your short presentation, giving some background information and highlighting some critical aspects. And afterwards, we will dive deeper into burning topics like software solutions, AI, and literature review during our panel discussion, which we're basically starting shortly after the presentation. So, I actually have to start with a brief disclaimer. The content of this presentation is only meant for educational purposes, only representing the opinion of AKRA team and not necessarily the opinion of other stakeholders. Okay, let's start with the agenda now.

Ulla Gertig - 04:50
While I could actually talk for hours on the topic of literature reviews, I'm limited to 15 minutes today, and I have to admit it was so hard, so hard to limit the content to 15 minutes. I will talk a little bit about where literature reviews are essential across the product lifecycle, how to design efficient reusable processes, I will present some fundamental tools that you can use today, and we will also touch base on how AI and large language models can save time and elevate your strategy, whereby the last part will be mainly addressed during the Q&A session and the panel discussion. 

Okay, let's start with the product life cycle. First of all, the life cycle of a product or medical device is actually not defined in the EU MDR. However, we are lucky because there is at least an IMDF document existing which defines the life cycle as all phases in the life of a medical device from the initial conception to the final decommissioning and disposal. The life cycle of the device is basically understood as the period between the first conceptual ideas during the research and development process, and it ends with the last device, which will then disappear from the market.

Ulla Gertig - 06:12
So when we now look into the question where are literature reviews essential across the product lifecycle, we are really studying during the R&D cycle. During the R&D cycle, literature reviews are usually used to identify the competitive landscape and to identify or define the added value of the device, and also to search for relevant safety and performance parameters which can be used for preclinical and clinical investigations. This is especially important when comparing the device data to state-of-the-art data later on. It is not that unusual that the endpoints investigated via extensive and expensive studies do not really reflect what is usually used as an outcome in the state of the art, missing important parameters. So, a proper literature review during your R&D can really prevent you from redoing clinical investigations or from performing additional PMCF activities just to address those missing endpoints of relevance.

Next, as you probably all know, you need a literature review for the clinical evaluation, in which you describe the state of the art in the respective medical fields, the therapeutic alternatives, gold standard, benchmark devices, and in which you also define the endpoints. Define the endpoints of those parameters, and in which you're also looking for your subject device data. Next, you of course also have literature review during your PMS and PMCF activities, which most of you probably know. So, the goal of the literature search is basically to identify and describe the state of the art for a particular therapy.
Thereby, it is important to keep in mind that state-of-the-art is not the most recent hype or trend in technology.

SOTA is rather understood as the current knowledge and good practice acceptable for medical treatment of a certain condition using medical devices. The literature search is also used to identify meaningful, measurable, and clinically relevant parameters on safety and performance, and to define the acceptance criteria that have to be achieved by your subject device.

Ulla Gertig - 08:47
So the literature search is basically providing the basis that enables a proof of compliance for your subject device, showing that the device is in line with acceptance criteria established from the state of the art. 

How to design efficient and reusable processes? To perform a literature review, you need to have your defined sources of information, a team responsible for the search and also for appraising the literature, you need to have detailed and correct key data of the subject device, a software that is organising your literature, and of course also good and sound planning, structuring, and a good search strategy. When you consider the literature search cycle, it is starting with defining a good literature search plan in which you are extensively describing what you will actually do.

Next, the literature search has to be executed. So you are doing the search and everything that was planned as described within the literature search plan or protocol, which means screening of literature, excluding and including articles, and also appraising the articles.

And lastly, you will then extract your information into various documents, and you will describe all your results in the literature search report, where you present how you assess which paper and how are the assessments of all the included articles.

Even though this is something that actually needs quite some time to do, therefore we are really recommending you use standardized templates and processes. If you are a manufacturer with more than one or two devices, templates can really save so much time. They are enormous time-savers, and you can add all generic information inside so that you only need to adapt device-specific information, keeping the remaining content consistent throughout your documentation at devices. In case you do not have a template in place, take the time to create your template or simply buy one. I'm working for AKRA Team, so I'm using the AKRA Team template on a daily basis, and I can only recommend it. If you want to know more about that, ,feel free to contact me via LinkedIn, just drop a comment, or otherwise create your own template.

Ulla Gertig - 11:34
Usually you have one protocol and report for the SOTA, so the state of the art data, and one protocol and report for the safety and performance search of your device.

And in the LSP, you are usually defining your search terms, the data sources that you are using, like PubMed or Embase, the strategies that you are applying, which is usually PICO, the timeframe, your inclusion and exclusion criteria, as well as your selection criteria for the literature appraiser in the literature search report, you are typically repeating some of those key elements described in the LSP, but you are also adding the results, like who carried out the literature search and when, what were the exact search strings used, and how many outcomes were obtained per search string, which references were excluded for which reason, on which level of screening, and which references were included and how is their appraisal.

And while the LSP and LSR template can really decrease the time spent on the whole literature process, there are also certain aspects which require special attention. For example, the creation of search terms. When you create search terms, please make sure to take your time, as search terms are especially susceptible to errors.

Just this morning, I had one of those errors again in my search terms, and it took me ages to figure out where it was really. So you can see a search string here on the left side that we are typically using in our template, and one missing or wrong letter or character or one space in the wrong place can really mess up everything. Thus, we would really advise always to double-check for correctness. It is better to take extra time to request a second pair of eyes to have a look than to notice later, when you're basically done with literature review, that there was a mistake that you have not noticed earlier. And believe me, those errors happen more often than you might think. Another important aspect is that you should really design the search terms in a way that fits your needs.

Ulla Gertig - 13:47
If you have an unreasonably high amount of fits, you can just consider to exclude unnecessary data sources, or you can limit your search, for example, to major topics, which is an imitation that can be used by PubMed and Embase, which is limiting the data in which your keywords basically appears as a major topic only, and based on experience, I can also recommend to always exclude certain data sources like comments or abstract. So while we are in our search, where we are focusing here, for example, on including just guidelines and consensus documents, we noticed that a lot of conference abstracts are appearing in the search when you are not specifically excluding them.

And while it is not hard to exclude them later, you often really need to wait until your full-text level to exclude them, as it is not clear before that those abstracts are actually just conference abstracts. So by eliminating them through the search directly, you will not end up with this problem, and you will save some precious time. Here's an overview of the search subjects and filters that we are typically using.

Ulla Gertig - 15:09
So our first search usually covers the clinical condition and indication, and we are typically restricting the search to guidelines and consensus documents only. Guidelines are really a great basis to understand and describe the condition and to identify the typically used gold standard and other alternative treatment options. We are also typically limiting the search time to five or ten years depending on the number of fits and sometimes depending on the age of your device, so if it's a really old device, you even need to go back to 15 years sometimes to get any data at all, because just not the focus of research anymore. Sometimes it's really ridiculous.

Ulla Gertig - 15:59
After looking for the clinical condition and indications, we are typically checking the clinical intervention and also alternative treatment options. Here we are limiting our search on guidelines, meta-analysis, and systematic reviews, as we do not expect to extract any safety and performance parameter-related data. The time restriction that we are applying here is usually the same as used for other searches. So, if we decide to go for five years in the clinical condition search, we typically have it throughout all the searches that have a time restriction. 

Next, we look for information on the generic device group and benchmark devices, which is very similar to the previous search, just that we have now included basically all groups of clinical studies as we can already get or extract some nice, detailed information on safety and performance. And lastly, we have a similar device.

Ulla Gertig - 17:04
It's always good if you have similar devices defined. Sometimes you don't have it, and you need to be more detailed in the general device group. But if you have similar devices, we typically do not restrict the time, and we really look for all things that were published for the devices to get as much safety and performance-related information as possible. I have more slides with a bit more detailed information, but due to the time, I will not go through them in detail. We are having highlights. One thing that I have to mention is to please be aware that you really need to consider variants in spelling and writing when you make your search terms.

Ulla Gertig - 17:51
So, for example, some people are using Johnson & Johnson, some are just writing with a complete AND, some do not even mention it, and some just use abbreviations, so you need to consider all those variants within your search terms. When we go for the safety and performance search, where we are only considering device or equivalent device data, we are also not doing any time restriction. The search is basically the same as we are performing it for similar devices. So what else do we actually need? Next, we also need the software for organizing the literature and the PDFs. And probably everyone knows and uses PubMed and Embase, which are basically search platforms. While PubMed is free, Embase is actually a fee-based service.

Ulla Gertig - 18:50
Yeah, those two softwares are typically used to perform the search and to enter with a specific number of relevant publications. Apart from that, you also need to use a citation and reference management tool. These are, for example, EndNote, which is also a fee-based software. Zotero, which is, I think, at least when I used it for free, and Citavi.
And apart from that, you can also use other tools which are available, for example, the Literature Review Software DistillerSR, or Adobe AI assistant which basically uses AI so you can ask questions to your PDF, and then the AI is giving answers based on the content of your PDF. It can also be quite useful.

Ulla Gertig - 19:47
So we at AKRA TEAM are mainly using Distiller as a review tool, so I will give you just some more details on this software. In Distiller, you can upload your list of search results from PubMed and Embase and whatever other search databases you are using. You can use an automatic deduplication function, and you can screen your results on title, abstract and full text level. And you can of course also appraise your included publications. All of those steps are customizable to your needs.

Once you have your literature search protocol or plan ready, you can use the inclusion and exclusion criteria and also the appraisal criteria that you defined there, and add those as options to select from within your screening and appraisal at Distiller, which is really nice. So the purpose of Distiller is to create a literature database to support the selection process, and it is specifically meant for a team of persons. I have to say that it's really easy to work together on one project at the same time.

Ulla Gertig - 20:59
We are doing that quite frequently. Distiller is also using some AI and intelligent workflows throughout their platform, and what we are actually using most, at the moment at least, is the keyword highlighting. So you can add specific keywords specific to your indication device or whatever you can highlight. You can let Distiller highlight those words within different colors, and then it can highlight all those keywords in title, abstract, and also full-text screen, which is really nice, especially for full text, because you can actually upload your PDF to Distiller so every reference can be attached with the PDF. And then you have it all colourful with your highlights, and you can really focus on what you need to look for. But Distiller is offering much more, for example, automatic PICO and adverse event detection on an abstract level, just not a lot of things that I'm using at the moment, so I will stop describing it here. That's everything that I wanted to tell you.

Ulla Gertig - 22:22
I have just one more slide left, which is the key takeaways. So remember, literature  review starts in the R&D phase, and it's really important to be used there. If you are a manufacturer of more than two devices, create and use literature search protocol report templates. Always double-check your search terms and adapt them to get a manageable number of searches and, if possible, utilize software solutions, which can really accelerate your review process. And with this, I'm done with the presentation.

Sean Smith - 23:08
So, there is a poll on the screen, you should be able to see it. Please take a moment to answer the poll questions. We can move into the next part while this is running.
We'll let it run for about a minute and a half more and give people a chance to answer the questions. Couple notes, the webinar will be recorded. It will be made available,
and tomorrow, by email, it will be sent to your email by Zoom, and there were a number of links and opportunities and stuff put in the general chat so please take advantage of those as well. That's it for me.

Ulla Gertig - 23:57
Okay, in the meantime, while the poll is still online, I would like to ask Dianne, Romney, and Dimitriya, would you please like to share one or two key takeaways for the audience?

Diane Legere - 24:11
Sure, I'll go first. Thank you for that presentation, Ulla. That was very helpful. And I'm going to say, as a Notified Body Clinical Reviewer, I really hope lots of people really listened to it. For me, literature searches tell me how much you know about your device. And they are probably one of the most important elements of your clinical evaluation that helps me build trust in your review. So that's my takehome.

Dimitriya Bozukova - 24:44
From a manufacturer's perspective, I'm very happy to hear the notified body perspective, and from a manufacturer perspective, we would like to be able to design devices which would really bring benefits to the patient and to the user respectively during a procedure. So to be able to do that, we have to have a very good understanding of the current context in which we are offering the device, state of the art was mentioned throughout the presentation, and is a very important input to the design of new devices, and then of course throughout the lifetime of the device, we would like to make sure that it is still performing as expected or even better to capture any safety signals. So, I think literature search is key for tha, and having it well done is very important indeed.

Romney Adams - 25:34
I'll round this out. Thanks, Ulla, for that presentation. I feel like you did part of my job for me by talking about Distiller so positively. One thing that I would say is, you talked a lot about kind of templates and processes and things, and, for me, once you've got processes standardised and perfected, there are so many opportunities you have to augment and expedite your literature reviews in a metered, human-validated way, and it doesn't necessarily have to involve AI either, so that would be something like people to consider going on from here.

Ulla Gertig - 26:35
Okay, let's just start with the panel discussion. The first question is already related basically to the medical device lifecycle. So you hear that, at least from our consultation perspective, it's really necessary to start with that in the research and development phase, but we would like to know at what specific point in the medical device life cycle do you consider literature reviews to be non-negotiable and why? So that would be a question for Diane and Dimitriya, if you would like to say something about

Dimitriya Bozukova - 27:31
So a literature search is conducted at any time, of course you mentioned that it's non-negotiable at some point of time but of course it is happening all the time because it is also bringing insights for new product development so it is inspiring innovation and there that may not be really a must have because information can be collected by different ways but it is powerful source. However, as a non-negotiable or I would say even mandatory step, this is when we enter into the design control stage and when we start defining the user requirement specification of the device in a very formal and quality-managed way. So from that point of time, the literature search should be done in a way which is exhaustive, extensive, specific, and giving us all the relevant information that would allow us to build our user requirement. And at that point in time, we also start the risk management, so any risks that may have been captured from systematic literature search on the state of the art would then be inputed to the risk management process and to the design control in this case as well. So, that stage of the device lifetime and life cycle is the start of the non-negotiable phase.

Diane Legere - 28:37
That's perfect for me to jump in, because as a notified body, we are involved once the device is ready to go on the market. So from then on, it is a non-negotiable. So, for me, I need to see a literature review that's planned for at least annually. Depending on how stable the state of the art is for your device, it might need to be more often, or if there's a lot of novelty in your device, so there wasn't much on the market, and so you'll need to continuously be reviewing it. But I had one comment on the poll.

Diane Legere - 29:24
I don't understand why that's not 100%. So that was one thing that was interesting for me, and then only 10% for post-market clinical follow-up. So I'm not sure if that's just because you consider it part of your PMS and not so much your PMCF. But there, speaking of non-negotiable, for both of those parts of question two, literature searches are non-negotiable.

Dimitriya Bozukova - 29:56
I agree. I mean, from early design phase until end of life, literature searches are happening all the time throughout the lifetime of the device. Correct.

Ulla Gertig - 30:09
Okay great, I think that is very important here for some other manufacturers.
Okay, coming to the next question. What metrics do you use to measure the effectiveness and efficiency of your literature review processes? So this is now a question for, yeah, again, Diane and Julia.

Diane Legere - 30:31
I found this question very challenging because, for me, I just look at when I feel saturated. So when I see the literature review is not finding anything new, or I start looking through reference lists, and I start to see that I'm recognizing authors and topics, and when the literature review aligns with, you talked about professional guidelines as part of your review. So I'll review those and see if what I'm seeing in the literature review aligns there, or if there's something in there that's missing on either side, but when those two alignm then I start to feel very comfortable with the review and feel that it was maybe not exhaustive, and I don't know what that even means with things coming on all the time, but comprehensive is the goal.

Dimitriya Bozukova - 31:32
So again, from a manufacturer perspective, I'm really happy to have this kind of very diverse panel because we also learn from each other, from our stakeholders. Thank you so much, AKRA TEAM, for setting up this panel, because now I see what is the perspective of a notified body. From a manufacturer's perspective, of course we would like to be exhaustive. We don't want to miss data. And to be able to do that, first we have to define, and Ulla you mentioned that during your presentation, a very well- structured search term line or search term criteria so that we don't miss any information. And that's an input to the literature search process.

Dimitriya Bozukova - 32:17
So this is actually validated with several different stakeholders to make sure that we capture the proper search terms, the proper keywords. But once we are doing the search, of course, for us it is important that we identify the right sources of information, but not too many sources which are not applicable, because then our screening stage would be really extensive. So you would see also certain metrics in the literature published on the topic, like specificity and sensitivity. These are also applicable for literature searches. So, how many data sets do you retain? How many do you reject?
And the ones that you reject, were they really not relevant, your search terms were not properly designed, or maybe your tool didn't work? So, it depends on the input that you're providing to the methodology.

Dimitriya Bozukova - 33:05
Then, of course, we would like to save time and gain inefficiency, so that's why automation of the process is very helpful in this case, or digitalization of certain at least certain stages of the search process to scale up, to accelerate because now we are reset when we start the literature search, but we have to remind that for some devices, we have to do it annually or even more often, depending on the novelty of the device, and if the literature search process is very slow or not very efficient, you may easily lose a lot of time or be late with timelines. Time and the quality of the outcomes, which are identified are essential metrics for us.

Ulla Gertig - 33:54
Great, thank you. That gives us a bridge to the next question. So, I remember that you used also Distiller, so I would like to know which specific features have proven most valuable for your medical device literature reviews, Dimitria?

Dimitriya Bozukova - 34:13
We are still in the process of implementation of a digital tool to be honest, so far we have been doing that more on a manual basis I'd say. But what we really found helpful with Distiller as a digital platform is that it helped actually the communication with our stakeholders, we are using not only for the clinical evaluation, but we are using now also, we are trying actually to use it, it's not yet the case, but to use it for market intelligence. So we are partnering with our colleagues from different functions who may be actually conducting literature searches also for their processes. So, it is a great platform for collaboration within the different cross-functional stakeholders. Regarding the actual systematic review process, what we also find very helpful, scalable, and sustainable is that once a literature search is completed, the data is quality auditable. It's helping us standardize the process. So, from a quality perspective, it's fantastic. We can also reuse certain data analysis of an article if another user from another function would like to assess the same article. They don't have to redo everything, they can jump into that record and complement it with new questions or new data. So it's really a scale-up, an accelerator of stakeholder engagement and promoting sustainability in the literature search.

Ulla Gertig - 35:53
Great, thank you.
Romney, as a Distiller expert, would you like to add something?

Romney Adams - 35:59
Yeah, absolutely, sure. Dimitriya, she hit on some really great points and features that we do have. A couple of the more basic ones that I would say that we have in terms of features is that Distiller has an API connection with the PubMed database. So, rather than needing to search PubMed externally and then bring everything into the platform, you can just do all that searching from within the platform itself, and you can capitalize by scheduling those PubMed searches and scheduling the imports that come in and automate that process. So, in the case of if you're doing like a living review or if you're updating your post-market surveillance review or anything like that, you can just set and forget all those updates, because that can all get taken care of in the background, and Distiller will bring in the updated literature from PubMed for you, and then it's just ready there for you to screen and extract the data from, which is a huge time saver.

Romney Adams - 36:56
In terms of working with full text, if you have PDFs for your references or for those papers that you are bringing in as part of your literature review, there's a matching functionality in Distiller where you can just upload your PDFs, and Distiller will again work in the background while you're off at another meeting or doing something else in the platform, and it will actually match and attach those PDFs to the correct references, so again a huge time saver there as well. Dimitriya mentioned our data reuse layer, CuratorCR, so that's where if you're reviewing the same paper again or if somebody else is reviewing a paper that another person in your team has already reviewed as part of a different literature review, that human-validated extracted data will be available for you to call upon, so you don't need to comb through the full text of the PDF again to find out all that information. It's already done for you; it's been done by a human reviewer so there's a trust element there and you can just bring that across. One other thing that I would say too is not so much a feature of Distiller the software platform but Distiller as a company we do have an extensive training suite and also an onboarding process, so, if you're new to Distiller and you would like to have some assistance with setting up your templates and getting those built into the platform. That's definitely something that can be available to you and is a big help in terms of getting started, and getting underway, and getting those templates and processes bedded down.

Ulla Gertig - 38:25
I have to add that Distiller has also a really great help support whenever I really need some help because I just cannot figure it out myself, and I make a ticket, I will have the answer on the same date. I just made one this morning, and I have already received three fly with you.

Ulla Gertig - 38:48
Yeah, I used one of those reusable lists. So, now I have a question for all of you, basically.
I would like to know what complementary tools do you recommend for managing the review process beyond those search platforms?

Dimitriya Bozukova - 39:09
From a manufacturer perspective, I may say that, as Romney mentioned, tools like Distiller are connected to the peer-reviewed databases like PubMed, but we also search in other databases which are not yet connected to the tool, and which where we actually do our manual search, but there's a powerful source of information in the clinical trials databasee, like clinicaltrials.gov and competent authority databases. These are all the standard sources of data which are also recommended as part of the regulatory guidelines that we search. We also have a very extensive communication with people who are assisting events like conferences where they would capture any novelty which is coming which may be Important, not maybe immediately, because it may not have been published yet, but at least would keep us open and vigilant and careful. But of course, on top of the peer-reviewed publications, we are monitoring clinical trials, competent authority databases, guidelines of health associations, books, all of that is input, of course, the review process.

Diane Legere - 40:27
I'm going to say for this question, thanks Dimitra for that, I had to go to co-pilot because of my role, I'm not, I mean I do my own literature searches to correlate with those that are provided but for me it was really, I found something called illicit consensus connected paper so it was a real eye-opener for me because I haven't, well I haven't been in academia for a little while so not doing it but looking at all the tools just to get an idea of what's available and the pros and cons of those to be used. So I'm going to have to leave it to the rest of the panel to talk about other platforms.

Romney Adams - 41:12
Well, obviously from my end, here representing Distiller. Distiller is something I'm very passionate about, but in terms of a couple of other tools that you can integrate and use with Distiller, my other hat is as a librarian, so I'm a big advocate for free access to information and all of that sort of thing. So two tools, one of which has been mentioned by Ulla already, that can integrate with Distiller and also make your life so much easier.
Zotero, that is free, so the reference management software, so for anybody out there who's watching a budget which I feel like is everybody at the moment, Zotero, if you're not using a reference management software, that's a really good option. And then something else as well is a browser extension that some people on the call might be aware of called Unpaywall which will essentially look for free legal versions of full-text PDFs housed in various repositories online. So it can be another way to save time in terms of that process of getting the PDFs into your review. If you're using Distiller as a platform, getting them into the platform. So yeah, big time and money-saving benefits that can come from that.

Romney Adams - 42:18
It's a very simple little browser extension. They're the two tools that I'll often recommend to people as well, as well as Distiller.

Ulla Gertig - 42:35
Super good. Okay, so coming to the next question. What specific tasks in the literature review process are most suitable for AI assistants today? So here I'm very curious about your opinion, Romney, and also your opinion, Diane.

Romney Adams - 42:53
I'm happy to lead with this one. I think right at the moment,  where we're seeing the strongest developments in AI have really been on the screening side of things, on the screening side of that review process. Ulla mentioned in her introduction, the keyword highlighting that Distiller has. So some of that is manual human input, but then we have others as well for things like flagging PICO terms in an abstract, flagging drugs and adverse events that are mentioned as well. So those functionalities are powered by AI, and they're just drawing the reviewer's eye to those key terms that may be useful for you and help you make that screening decision faster. You can also use AI in Distiller to label your references in a particular way, to add contextual metadata to those references that can be useful for you down the track. So things like what type of paper it is, like whether it's just a literature review or whether it's a systematic review or a randomized controlled trial, things like that. And you can even use AI to automate that screening process entirely, and use it as a second reviewer if you're working as part of a smaller team, or if you're under particular time pressure to complete a review you can actually help automate that review process in terms of screening. So that's what I would say the focus has been on in the industry to date, but we're seeing some really exciting movements in the last 6 to 12 months, particularly in AI and data extraction. So helping to automate some of those processes and helping to kind of expedite the extraction of those various data points you need for your reviews. That's something that Distiller has some capabilities in at the moment, and we're working to build upon, but it's a really exciting area for sure.

Diane Legere - 44:52
Thank you. That's a very comprehensive answer, which makes my answer very easy. I was going to say exactly what you said, the searching and summarizing, because to me that seems like the bulk of time that would take in a literature search, especially if you're lucky enough to have a device that has a lot of hits. I was going to say also that the ability of it to keep sort of searching and ongoing search and just not wait a year for you to do your search manually as part of your post-market plan, that it is going on in the background while you're sleeping. And when you wake up in the morning, you can see that there is something that, you know, is germane for your device to take a look at. But for my end, what I see in my reviews that use a product like Distiller is how organized the report is. I love how it's standardized. It's just the way it's presented, honestly, for me, it makes it easy, it makes it pleasurable. I can follow the logic and it puts me in a good mood when I'm reviewing.

Ulla Gertig - 46:06
That's the best part. Then I have now a very specific question to Dimitriya. So from a manufacturer's perspective, what are your biggest concerns when you think about integrating an AI solution?

Dimitriya Bozukova - 46:24
I mean, that industry has to follow certain quality standards. We have to be very careful when we implement new processes, equipment, procedures, etc. So, extensive validation and qualification is key. And of course, the more complex a system is, the more challenging it is to get it implemented from quality perspective, and that's valid also for artificial intelligence systems, not necessarily for systematic literature search, but any artificial intelligence which may lead to decision making that may impact the safety and performance of a device or the safety and the health of patient is very thoroughly scrutinized. So the implementation of digital systems in general is quite a quality-controlled process, and when they emply artificial intelligence tools, it's even more difficult but not impossible. We have now a very huge program in place about artificial intelligence integration at all levels, not only clinical evaluation and literature searches, but any aspect of the business, and there is a lot of empowerment of going in this direction, but of course quality standards have to be covered, risks have to be mitigated in order to avoid erroneous decision making which may impact the safety of the patient.

Ulla Gertig - 47:57
Wow, I'm impressed how many efforts you're already doing in that direction.

Dimitriya Bozukova - 48:01
Yes, and it is very interesting, believe me. It's not only a buzzword. It's really our new normal that we have to implement properly. The biggest challenge actually is learning the AI. So it's an input to Romney and the Distiller team. Once we have trained AI, of course, we are training it with the data which we have today, but it is evolving, and maybe what is right today is not gonna be right or maybe not complete tomorrow.
So the unlearning process is a challenge and a very important one.

Ulla Gertig - 48:43
Okay, great. I think we have one for one last question. So Diane, I would like to ask you from your perspective as a clinical reviewer to a notified body, what are the most common deficiencies that you're actually observing in literature where you submitted as part of the technical documentation?

Diane Legere - 49:03
I'm going to say my number one because I noticed the time. So my number one deficiency is literature searches are much more than find and summarize. What I really need to see and often don't see is a synthesis of that literature that you found. So you found it, you maybe use a tool to grade it for the methodology and whatever, but there's no critical narrative on how this that relates back to an existing body of knowledge, if there's any more gaps, if anyone actually looked at those gaps and did another search, or sorry, another study to try to fill in those gaps. I'm really taking it to the next step. So, you're going to have all those dots that you have of your literature search, if you will, and then you connect the dots and then what does it say? I want to see a synthesis from the literature review.

Ulla Gertig - 50:18
Okay, great. So I think we can already go to the Q&A session.

Diane Legere - 50:25
Just in time to say I have lots more deficiencies if you want, but I noticed the times.
I can only do these.

Ulla Gertig - 50:33
I'm sure you have quite some stories to tell.

Dimitriya Bozukova - 50:38
This will be a powerful input to the new guideline for clinical evaluation, Diane, like any observation from notified bodies, which can help the industry and clinicians to comply, would be welcome.

Diane Legere - 50:53
I try very hard to put my thoughts on LinkedIn whenever, so follow me as I am often giving my thoughts there as well, but yes, thank you.

Sean Smith - 51:05
So we have a lot of questions. We're not going to get to all of them, obviously. And some interesting ones. What is the difference between alternative treatments, generic slash benchmark devices and similar devices?

Ulla Gertig - 51:24
That is a good question because sometimes it's very blurry, actually. So alternative treatments. Okay, let me think of a certain. We have, for example, a device which is treating cancer, and alternative treatments are a liver transplant, or first we have bridge to transplant alternative before you do something, or even just medical treatment. So those could be alternative treatments to device treatment. As a generic device group, you could then have local regional therapies, which could be something like radio frequency ablation or cryoablation or whatever. And then you have your generic device group and your benchmarks from the generic device group which is then a very specific type of therapy then just a certain type of cryoablation and then your similar devices are really the devices which are using the same principle as your own device. So, it's not that you do not always have similar devices. Sometimes you really need to go back to alternative treatments, because there's just nothing there which is treating everything that you want to treat, for example. But that's how we are usually describing the difference.

Diane Legere - 53:13
Can I add to that? I would say that all of them, the alternative treatments, similar benchmark, all come down to the bottom line of benefit-risk. So, when you speak to alternative treatments, you need to talk to me about what populations might be better to use medicine, pharma instead of a device, where those that fail at pharma maybe then come onto a device for a certain age or comorbidity or whatever. So it gives me something to understand the current state of the art and also gives me something to see how other devices are doing. What are the acceptable risks? All devices have risks.

Diane Legere - 53:55
Don't tell me all of your risks have been mitigated and so you don't have any residual risks, because I'm going to come back with a no. So, what are acceptable risks that practitioners and patients obviously willing to accept? And then the benchmark is again how Ulla talked about in her presentation, that's how you're going to meet the GSPR number one that says your safety and performance needs to be a state of the art.
So this device is similar to mine. It has this many risks at this percentage, and I'm going to compare mine, and it's going to be at least as good, but maybe better. And that’s where you're going to tell me why. Maybe the target population is sicker or whatever. So all of what those three come down to the assessment of benefit risk.

Dimitriya Bozukova - 54:48
To complement on both Ula and Diane's response, that really depends on the novelty of the device, of course we have always to screen all these options, but as Ulla mentioned,  some devices are so novel, so you may not have a similar device, a similar device would be typically similar technology, similar functionality, same target population, same intended purpose, same indications. But you may not have such a device, and then you have to screen the other options. What is the current standard of care for that disease or for that indication, and that is your comparator in factor. So, we have to have an open vision on all of those, but depending on the maturity of the technology, some may be more relevant to the device under evaluation than others. For example, for very well-established technologies like needles and syringes, you may not even look for a lot of alternatives, because they have been on the market for so long that they have become the only possible way, or in some cases, there are some exceptions.

Sean Smith - 55:56
In a similar question, if no benchmark articles are available for a medical device, what alternative steps can be taken to define safety and performance acceptance criteria?

Diane Legere - 56:12
Yeah, that happens more often than not, especially as Dimitriya talked about, what do you do with a syringe and needle? Those very sort of standard of care devices. Well, they'll have similars, but not necessarily published, so it still puts you in the same place. So if you're not novel, you're going to have to go back to your own risk assessment and use it and justify it. If you're completely and utterly novel and you have no risk management, to rely on, it is a very challenging position, and you're looking for devices that are in the generic group and as close as possible just to get a sort of guardrails, if you will, and then you're going to really have a top-notch post-market clinical follow-up plan that looks much more frequent. That's how I've managed it with clients previously.

Dimitriya Bozukova - 57:21
Benefit-risk is something that we have to revise at any moment. So it is not something that we fix once and then it stays for life. No, it is throughout the design control process and also from early ideation until certification of the device. This is a very regularly reviewed and may change because maybe during the design process we have new information which may change the benefit-risk profile of the device. Of course, having pivotal studies before regulatory approval is giving you this kind of specific device-specific information, which may allow you to calibrate your benefit-risk and risk management documentation. So, it is not something that you set in stone and it's there for life. It is something that you have to challenge every time and throughout the entire lifetime of a device but again depending on the novelty you may have it as a information from similar benchmark devices, or even if you have a broad portfolio of products, you have your own predicates which are giving you already a lot of information that you can use as an initial input.

Sean Smith - 58:30
So, we're about out of time. I would like to thank the speakers and the attendees for your time and attention today.