About this webinar
Castor CEO Derk Arts joins RTI Health Solutions consultant Ari Nanasakthi and SOPHIA Clinical Research Co-CEO Willie Muthausen to examine regulatory feedback on patient-reported outcome (PRO) and electronic clinical outcome assessment (eCOA) submissions. Built for study sponsors, CROs, and clinical trial teams designing protocols, this session reviews published FDA and EMA critiques across instrument selection, study design, and data quality. The speakers discuss why copying questionnaires from prior trials leads to diary fatigue and missing data, how regulators conduct independent PRO analyses for benefit-risk assessments, and how bring-your-own-device (BYOD) options, diversity action plans, and linguistic validation affect participant compliance and regulatory acceptance.
Key takeaways
- FDA critical comments on PRO submissions cluster around whether instruments are fit for purpose, study design flaws, demonstration of treatment benefit, and statistical or administrative issues, with oncology studies drawing distinct feedback because patients are very sick.
- Copying questionnaires from older protocols or stacking up to 15 instruments in a single visit to satisfy multiple regulators and reimbursement bodies causes diary fatigue and missing data.
- Even when sponsors do not pursue a labeling claim or report no noteworthy group differences, the FDA will conduct its own PRO data analysis to inform benefit-risk assessments.
- Both the FDA and EMA accept BYOD for primary and secondary endpoints when studies maintain equivalence, security, and audit trails that prove contemporaneous data entry.
- FDA guidance on diversity action plans requires sponsors to plan how they will recruit and retain underrepresented populations, which introduces linguistic and cultural adaptation considerations for ePRO administration.
Full transcript
Can we do some quick introductions? So I'm, I'm Derk Arts, CEO and founder at, at Castor, hosting this webinar today, with with Ari and Willie. Ari, could you, give us a really quick recap of who you are for those people that don't know you already?
Sure. Thank you, Beck. So my name is Ari Nanasakthi.
So if you see the g as silent, then you'll get it right Nanasakthi.
And my wife tells me that that's the only thing that is silent about me.
But, anyway, I've been in the industry for about, thirty five or years.
Most of my time, I was at Novartis Pharmaceuticals for about twenty five years based in the UK and in Switzerland. Now I'm in the US, working for RTI Health Solution as a consultant, and, happy to be here. Thank you.
Right. How about you, Willie?
So my name is Willie Muthausen.
Nothing silent about me.
I've been in industry for twenty six years now.
Worked in, different positions, different companies across Europe and the US.
Mostly has been involved have been involved in ECOA or ePRO, collection on the vendor side quite often, but also with two large CROs.
And I'm currently the, cofounder and co CEO of SOPHIA Clinical Research.
We started about three and a half years ago with five people. Today, there's about fifty five of us, and we do everything and anything around Ekoa. And one of the last things that you've probably seen is we just released a a new product, library. Basically, we call it Gloat, the greatest library of all time.
And as you can see, tongue in cheek, so we're we're, you know, trying to have some fun. And happy to be here today with with Derk and with Ari. And Ari and I both call each other grumpy old man, so this will be a good session here today.
Yeah. So I just asked you this question. So I believe you've known each other for fourteen years. Is that right?
Yeah. Fourteen or fifteen years, something like that. Yeah. Yes.
That's nice.
So fourteen fourteen years ago, I was, like, halfway through med school, I guess. Something like that.
Now you make now you make us feel old.
No. I mean, I don't know. I'm just I'm just young. That's the I think that's how you should look at it.
So, tell us a little bit more about GLODES. I'm I'm very curious. I mean, I love the I love the toe in cheek, name. What what what's the main problem GLODES is going to solve for the world?
So, yeah, one one of the things that we've seen over and over again, that we have delays with, study starts, and that is the licensing and the implementation and getting the translations basically. But licensing, contracting, seems to be a problem. So we now have a library, that basically will help over time to, alleviate these problems and make it quicker. It's more like, Amazon for questionnaires, basically, where we have a lot of metadata, a lot of information about that, and it's a self-service or a managed service, project.
So Beautiful.
And so you're going to aggressively cut back the time it takes to to license and and validate questionnaires.
That's the idea. Yes. That's what we're already doing with some of our partners. Yes.
That's obviously, I think, something that every single viewer here is gonna be very excited about. So, make sure to to hit up Willie if you wanna hear more about that.
But enough advertising for for one session. Let's go into, the meat of the of the webinar. Alright. You're you're a prolific, academic author, and, there's a paper you published not too long ago where you and your team or your colleagues, spoke about the critical comments that the FDA had about, submissions that included PRO data.
I thought that was very interesting. It was kind of the foundation for, for this webinar. Although in our prep call, of course, we touched on many more topics, but I I do feel it's an a great foundation for a webinar because it's, it really branches out in so many different directions. So, Ari, could you give us a quick overview of this paper? What did you do? What did you find? And, where do you, where do you think we should start?
Yeah. Thanks, Derk. So just to kinda start off, where we started off from is that, many companies or, my clients, they submit the data to the regulators, FDA, and the EMA, and then they get feedback, saying what's good, what's bad, and all the rest of it. And as the industry didn't have a compilation of these comments, published comments, anywhere. So this is that that's the impetus for, publishing this paper. So just look at what sort of comments did the critical comments, the FDA made.
And if if there is a if there is a theme, there's a pattern that emerges. So that's that's that's what that's what we did. And so it sort of fell into five or six categories, like, if the if the questionnaires were fit for purpose, if, concerning the study design, if was there a treatment benefit shown on the PRO questionnaire?
And there were some administrative issues and statistical issues as well. So, so, anyway, so we kind of published it. One of the main thing that came out there was that the the the comments relating to oncology was somehow very different, not very different, markedly different, compared to the other therapeutic areas, because oncology is is, has its own, kind of quirks in it, not only the study design, but also in terms of, collecting patients reported outcome because patients are pretty sick.
So, anyway so that's that's roughly, where we can start off with, Derk.
Yeah. So I think that that fit for purpose comment is actually very interesting.
Could you speak a bit more about that? Like, what what does the FDA consider fit for purpose?
When I read that, you know, if you ever have a bleak view of of, a comment like that, you could read it as well. The FDA doesn't doesn't actually think that this that these data are useful, in the submission, which seems a bit bit bit harsh. Right? So there there's, I think a lot more subtlety there. Can you can you speak about, can you speak about that? What do they mean when they say that?
Sure.
So before we go to the FDA or regulators in general, we could we could think about, we could take a look at it from the sponsor standpoint.
So when when we have a study protocol, study starting up in certain disease area, you know, you look for something that's readily available, so you don't have to spend a lot of money and time coming up some something new, and, make use of that. Now not all sponsors all the time for the disease areas won't label.
They know that it's going to be a high hurdle, especially if it's a secondary endpoint.
So they will include a questionnaire that captures most of what patients perceive to be burdensome, for the purpose of publication or for market access, not necessarily to please the regulators.
But still when you give the data, the regulators will make a comment on it. They will say, well, this instrument is not fit for purpose.
But, yes, it is not. And it shouldn't come as a surprise to or it doesn't come as a surprise to many sponsors because they knew that. Because all they want is some evidence that supports the primary endpoint that can also be published, to make it impressive. Now, if you have a primary endpoint as, then you got to make sure that your questionnaire or the instrument is fit for purpose.
Otherwise, you will not have a drug, rough more or less, roughly speaking. So although there there are all these comments, it's it's when these clients or the sponsors will see them, it shouldn't come as a surprise. People know beforehand that a certain SF thirty six or certain questionnaire is not going to cut it with the regulators. They know that.
You'll be surprised if they if if they find a comment from the, from the, regulators, to be critical, and if they are surprised, then you're in the wrong job. You shouldn't be surprised.
So so you're basically saying, often instruments are selected based on their availability, how well known they are, you know, how well they may be resonated with the KOLs even though they might not support, labeling claims. Is it?
I mean, sometimes what you see Yeah.
Because, I mean, sometimes we use or the clients' sponsors use a certain questionnaire because that's what was done before and before and before.
So you can compare the findings in some conference, and you say, well, my drug shows to be better. So there are many reasons why, PRO or OBSource, I mean, in in pediatric studies, it will be, observed or reported outcomes are collected not necessarily for labeling, but to be consistent with previous findings.
That is not always a good idea because all you are doing is repeating someone else's mistake sometimes.
Yeah.
But, anyway, so there there are different reasons why, sponsors use a questionnaire that doesn't quite hit the regulatory standards.
So, Willie, is this also something you've seen in your career? I mean, you've obviously been very close to to the selection of instruments.
It's been your business for for a long time.
We we see that we see that too. We see it on a regular basis. Well, it's just like Ari said. Quite often, we find that, there's a lot of copy and pasting going on, between protocols within the organization or also across other organizations, see what the competitors.
And in oncology, quite often, we see that and some I've seen protocols with fourteen, fifteen questionnaires to be done by a patient in one visit. So which then basically question is, how much good data are we still getting from questionnaire, whatever, seven to fourteen, the last ones in in that list. And not all questionnaires are equally long, so some of them are rather short, and that's probably okay. Now this is rather long, but it's we see quite a bit of of this one as well, that there's too too many endpoints not even endpoints. Sorry. Too many questionnaires, being thrown at patients.
Yeah. So if if this plays out like this, is it fair to say that this the data from this bureau that these patients have put so much effort into complete or hopefully complete, and then we'll get to the data quality and reliability later.
But but the actual patients using the drug won't actually be exposed to that information. Right? Because farmers aren't allowed to use it in their in their materials or at least in their in their claims. How what is the what is the subtlety there?
Like, what is the we go from, okay. There's a scientific publication. Obviously, you can report it there. So, of course, the people who actually read these scientific papers, which is, I think, ultimately a subset of the medical community.
You know? A a large portion of the medical community just reads the clinical practice guidelines and never looks at the underlying scientific papers.
So, like, how does it ultimately how does these these data that that that these patients have been burdened with ultimately make its way into, into the real world in terms of, the your your your primary care physician being aware or of of that information or even the patients, you know, making a choice between two drugs? How does that play out?
I'll take that, Willy, if you want.
So if if a claim is not in the label, then it cannot be advertised or published, in the US. It can't be done anywhere in Europe, but certainly in the US.
So it's it's left to publications for people to read. But, recently there have been developments, or guidance that came from the FDA, which is still not very clear to many sponsors is.
To be consistent with FDA labeling, you can promote it through your website.
For example, you can you can promote it as long as it meets a certain statistical standards, a scientific and statistical standards. Now, we don't know what that is. So, you know, there there is an ambiguity there about what actually meets that standard, but people do. I mean, we have examples.
Standard, but people do. I mean, we have examples. I might be able to show one, where sponsors have, made use of that sort of evidence to promote. I can I hope I can share this? Let me let me quickly share this example. I think it's pretty good.
Are you seeing something?
It's yeah.
It's coming up.
Fortibda?
We're seeing an elephant.
Okay. There's an elephant. Now the, I mean, the reviewers made a positive comment about tolerability of this cancer drug, and it's actually promoted, on the website by the company, and we took a look at it, and it's it's perfectly reasonable to do so.
There's a lot more to this than, what what you see here, but it's consistent with the FDA labeling.
So, you can actually make use of messages or claims that meets certain statistical standards, even if it's not in the claim.
Now if you take, you know, one of the very, highly prescribed, cancer treatments, it has something like twenty, Nivolumab, I think, twenty different, indications, lung cancer, colorectal cancer. So, a label goes into pages, like, hundred and thirty odd pages. So you can't you know, you the FDA doesn't want to keep on adding pieces in it. They say, look. If you have something good to say, go and say it, but it won't be in the label. So that actually works in favor, to the, sponsors. So they they do things like this.
And there are people who go slightly overboard and make claims on things that are very, very weak, in evidence, but that's left for someone else to police that.
So it's possible.
Interesting. So given the the downstream impact, do we think it's ethical to to add this additional participant burden to to study, or should we actually spend more effort on making sure that, these data really have an impact on on these important decisions?
Willie, what is your I mean, this is not there's not a right or wrong question. No.
There's there's a lot of gray between black and white. Right? So, I think quite sometimes, and, Ari, correct me if I'm wrong, but sometimes, I think, sponsors, when they pull these protocols together and these endpoints and these questionnaires, don't necessarily know, that's why it's an experiment, what the result will be. So they don't necessarily know what the outcome is and whether it's something you wanna shout about or not. So I think it's it's a little bit of a difficult decision to a certain degree upfront to decide what to ask and what not to ask.
And I get the impression that, quite a few of of the sponsors air on the, let's say, positive side or what they perceive positive by adding more questionnaires to that rather than reducing the questionnaires to the to the necessary, questionnaires.
And that's when you end up with a lot of data that probably never even see a publication, you know, that never goes anywhere. It's just sitting in some database.
Mhmm. Mhmm.
So what do we well, how do we prevent that from happening? Like, what is the what is the advice to the audience here to to minimize that risk and to have a strong story also to all the patients that participated in the trials and say, hey. Your your data wasn't collected for for not.
Yeah. Well, it's it's a I don't know if the right phrase, the double double edged, sword because, most of the time now, we have there's very little time between phase two and phase three studies, phase one and phase two. The timelines are very short for, for people to reasonably go and make a, informed decision as to what sort of questionnaire to use and so on and so forth. So what tend to happen is, like I said earlier, you go and see what the previous guy has done.
What did Pfizer do three years ago? You know? What is what did Merck do? And then you copy that pretty much.
And sometimes it's it's a right thing to do because you want to know what the competition has done. But to copy what other people have done is as I said earlier, it's just repeating someone else's mistake. So that's that's one.
Two is that when you don't have enough time, you try to put in as much as possible. Just throw in, hopefully, that, you know, the dot will hit something, to give you publications. But the the end result of that sometimes in place in within fields like oncology where patients are very sick or in HIV, Patients are very sick that you end up with a lot of missing data. So you don't have a lot to say, even if you have something to say. You may so trying to be focused, and collecting information only what's required, is is a good thing, but often people don't have time to make that decision.
And the third point, the and the final point is, it used to be the case, when I started many, many years ago, is that you just had the FDA and the EMA.
Now we have the FDA, the EMA, the EQIG, the NICE, all sort of stakeholders need to be satisfied. So we include questionnaires or the sponsors include questionnaires not targeted at regulators, but targeted at some, reimbursement body.
So you have all these multiple questionnaires to make different people happy, and they they have overlapping questions. So you ask, how's your pain? How's your pain? How's your pain? In different question questionnaires, and that sometimes upset the patients. It's like, oh, you just asked me why.
So it's it's it's it's a difficult short difficult choice. It's there's no easy answer. I mean, there there are, forces that against you and for you, and if you don't include something, you will upset Germany. If you don't include something, you might upset the French.
So nobody wants to upset anybody and.
Like someone said, your name is not Tequila to make everyone happy.
Okay. Someone has to go. And so my suggestion usually to, sponsors is that you can't make everyone happy all the time. You have to make a choice. You either going to have good data from patients, from good instruments that's going to maximize your value proposition, or you try to make everyone happy and no one is going to be really happy because you won't have much to do. So it's it's a it's a difficult choice.
Interesting. I I I wanna go a little bit deeper on good data, and you also brought up data quality. It's actually, I think, the third point mentioned in in your paper, twenty point six percent relates to issues with, with data quality.
And we leave also to read the paper. How do you how do you reflect on that section, and and what do you think is, you know, leads to to to that that feedback? And then what could we possibly do about it?
So I I think, so let me just think about that.
One of the issues still is that there's too many, questionnaires or too many questions per day with the diary of that patients or too long of a duration that they have to have to, engage with an answer.
And then you get, what we call, diary fatigue or, co op fatigue as well where patients just don't want to do it anymore, and can't do it anymore.
And, again, different therapeutic areas have have different numbers attached to them.
My point always is when we have these discussion with sponsors, not necessarily just the, regulatory point of view or the reimbursement point of view, but it's also, do you really need that data, and do you think patients will be able to answer all these questions? Because like Ari said, you can have as many as you want. If if nobody answered these questions consistently over extended period of time, then you're just missing missing the point. We have, studies where, for example, as an anchor, you have the, questionnaire or a couple of questionnaires just before, you go into treatment or placebo, whatever it is, and then you answer a couple of times afterwards.
If that anchor is missed, then you have nothing to compare it against. So what we also see is there's sometimes not enough attention paid to the actual, compliance rates, for example, or the process. And sometimes that's has to happen at the site. Sometimes it has to happen, remotely, so we need to remotely monitor that.
But, our point always is, if you want that data and if you think it's important to you for whatever purpose it is, then you just need to treat it accordingly and just pay attention to the details and and basically, spend time and effort on that. This this is one one what we think is one of the key things that not enough attention is paid to the rollout and the decision which ones to use and how often.
Do you need to have do you need to have a questionnaire on every visit, or would every other visit be good enough? These are things that I don't see enough discussions around.
Yeah. Are we doing enough research on these things? I mean, I'm I I already know as I said from a previous webinar, I'm a huge fan of these, questions, like the bother type questions that actually roll up a lot of information in a single answer, which obviously introduces very little burden, but also very little precision.
But my sense is always that in the end, though the human perception, the human experience cannot be broken down into all these super detailed, underlying questions. In the end, you know, the human experience is just something that is very subjective anyway. And so it are we doing enough research and actually comparing those approaches with we I think we've gone pretty far down the rabbit hole with these super structured extensive questions and try to capture every single domain. Well, my sense is always, look.
If you just run, you know, an additional arm and you you you capture a fifth of the questions or even less, do you really get less information out of that? What do we think is underneath that? Do we think we're doing enough comparative research to actually figure out if we could do less and have the same quality of of information in the end? Because what matters is how does it impact the real world?
And often I feel like once you go downstream, downstream, downstream, the impact of the real world on all this on this granularity might, you know, start to fade a little bit. What's what's your perception of that with that, Ari? Are we doing enough? And and Yeah.
What do you think?
Well, I think, I mean, people are doing, I think people are have worked and keep on working on trying to come up with some of these, answers to some of these very difficult questions.
Sometimes you do need to have granularity in, you know, what what's going on, because that's the way the patients thinks. And it also provide the type of evidence for the regulators to carry out benefit risk analysis. Now this is the I think lately, I've come to realize this is very important. You know, if we want to talk about you know, people tend to talk about it. We want a label. We we don't get a label, so what's the point of it all?
That is not the case because without, data from patients, it's very difficult for regulators to carry out, benefit risk analysis.
Now if you if you if if there is a situation where you can ask one question, sort of, overall, are you feeling better or worse, la di da, a sort of global question, sometimes that's good enough, and sometimes it's not because if you have a placebo controlled study, sort of a loose kind of questionnaire would be okay to pick up differences. But if you have a comparative study, two treatments, close to each other, you you need granularity to be able to distinguish what is it that is separating the two two drugs. And also remember that the even if the sponsors don't, provide the evidence, we have seen many examples where the FDA has gone and done their own analysis anyway. So sometimes there's a perception that we won't you know, we'll hide it. We won't show what we have.
That doesn't usually work. Let me show you show you another example here.
While I was pulling it up, I also just wanna invite everyone. Ask your questions in the q and a. We'll see what we can answer live, but we'll definitely keep some time at the end. So there should be a q and a button at the bottom to, to allow you to ask your questions, and we'll try to to address all of them.
Right. So you probably see, a screen here now. This is the, this is this drug, for breast cancer.
It was approved in two thousand twenty three. And the study sponsor during the meeting said there were no noteworthy differences between the two treatment groups and not, and no noteworthy changes over time. So it's like, look, there's no no big deal. And the FDA response was, look, We performed additional PRO analysis and interpreted the, entire on the entire trial population.
So they went and did their own analysis anyway between the arms. Okay? And they said, look. There's decreased appetite and nausea And as were measured by question and they said, look in your data, the majority of patients in both arms reported any, frequency of, nausea.
Rarely occassionally. So which is good. It's like, yes. You have these issues, but most of most patients are saying it's it's rarely occassionally. So it's not a big deal. So that's kind of a a positive finding, if you like, that the sponsor didn't highlight.
And this is the sort this is where the FDA says, look. We'll do the our own analysis if you don't tell us. And you don't want to be in that sort of situation.
You don't want the regulators to go and start digging in your data, because they will find all sort of things. So it's definitely proactive. Yeah.
That's an that's an interesting statement, but, I see I see where you're coming from.
Yeah.
But I I, I guess, we should all be we should all be ready for the regulators to do just that, I guess.
Yes.
Well, the outcome may not always be as positive in this case. Right?
Yeah. Exactly. That's right. Yeah. I mean, go try to Yeah.
But, I mean, you know, that's what they're there for. I mean Mhmm. Like, let's be honest. You know, you can spend, you know, a hundred million on marketing a drug, but if the drug ultimately, you know, isn't used, you know, because patients don't don't like it, then you've wasted a hundred million on marketing it. So, I mean, maybe maybe the FDA is doing you a favor, to to stop you from spending so much money on, on that particular activity. But that's a fascinating example, Ari. I I actually wasn't aware that that's, that that happened.
So, so good to good to be aware of and good to keep in mind, I guess, when you're designing a a study.
So maybe we can talk a little bit more about the, you know, reduction of, of burden for for participants. So we we spoke about just reducing the number of questions, you know, already. Obviously, that's, that's that's an open door. You know, the the fewer questions, the easier it is.
We can we can also look at the scheduled events and see what is it really beneficial to ask these questions every visit? Or, you know, if you do a sensitivity analysis, you show that if you ask it every third visit, it basically yields the same, results. What about some of the other aspects of, of of, PROs? Now, obviously, you know, I'm slightly biased, but, I would say that it might be easier to answer, PROs digitally as ePRO on your own phone or on the phone that's provided by the study versus paper and not having to send paper back.
Or I would say that, push notifications or text messages are are are nice just to make it easier to remember things. I mean, this burden also captured in, you know, the fact that you're always behind or if you remember, like, oh, shit. I I still have this stack of papers I need to fill in and then lead to a parking lot effect. Like, what are the some of the other spec aspects we should think about in terms of, reducing burden and with that increasing data quality, you know, coming back to the the FDA?
Why don't you take that limit? Because yeah. Go ahead.
Yeah. Sure. So look. I'm I mean, you probably know I'm all about that, reducing patient burden by giving patients a choice.
For me, it's always we can reduce the burden from a technology perspective by letting patients choose which system or what device they're gonna use. Do they wanna use their own device? Do they wanna provision device? Do they want to use the web based solution?
In the future, probably, they wanna use a voice assistant like Alexa and others.
So we've we've done a lot of work to make sure that, there is that we know the mechanics that would allow for equivalent data across different platforms and published a lot about that one in the last ten years. So, I absolutely believe that patient burden is not just purely the number of questions and the number of questionnaires that they have to fill in, but also how we ask them and when we ask them to do that. But, currently, you're also working on a couple of, AI algorithms that will allow us to basically find the sweet spot for patients, when is a good time for them to answer the questionnaire, with within the confines of the protocol, obviously, time windows.
But, again, I I'm absolutely sure, and I believe and we've seen evidence that, when we started to do more BYOD, that the compliance numbers, have slightly gone up. There's still not a lot of data that has been published in that space, so I wish we would do more work in that. But, you know, patient's choice. I always say that I'm I'm known for being mister BYOD, but I'm not really.
I really, not all about BYOD as well. Patient's choice. That's what's important. BYOD is one of the choices that patients should have.
Yeah. That's that's I think it sounds terrific. I do have a question about that. I also wanna point out it took us thirty three minutes to first mention AI, which I think, you know, pudus to us That sounds good. To, to to making it last that long.
So on this topic. Right? So I I'm I'm a huge fan of of just some methodological studies.
I was trained as an epidemiologist during my PhD. You know, it does, like, the science that makes the science better.
How should one do that? Because, obviously, we wanna spend all our money in actually getting, you know, pharma, some medical device companies on getting a new, you know, approval, getting a new drug or device to to market. So now you're in a situation like, oh, it would be so interesting to see. If we just do provision devices versus BYOD versus paper.
Let's see, you know, who has the highest compliance rates. However, of course, there is some form of bias that you don't introduce. Right? I mean, it's almost unavoidable that there's gonna be bias in in in in in responses.
Specifically, I think if you compare paper to to to digital when it comes to recall, for example.
So what like, how who who's going to, like, do that? Who's going to pay for that? Like, what's why would a sponsor consider doing a methodological, like, SIP study like that while when they just wanna spend their dollars on, you know, getting getting approvals as quickly as possible?
So some of these studies have already been done. Right? Yeah. And I I feel sometimes that, as an industry, we're we're forgetting some of the stuff that was done twenty, twenty five, thirty years ago. Right? So we have enough data to compare paper versus electronic, and we know that retrospective data, entry on paper is a common theme. And that's why in the end, FDA and other regulators have opted to support ECOA or EPO solutions, where we have an audit trail with a time stamp so we know when patients answer the questionnaires, or the the questions.
So sometimes these things get forget people seem to forget about that, and there's new people coming into the industry that probably don't know that that happened twenty five years ago. And that was published in the BMJ, if I'm not mistaken, or or New England Journal of Medicine. So there's a lot of stuff out there. Again, long time ago, but we still need to probably bring that bring that up again. What I'm seeing now quite often within, our company, what we see is sponsors wanting to allow retrospective data entry, which basically brings us back to where we were with paper, except now we have it digitally. So that doesn't really do anything for anyone. Now there are some there are exceptions, and there's, you know, reason sometimes where it's a good idea, but mostly, it's a bad idea.
And, there's other, study. Like, we've done the big study, where we did the equivalence to show that paper, tablets, provisions, and b by r d is all equivalent, can work.
So a lot of these studies have been done, and they have been published, and they have been presented at conferences.
But we probably need to do more, on on the education to to bring these up. Right?
I think if you tell me, look. I have your fifteen year old paper here that compares EPO to paper. I would be like, well, we probably need to redo that considering how EPO has has changed so drastically. Right? So that's the, I don't know. Even the challenge.
EPO was superior. Why do we have to show that again twenty years later? It's still superior. It's inherent the way those things are being rolled out.
I know, but I think it's also perception. You know?
I think Okay.
Yeah.
It's it's just like, well, is that still valid? You know, people will ask those questions. That's just a problem with science in itself. Like, people, I think, struggle to, like, if when I was a PhD student, my supervisor you know, if I include, like, a twenty year old paper, I'd be like, well, you probably need to find something newer.
Well, but this is a great paper. Well yeah. But still anyway, that doesn't matter. It's a different discussion.
There was a question here that I think we can fit in here. Well, maybe, Ari, you can jump on it, or or Willie if if, if you're comfortable.
So the understanding of regulatory acceptance of BYOD, ECOA data, for primary and secondary endpoints. Any any comments to to add to that?
So to my knowledge sorry. To my knowledge, there is no issue.
There have been examples of, sponsors using BYOD for primary and secondary endpoints.
You just have to show that you've done the right thing, and you didn't just you didn't just, let everyone lose do whatever. I mean, you there's a process, and as long as you stay within the regulatory framework about data protection and security and all the rest of it, There there's there's no other there there there's no reason why not. Willie, go ahead.
Yeah. No. A hundred percent agree with you. So all the interactions that I had look. I've been doing research for ten years with with several teams, with within our organization, other organizations.
And we we discussed that with the FDA and people at the FDA from the very beginning to bring them along and make sure that we don't, pursue something that they're gonna not approve of or that they have issues with. So we took them along, informed them on a regular basis whenever we get studies.
We even, presented at a DA meeting with staff from the FDA, and now you presented at the PRO workshop, in DC many years ago as well. So the regulators have been brought with us, on on a regular basis.
And, similar to what Ari says, I have no negative feedback, in general. There's some concerns about data privacy and all the other stuff, and that that's all good. That needs to be sorted out. That's a vendor problem, though. That's not a data problem.
So that all all work pretty well. And the EMA guideline from last year, basically has a whole chapter on BYOD. So, it's it's now acknowledged by the European agency that that it's a thing, and it's okay. There's certain concerns and certain things need to be addressed, but it is fine to use that.
Yeah. No. Just, I think a very clear and crisp answer. And I think, just let you on to the vendor comment a little bit. I would assume that in these modern days, just like Castor does it, data would be encrypted on the device and would also potentially leave your device. At least that's how we do it once we have confirmed it is now safe in in the cloud.
And, you know, I feel like the technology has come far enough to to really make sure that data privacy shouldn't be an issue anymore using, you know, the modern biometrics authentication on the device and, you know, the things I just mentioned.
So I feel like that problem has also been solved.
Well, let me let me I don't disagree with you there, but it's just some food for thought.
Ten or fifteen years ago or twenty years ago, there was five EcoA vendors. Now there's probably a hundred and twenty five EcoA vendors. So we can't assume that all of them coming from other industries and other areas that they always know what they need to do. A lot of them, when we talk to them, we find that they don't even know that there's an FDA guideline or there's an EMA paper. So it's it's education again. Right? So, technically, there should be no reason for anyone not to do what you do, but that doesn't mean that they all do what you do.
Yeah. Of course. Of course.
Why don't we take some of the questions, before we jump to jump forward?
Yeah. I agree.
I think, actually, as we're on this topic, maybe, Martina's question on sort of the latest, from the FDA, which obviously is also, you know, just a tie up for the webinar. But, I do think it would be good to kinda reflect on what do we think has changed in the past twelve to twenty four months, in terms of what the FDA has put out. There was just, of course, some some new, you know, DCT guidance that came out, that also covers some of these aspects. But, Ari, Willie, any anything that stands out from you in terms of how the regulars have changed their stance potentially? I mean, BYVD, we just mentioned. Anything else we should be, we should be aware of?
Yeah. Let me take this, William, then you go for it.
Oh, yeah.
One thing that's, which is peripheral to the topic that we are discussing, but I think it's important, is that, I think couple of months ago, there was a guidance on diversity action plans to improve, enrollment of patients. Okay. So the the traditionally, so far, sponsors, included patients who are easy to access because then we can recruit, retain, and close the study fast.
Now we've been asked to, and I think that's the right thing to do, to bring in populations that are in the margins of society.
Okay. So how this is new. How they're going to how the sponsors are going to do it? Because according to the guidance and for people who don't know this, according to according to the guidance before, program starts, before the study starts, the sponsors have to submit a diversity action plan. How are we going to who are we going to recruit? How are we going to, recruit them, and how what are what are our plans to retain them?
So I'm thinking about in the US, you know, patients who are in the middle of the country, miles and miles away from the nearest health care center or, doctors, or cancer treatment centers. How are we going to reach them? Are we going to deal with, native Indians? In Canada, people who lives right up in the north, often left behind, Aborigines in Australia.
Now bringing in all these things that we are talking about, EPO, BYOD, and all the rest of it, this is uncharted waters.
I don't know if they have access to some of these things, how they view this new technology. I did read a paper recently where this is referring to, Aboriginal population in Australia. They are visual people. They are not words.
They are not they they don't write and read. I'm sure they can, but in terms of communication, they are visual people. And the the study team administered the promise twenty nine questionnaire. It's not like s f thirty six or something that's old.
Promise twenty nine. And they said most of the words and the sentence structures, they don't understand.
So this is the sort of thing I think we're going to be facing with, going forward.
Yeah. I think that we can kinda tie this to, an anonymous question on, linguistic validation as recommended by ISBOR and translate it into all languages. You know, when we discussed this ahead of our webinar, you know, I think it raises a very interesting point that, a, how people interpret language can be very sort of culturally diverse.
Mhmm.
But then, b, and I think, Willie, this was your point, so I'd love for you to to add some more color to that. If I understood it correctly, there's then been the research been done that's okay. Let's say we have ten ten types of English. Right?
So we thought we had one type of English. Now we need to include a much more diverse population. So now we have ten types of English. So ten translations.
Well, this great news for translation centers, obviously.
Now I think you mentioned the study that they concluded that the outcome didn't actually change between, between these different variations of language, but maybe I misunderstood that. Could you could you give some some comment on that?
So, this is also one one of these topics where there's a lot of gray between the black and white.
So similar to when we talk about BYOD, there will be slight there is the potential to have slight difference how patients perceive that and answer the questions.
And that generally then disappears if it's not too bad in the noise, in the background statistical background noise, basically. So these are differences that don't matter.
When we look at linguistic validation and translation, so there's there's a recommendation by ISBOR from two thousand five, and there's a, I think, two thousand eight or nine cultural adaptation guideline from ISBOR as well on on similar topic.
And they are not, regulator positions. But having said that, it's the gold standard at this point, so everybody does that in general. But what we see quite often is that when the timelines are tight or when the budgets are tight, or when the authors are kind of a little bit more, relaxed about linguistic validation translation, that we don't always follow the full cycle of what's described, in the ISBART task force.
And depending on what you want to use the data for, if it's a primary or secondary endpoint, you may be a little bit more, tight on on that and do it do it the way it's described.
And quite often what's being led left out is not the translation itself, but the cognitive debriefing step because that takes time, and it's probably the most expensive part of the whole linguistic validation part.
And it's, again, it's a risk based approach. Every sponsor needs to decide for the endpoint that they want to capture, what they wanna do with the data, whether they wanna go the full circle, or whether they're willing to, do something less stringent.
And sometimes it's not the auth sorry. It's the author's choice. It's not even the sponsor's choice. The author will tell you what has to be done because it's their instrument, it's their IP, and and they'll defend that.
And sometimes they're a little bit more lenient when it's in late phase studies, and sometimes they say it's a phase three study. So, no, you have to go through the whole process. And by the way, we pick the vendor that you have to use. You can't just pick any translation company.
We tell you who our partner is.
So on the other extreme, we have these things that we have ten, seven, ten, eleven English version sometimes.
And, we're just in the process of running a project where we compare these English versions and just find out, do we actually really need them? Right? There sometimes when like Ari said, when you take, you know, subpopulations in in a larger population and they have a different lingo and different dialects or or different way of communication, then it may well be the best way to have a separate English version for that population.
But we have evidence, that, for example, you can use US English in the UK and vice versa. Right? So they change the zed for an s, and that's mostly it. There's a couple other words sometimes, that you have to look at, but that's more cultural adaptation. Like, there's a questionnaire where they they, the question is, how well are you doing with, carrying a gallon of milk? So a gallon of milk won't do anything for anyone, probably in the UK, although I'm sure they can kind of assume what that is.
But so, again, there's there's not there's not a straight answer how to do it best, but what I think is what we need to do, and then that's what we're gonna do, is shine a little bit of a light on that one and do some research as you suggested earlier. Do some research and publish that and find out what's going on. Right?
Yeah.
I think that's critical, and I'm very grateful that you're contributing to these types of activities.
We have a lot of questions coming through. So even though we have a bit more of the, the regulatory sorry, the, the regular webinar flow that we prepared. I think we can try to get through a couple of these questions because we we've probably run out of time. So, Ari, maybe do you wanna pick one from the list, one that speaks, the one that speaks to you?
You're muted now.
I'm sorry.
You know, I can't pick, this question is about AI and machine learning playing and analysis of ePro data.
I haven't seen anything.
I haven't seen anything. I have not heard anything about AI playing any part.
We are still not there for artificial intelligence. We are still dealing with natural ignorance.
We have a long way to go, so I don't I don't see that coming in any any sooner.
In terms of data validation, I think, same person, Vishali, asked another question.
Data validation, how should sponsors ensure the quality and reliability of PRO data collected through electronic method, compared to traditional paper? Now like Willie said, the we've done this many, many times over many years, that we we really shouldn't be talking about paper at all. It's like we don't talk about steam engines anymore.
Okay.
Or rowing boats. I mean, we should be, you know, we shouldn't we shouldn't go there. This is, I mean, to to me, electronic, forget about EPO, just electronic, methodology, on about data capture is kind of the natural thing to do. We shouldn't be comparing it with paper.
I mean, I I was waiting for the milk marketing board of England, I don't know, thirty years ago, and they were collecting electronic data capture of how much fat content in milk. The farmers were providing information to veterinary surgeons. This is back in the late late seventies.
So as a pharmaceutical industry, why are we still talking about paper PROs?
It it beats me. Anyway, I'm Paper sorry. Willy, you take over.
Paper PRO, if it comes up, it comes up as a backup solution. Right? Yeah. Saying, hey. What if the technology fails? And and, look, technology, like, every technology has its ups and downs, but every now and then, you do have failures, device failures, network failures, and so on.
But that doesn't necessarily mean that we have to use paper as a backup. Right? So most of the apps that are out there have an offline capability.
And if there's a failure on one device, when we do BORD, they just grab somebody else's device. And if you have a web based solution, you can use any device. You can use your laptop. You can use somebody else's laptop. So there's there's enough backups already from my point of view that I don't think we need any paper backup.
There's, from again, from a regulatory perspective, the the FDA guideline from two thousand and nine, and the new one, the draft one out there as well. They they also say that, in two again, the original one that, the regulators will review if the patients did the questionnaires when they were supposed to be doing this. So contemporaneous is the is the keyword there. And on paper, you can't prove that. So if you can't prove it, you have to assume that it's not done. And that's where we talk about the parking lot syndrome, and that brings us back to that twenty five year old publication where that was clearly shown that patients, when it's on paper, they just do it whenever they have time, whenever they want to do it, not when they're supposed to be doing it based on the protocol timelines.
Yeah. Exactly. Exactly.
Yeah. I mean, there was, another question, Derk. I just want to answer that, about retrospective Retrospectively.
Now there are two kinds of retrospective, so I don't know which one the, the the question refers to. One is asking someone, what was your pain last Wednesday? I mean, you think back. Well, that's one issue. The other one is, you you missed the visit or something, and, you, therefore, retrospectively asking the patient, two days later.
So in in the in on the on the first case is, proactively, you are asking what was it like when you started the study or the last visit.
Okay. There's recall bias and all the rest of it. But if you miss a visit and then two days later, you go and say, you know, how many times did he go to go to the bathroom? Then, again, it's going to be impacted. So if the questionnaire is to is to capture things that happens there and then, it should be, ideally, should be captured. Some things you cannot ask every day. You know, when was the last time you had a hospital visit?
You ask every two months or three months because it is a one off thing. So it depends on the context, of of the situation.
But if you can avoid retrospective, data collection, that's a that's a good thing.
Yeah. Okay.
Yeah. And, Willie, maybe you can speak about this, I think, Nikesh, I think the question is, about using text messaging to, to capture answers. I think this is quite an interesting question. We actually at Castor also support WhatsApp. So instead of you know, there's one you can deliver the URL to, to a web page that is, you know, the the ePRO, tool. And, you know, that's one thing Casper does. But you can also, of course, send the question directly as a text message and then process the answer that they just answer as a text or in WhatsApp captured the information.
And so the question, I think, is around, like, comparing this. So this is for a community that doesn't have smartphones. So you could use a a regular flip phone or old Nokia to basically answer your questions.
What do we know about that? Like, how do how do we think about the the validity there? Because, obviously, it has benefits when it comes to accessing a population that that, is economically disadvantaged.
So when we, when the consortium started many twelve, thirteen years ago, that was one of the issues that we tried to highlight and and tried to get an answer to by providing the guideline or the white paper. And we did some further publications, with within our organization, to show, what you need to do to keep the formatting close enough to the original that you can assume or you know that there will be no bias introduced to that. And this is, for example, one of the issues that we need to look needed to look into was scrolling. Right? So and you have scrolling when you have a text message, but the scrolling question has been answered as far as I'm concerned. So there was a pretty big study was done, by the Ecoa consortium.
And there was another study. It was done with a group of other people that found that scrolling itself, does not really, if if done right and even if you do it wrong and there's different ways of doing scrolling, it's difficult to mess it up. And this goes to what we've been saying for a long time as well. Patients are patients.
That doesn't mean they don't think. Right? They don't get it. Right? So scrolling is is a common theme these days.
Everybody has a phone. Even a flip phone, you have to scroll some of the message because they're longer than the screen is is is, is long, whereas high, height of the screen. So, I think that the scrolling part, we can play. That's probably okay.
What we don't know is because the answers, depending on what SMS and how you send the SMS, where you allow them free text and they can type whatever they want, or you get preselected questions and or answers, and then they have to select one. Similar what was on a screen based device. So there's been very little research from what I understand, what I know, or I know been done to show that it actually works.
We had a similar thing. We just recently with with another vendor, we did a study on zooming in and zooming out because if you want to allow accessibility, which is a big topic moving forward and has been now for a while, you need to be able to let patients make this funds bigger so they can read that properly. And there's other things as well that we'll look into soon as well, like read back, apps and all that kind of stuff. Yeah.
So we're always saying it should be a patient choice, but we have to make sure that the data that we get is equivalent to what the original, questionnaire looked like. And in many cases, that's still paper.
And that's missing on the on the SMS side. I totally on WhatsApp, similar. I totally believe, though, I can't prove it, but I totally believe that it'll work okay.
Yeah. Yeah. I I think so too. But, again, I think an invitation to to study this more thoroughly, specifically, I think, with, kind of closed questions. Like, I think the the example was a a Likert scale. So, you know, that's obviously easier.
Ari is starting to answer to It was a three point scale.
Right? Now three points may not be may not be, enough granular. So, generally, I think it goes as low as five as I see sometimes five, six, seven. I've seen eight. But I think five is kind of quite often where I think that's probably the the bare minimum. So three would be a little bit too few. It's not enough.
I Yeah.
Exactly.
We have more questions open, but I think it's time to wrap up. And I wanna give Ari and Willie an opportunity to to leave us with their with their recommendation given the context of of what we discussed here.
So, Ari, maybe you can go first. Considering the paper, considering everything we discussed, if I'm a sponsor CRO and I'm involved in designing a study, and, obviously, I'm going to include PROs in in in those, and, hopefully, that's gonna be done on, you know, using ePRO.
What what are your recommendations to to take into account to to just maximize the the impact of of that data? So financial, you know, value, obviously, but also just the actual impact on giving patients better opportunity to select the drug that's right for them. How how would you, what would be your recommendations?
My my kind of, takeaway and recommendation is that, the sponsors should stop thinking about labeling, or not. There's a whole, gray areas in between.
Some are better than getting a label.
So we talked about the consistent with, labeling guidance. So there are good things that you can do.
So my thing is stop worrying too much about labeling and all the other activities, but consider that without good quality patients' data or observer data, whatever the other data or the other COAs, it's very difficult for, regulators to carry out an appropriate benefit risk analysis.
So it's not an on or off situation. It's it you have to provide the right kind of information so they can, carry out the, benefit risk analysis. And you saw at least one example that I showed you that if you don't, they'll go and do it anyway, which, which is not always a good idea because you are you are losing control of your own data.
Really.
Yeah.
So yeah.
And and my I reckon what we try to get across to our clients and partners is that, doing, or collecting this data is is not as simple, as everyone thinks. There are so many moving parts. It's individually, all the parts are not very complicated and easy to understand, but there's so many of them. You know?
But what we quite often see is that not and Ari mentioned before as well, it's not enough effort put into that one in the beginning, not not recognizing that you need to have resources in the beginning to first pick the right instruments, and then once you have them, pick the right technology and pick the right setup for patients so that they can actually be compliant.
Because I don't think that any patient goes into a study trying to not be compliant. I'm sure that all of them try and in the very beginning want to be as good as they can, and we're just making it sometimes too difficult by by not having the right solution in front of them at the right time.
And to do that right, like we said before, you have to take a step back sometimes and look at the bigger picture and then basically revisit what you wanna do. And based on the questionnaires that have been picked, whether it's a diary or whether it's a biweekly or monthly or three monthly questionnaire, all of that needs to be done properly, and then you will have good data. And once you have good data, you can do anything you want with that data.
Yeah. I, I fully agree. I think, getting the fundamentals right is gonna be absolutely essential, and you'll you'll probably benefit from that from for years to come.
We are we're open time. We will, produce, at least a blog and probably more content from this, from this discussion. We'll also include a lot of resources that have been brought up, to to make it easy to to kind of go back to all the recommendations that we've gotten from these amazing experts. For now, I want to thank you both so much for joining me today and for, providing us with, a lot of great insights, I think, in terms of, PROs and how the FTA and EMA might might assess it might assess those data. So thank you so much. And, also, the audience, thank you so much for joining us, and, have a great rest of your day.
Thank you.
Bye, guys. Bye.



