0:01

Welcome to the

0:03

Skeptics Guide to

0:05

Emergency Medicine. Meet

0:07

them, greet them,

0:09

treat them, and

0:11

street them. Today's date

0:14

is December 20th, 2024,

0:16

and I'm your skeptical

0:20

host, Cadm Milm.

0:22

The title of today's

0:24

podcast is Not

0:26

a Second Time.

0:28

Single-Center. fail to

0:31

replicate in multi-center

0:33

randomized control trials. And

0:35

our guest skeptic for this episode

0:37

doesn't need any introduction, but I'll

0:39

see if I can do one

0:42

just with the voice. Weingart here.

0:44

Yes, it's Scott Weingart. He

0:46

is an ED Intensivist from

0:48

New York City. He did

0:50

his fellowship in trauma, surgical

0:52

critical care, and ECMO. He

0:54

is a physician coach concentrating

0:56

on the promotion of eudimonium.

0:59

and optimal performance. Scott is

1:01

best known for talking to

1:03

himself about resuscitation and critical

1:06

care on that extremely popular

1:08

and well-done podcast called MCRIT,

1:11

which has been downloaded more than, and

1:13

I have to do this with my

1:15

finger, 50 million times. Welcome to the

1:17

SGM Scott. It's such a pleasure I

1:19

was waiting for when I was going

1:22

to get invited to do an official

1:24

SGM, and now I'm here. I am

1:26

so happy that you're here. I'd like

1:28

to clear up a few things, go

1:30

through a few things before we get

1:32

started on today's episode. First of all,

1:34

that theme music was not from the

1:36

1980s, which is my wheelhouse. It's from

1:39

the Beatles. Are you some big beetle

1:41

fan? I'm not. I'm a stones fan if you

1:43

had to put me in a box, but

1:45

it just seemed to fit so perfectly for

1:47

this show. Yeah, and you know what? People

1:49

always ask me, do you pick the theme

1:51

music first can or do you pick the

1:53

paper? We start with the paper and then

1:55

we work back to the theme music. I

1:57

think I'm a bit more of a Beatles

1:59

fan. than a Stones fan, but both

2:02

epic, epic bands. Second question I

2:04

had for is, what is this

2:06

eudemonia? I mean, I'm sure other

2:08

people, at least one other person,

2:10

would have to look it up

2:13

because I was that one person

2:15

that had to look this term

2:17

up. What the heck does that

2:19

mean? Yeah, this is harkening back

2:21

to my Greek philosophy studies, and

2:23

the Greeks, they separated happiness into...

2:26

different kinds. Now you're all familiar

2:28

with the hadonic forms of happiness,

2:30

right? Those temporary blips of joy,

2:32

but they also talked about eudemonic

2:34

happiness, which is more along the

2:37

lines of flourishing. It's really a

2:39

deeper life satisfaction. It's a joy

2:41

that is not momentary. So eudimony

2:43

is really what we all want.

2:45

Eudimony is lasting happiness. So to

2:48

say to thrive to flourish to

2:50

thrive that good exactly it. Yeah,

2:52

that's exactly it Yeah, that's exactly

2:54

it So you just don't want

2:56

you to quote succeed and have

2:58

that one singular moment or one

3:01

little blip you want to be

3:03

thriving and flourishing growing and and

3:05

and that overall happiness Exactly finding

3:07

your purpose and then a viewing

3:09

to it Well talking about finding

3:12

your purpose the last thing I

3:14

wanted to ask you before we

3:16

get started was you and I

3:18

We're both graciously invited to attend

3:20

this amazing conference and contribute to

3:22

it, and it's in Murphya, I

3:25

have to say it with a

3:27

little bit, like I got my

3:29

retainer in, Murphya, in Spain, incrementum.

3:31

Can you give them a bit

3:33

of a shout out? Yeah, I'll

3:36

give a shout out any opportunity

3:38

I have because these folks are

3:40

filling a gap after the smack

3:42

conference disappeared. I didn't get to

3:44

go and meet my international friends

3:47

and all the people I love

3:49

to talk to around the world.

3:51

And it was really a loss

3:53

and these folks are filling that

3:55

gap. They are doing insanely good

3:57

conference. I, you know, these are

4:00

first time conference presenters and I'm

4:02

like, ooh, I don't know. They

4:04

have been absolutely nailing every single

4:06

aspect, administrative, venue, lecture, Every single

4:08

piece of it, they are absolutely

4:11

firing on all cylinders. And I

4:13

can't imagine it's going to be

4:15

anything but an epic conference. So

4:17

I just can't wait to go.

4:19

Yeah, me too. I can't wait

4:21

to go either. They even sent

4:24

me a t-shirt. And this is

4:26

why I don't take me a

4:28

t-shirt. And this is why I

4:30

don't take anything from industry, because

4:32

they give me a t-shirt. Now

4:35

I'm going to be their best

4:37

friend forever. Smack was 2019 in

4:39

Sydney Australia. I still remember meeting

4:41

you with my wife walking across

4:43

the causeway and I'm like, there's

4:46

Weingart. And she's like, who's Weingart?

4:48

And I'm like, Weingart. He's like

4:50

a singer with only one name,

4:52

Cher, Pink, Beyonce, Weingart. And so

4:54

that's been five years and you

4:56

know, since then we, you know,

4:59

a huge loss, I think, a

5:01

huge loss in that. connectivity, that

5:03

socialization, the shrinking of the world

5:05

that FOMED has allowed us electronically,

5:07

but even more so to get

5:10

together in person to celebrate the

5:12

joy we have and providing care

5:14

to patients. So I am super

5:16

looking forward to it. It's in

5:18

March 2025 and I'll put a

5:20

link in the show note so

5:23

people can register and get into

5:25

Spain for that conference. Fantastic. All

5:27

right, well, that's enough preamble. Let's

5:29

get into a case. What did

5:31

you bring today? 40-year-old male presents

5:34

to the ED with severe respiratory

5:36

failure from bilateral pneumonia and you

5:38

tried the patient on non-invasive positive

5:40

pressure. They're not making it. Their

5:42

mental status is starting to fade.

5:45

So you've made the decision to

5:47

intubate. So the question is, should

5:49

your first pass attempt be done

5:51

with a boogie or a stylated

5:53

endotracheal tube? Yeah, and that's a

5:55

great clinical question and you always

5:58

wonder, is there anything that could

6:00

inform my clinical care? Not dictated,

6:02

but just inform me on how

6:04

best to take care of this

6:06

patient. And the role of a

6:09

single center randomized control trial, because

6:11

that's what we're going to be

6:13

talking about today, in advancing medical

6:15

knowledge. is really significant, especially in

6:17

the field of emergency medicine. These

6:19

trials often serve as a foundation

6:22

for us to be basing our

6:24

care on and exploring interventions, hypothesis

6:26

generating, and providing us a focused

6:28

and controlled environment in that randomized

6:30

control trial, and especially if it's

6:33

a single-centered randomized control trial, you've

6:35

got this real controlled environment to

6:37

test this specific hypothesis that you

6:39

have. Yeah, but the problem is

6:41

that the applicability of their findings

6:44

to broader clinical settings could be

6:46

limited because it's only coming from

6:48

one place. It's a localized context.

6:50

Multicenter randomized control trials are often

6:52

seen as a necessary step to

6:54

validate those initial findings and give

6:57

generalizability across diverse patient populations and

6:59

health care settings. The process of

7:01

validation is critical as it addresses

7:03

external validity. cornerstone of EBM. Yeah

7:05

I really I really think this

7:08

is important that and there's two

7:10

things in there to sort of

7:12

unpack one is the hey we

7:14

should try it again and and

7:16

maybe if we're gonna try this

7:18

again we could do it in

7:21

a single center which would add

7:23

to the knowledge base. But the

7:25

second step of that is, can

7:27

we do it in a multi-center

7:29

setting? Because there might be some

7:32

champions that are working in that

7:34

individual center that may have some

7:36

biases that won't be captured with

7:38

our standardized tools. So we need

7:40

to move from that single center

7:43

randomized control trial, test the hypothesis

7:45

again, but move it to that

7:47

multi-center setting. Because you know that

7:49

different institutions, where you work. is

7:51

probably a lot different than where

7:53

I was working the other day

7:56

in a small rural hospital with

7:58

you know x-ray goes home at

8:00

four o'clock the lab tests we

8:02

got a call in the dogs

8:04

come on come here lab you

8:07

know no CT scanner the only

8:09

cat scan we have has four

8:11

legs and the patient population their

8:13

expectations their demographics in my rural

8:15

setting maybe and I don't know

8:17

if you've seen any in those

8:20

Glock home fleckin videos about, you

8:22

know, it's a rural farmer coming

8:24

in, you know, that's a code.

8:26

I mean, poops happening, right? So

8:28

really, we'd like to see it

8:31

in a variety of settings. The

8:33

single center, I mean, again, it

8:35

can give you some insights, you

8:37

can reflect on, you know, the

8:39

real world complexities of what's going

8:42

on, but there are some restrictions.

8:44

We see everything, right? And we

8:46

don't know what's going to walk

8:48

through the door. Even if you're

8:50

working in New York City, a

8:52

farmer could show up on holidays.

8:55

It could happen. And so we

8:57

really like to see robust evidence

8:59

that is all moving, you know,

9:01

as the same direction, hopefully the

9:03

same magnitude, but same direction in

9:06

multiple settings to guide our clinical

9:08

pathways. Yeah, no, absolutely. I mean,

9:10

but despite the apparent hierarchical superiority

9:12

of these multi-centered RCTs, they just

9:14

sound so much better on paper,

9:16

there are debates about whether they

9:19

consistently are going to confirm the

9:21

results of single center RCTs. And

9:23

this discussion is pivotal in understanding

9:25

how findings can be generalized and

9:27

integrated into clinical guidelines as emergency

9:30

docs. evaluating the interplay between a

9:32

single center RCT and its subsequent

9:34

multi-center RCT, not only helping assessing

9:36

the reliability of evidence, but also

9:38

in shaping the way we approach

9:41

the implementation of interventions in our

9:43

clinical practice. So Scott, what's the

9:45

clinical question we're going to try

9:47

to answer on today's podcast? How

9:49

often are single center RCTs of

9:51

critically ill patients reporting a mortality

9:54

benefit confirmed in multi-center RCTs? So

9:56

we're going from single centers to

9:58

multi centers and we're going to

10:00

have critically ill patients in that

10:02

population and the outcome is going

10:05

to be pretty objective. So let's

10:07

get specific. What was the population

10:09

that they included? So the population

10:11

of this trial was actually other

10:13

papers. So the population is single

10:15

center. RCTs published in high-impact journals,

10:18

this was New England Journal, Jama,

10:20

or the Lancet, that reported statistically

10:22

significant mortality reductions in critically ill

10:24

patients. Yeah, I see they left

10:26

out my favorite journal, the BMJ.

10:29

I think that's my favorite right

10:31

now. Still legit, yeah. They excluded

10:33

quasi-randomized trials or non-randomized trials. So

10:35

they really wanted to look at

10:37

are these RCTs, and then of

10:40

course they didn't include multi-centered trials

10:42

in that group. They were looking

10:44

at adults only. and it had

10:46

to have mortality basis. So let's

10:48

be very specific. What was quote

10:50

the intervention that they were looking

10:53

at? So the intervention by this

10:55

rubric would be single center RCTs.

10:57

And then they were comparing it

10:59

to what? Subsequent multi-center RCTs. And

11:01

their primary outcome of interest? Mortality

11:04

assessed at specified time points such

11:06

as hospital discharge or predefined follow-up

11:08

periods. And I like these primary

11:10

outcomes that are less subjective than

11:12

others. And I always use that

11:14

princess bride quote, you know, they're

11:17

not mostly dead, right? It's mortality.

11:19

Most of us can figure out

11:21

dead or alive. Ooh, that's a

11:23

New Jersey song, isn't it? Dead

11:25

or alive. The other thing about

11:28

mortality is it was all cause

11:30

mortality. So it wasn't disease specific

11:32

mortality. We just wanted to count

11:34

numbers there. Now they did have

11:36

some secondary outcomes. This was guideline

11:39

utilization of single center randomized control

11:41

trials results. So did the guidelines

11:43

add single center randomized control trials

11:45

into the recommendation? And did they

11:47

change their guidelines once a multi-centered

11:49

randomized control trials were published? Ooh,

11:52

I wonder what the results are

11:54

for that. Okay, so what type

11:56

of study was this? This was

11:58

a systematic review that followed the

12:00

PRISMA guidelines and was registered in

12:03

the Prospero. International Prospective Register of

12:05

Systematic Review. So the author's conclusions

12:07

were, quote, mortality reduction shown by

12:09

single center randomized control trials is

12:11

typically not... replicated by multi-center randomized

12:13

control trials. The findings of single-center

12:16

randomized control trials should be considered

12:18

hypothesis generating and should not contribute

12:20

to guidelines. Ooh, that's pretty firm

12:22

there, isn't it? Hmm. Okay, let's

12:24

go through their quality checklists for

12:27

systematic reviews. First question, was the

12:29

main question being addressed, clearly stated?

12:31

Yes, the systematic review addressed whether

12:33

survival benefits observed in single-center RCTs.

12:35

in critically ill patients were confirmed

12:38

by subsequent multi-center RCTs. Yeah, so

12:40

it was a real specific question,

12:42

you know, multi-center, single center, and

12:44

it was adults, and they had

12:46

to be critically ill, and mortality,

12:48

I like that. How about the

12:51

search? Was it detailed and exhaustive?

12:53

It was, I mean, they didn't

12:55

make much work for themselves, because

12:57

they limited it to a very

12:59

specific set of journals, but yes,

13:02

they did everything they needed to.

13:04

Were the criteria used to select

13:06

the articles for inclusion appropriate? Yes,

13:08

the inclusion criteria required single center

13:10

RCTs reporting statistically significant mortality decreases

13:12

in adult critically ill patients and

13:15

subsequent multi-center RCTs addressing the same

13:17

research question. And do you think

13:19

the included studies sufficiently valid for

13:21

the type of question that was

13:23

asked? Yep. Were the results similar

13:26

from study to study to study?

13:28

Unshore. Yeah, we haven't done any,

13:30

I haven't seen any other publications

13:32

looking at how how single center

13:34

studies replicate, so really nothing to

13:37

compare to there. How about financial

13:39

conflicts of interest? No, the authors

13:41

declare they have no competing interests.

13:43

And so where did the money

13:45

come to do to study? It

13:47

was funded by their academic institutions.

13:50

All right, so the results. The

13:52

review included 19 single center randomized

13:54

control trials that they could find

13:56

in those high impact journals with

13:58

those specific inclusion criteria. criteria, and

14:00

they found 24 subsequent multi-center randomized

14:03

control trials. Now, 16 out of

14:05

the 19 addressed were followed

14:07

up with multi-center randomized control

14:09

trials, and the majority of

14:12

the multi-center randomized control trials

14:14

found no mortality difference,

14:16

so no statistical difference, compared

14:19

to the significant findings that

14:21

had been reported for a

14:23

mortality benefit in the single

14:25

center randomized control trials. So can

14:28

you summarize that Scott into a key

14:30

result? Yeah, it's pretty easy. Single

14:32

center RCTs often do not

14:34

replicate in multi-center RCTs. And

14:36

there are some number people that listen

14:39

to this show. They want numbers. So

14:41

can you put a number on that

14:43

primary outcome of how often do they

14:45

replicate? Only one out of 16

14:47

actually replicated. That's 6% of a

14:50

single center RCT that was later

14:52

confirmed by a multi-center RCT. So

14:54

that's our English language of... often

14:56

do not, in this case,

14:59

means 6%. So 94% don't

15:01

replicate, often do not. All

15:03

right, how about their secondary

15:05

outcomes? So 14 of the

15:07

single center RCTs were referenced

15:09

in at least one international

15:12

guideline. And of those, 43%

15:14

six out of 14 have

15:16

since either been changed to

15:18

suggesting against or removed out

15:21

of the most recent version

15:23

of those guidelines. So the flip

15:25

side of that is 57% haven't been

15:27

updated, right? You know, and this is

15:30

a problem with knowledge translation.

15:32

Once things get into guidelines,

15:34

and if the guidelines aren't

15:36

living documents are updated frequently,

15:38

they're there for years and

15:41

people might be relying on

15:43

those single center RCTs that

15:45

claimed a superiority to some

15:47

intervention, some medication on a mortality

15:49

benefit for critically ill

15:51

adults. And the multi-centered

15:53

trial said, uh-uh. No,

15:55

not happening. So, all right.

15:57

Now I get to talk nurse.

15:59

with you Weingart and I'm looking forward

16:02

to this. I'll lead it off with

16:04

something gentle and this is about the

16:06

Prisma guidelines. So the Prisma guidelines, this

16:09

is a standardized set of guidelines

16:11

and I'll put a link in

16:13

the show notes and these are

16:16

the preferred reporting standards for systematic

16:18

reviews and meta analyses. And this

16:20

paper adhered to several essential Prisma

16:23

checklist items but they did fall short

16:25

in some key areas. They didn't provide

16:27

the full search strategy, at least one

16:29

that I could follow closely enough to

16:31

replicate. There were some issues

16:33

with reporting bias, certainty assessments,

16:36

and a detailed risk of bias assessment.

16:38

So my conclusion is the study doesn't

16:40

fully satisfy the Prisma 2020 quality criteria.

16:42

I'll put a table in the show

16:45

notes for that, but we were talking

16:47

earlier and I think when I read

16:49

in a paper that says, you know,

16:51

that they followed the Prisma guidelines, I'm

16:53

going to... Think of more like a

16:55

movie that comes out and says, inspired

16:57

by a true story, based on a

17:00

true story. These were based, this study

17:02

was based on the Prisma guidelines, or

17:04

inspired by the Prisma guidelines. I'm going

17:06

to be skeptical. Well, Ken, you know,

17:08

they're using the word guidelines, and

17:10

if we apply the same way we're

17:13

supposed to use clinical guidelines, as they're

17:15

using the Prisma guidelines, then I think

17:17

they're well in keeping with the intent

17:19

of a guideline, rather than using it

17:22

as a firm checklist. You are correct

17:24

and you know they are to be

17:26

guiding your care but I guess it's

17:28

what what you lose in a sentence

17:30

you maybe gain in a table that

17:32

you can say well I'm interested in which

17:34

ones didn't they follow because some are

17:37

more important than others you know did

17:39

they did they list certain things or

17:41

I'm interested in very key areas of

17:43

those multiple checklists questions that they have

17:46

in the guidelines and so I I

17:48

think you know it would be great

17:50

if in the document itself they would

17:53

list the prism of guidelines and had

17:55

a yes no or unsure and where

17:57

can I find that in the manuscript

18:00

And so that just really helps me.

18:02

Now, they did have that checklist in

18:04

the subsequent supplementary material. But

18:06

people have to dig and look for

18:08

that and find it. But they did have that

18:10

in there. So that was good. I mean,

18:12

if we're going to talk nerdy,

18:14

we should also talk publication bias.

18:17

This occurs because the likelihood of

18:19

research results being published is influenced

18:21

by the nature and direction of

18:23

the findings. Studies with quote unquote

18:26

positive. or statistically significant or novel

18:28

results are more likely to be

18:30

published, while those with negative or

18:32

inconclusive outcomes often remain unpublished

18:35

or delayed. And I have a

18:37

positive finding that supports this position.

18:39

We can quantify publication bias. There

18:41

was a systematic review that found

18:44

that studies reporting significant outcomes were

18:46

more likely to be published than

18:48

those without. And you can get

18:50

a pooled odds ratio of 2. And

18:52

so this indicates that studies

18:54

that have significant findings, so

18:56

positive findings, have a 2.8

18:58

times higher odds of being

19:00

published compared to studies with

19:02

non-significant findings. And this imbalance

19:04

can skew the body of available

19:07

evidence leading to overestimation

19:09

of intervention effects, misrepresentation

19:12

of true outcomes, and flawed decision-making and

19:14

clinical practice. policy development or future research.

19:16

We should try and move away from

19:18

thinking of studies as positive or negative.

19:20

If you've asked a good question and

19:23

used appropriate methods, then it does not

19:25

matter if the results are positive or

19:27

negative. Science has been moved. forward. And

19:29

these results should be part of the

19:31

medical literature to minimize publication bias. Oh,

19:33

I love what you just said there.

19:36

Yeah. I mean, we need to get

19:38

away from that cognitive experience of running

19:40

a trial or running a study. And

19:42

then we get these positive results and we

19:44

start high-fiving. Whoo! And you get quote negative

19:46

results or non-significant findings. You're like, wah, wah.

19:49

And it ends up in the desk drawer

19:51

or the bottom of the desk drawer. And

19:53

the same thing when you submit it when

19:55

you submit it for publication. Here's the findings

19:57

and we submit it to an article. a

20:00

journal and I'm a senior editor

20:02

of a journal, I want to

20:04

publish those findings. I want to

20:06

mitigate against that bias. So important.

20:08

All right, the third nerdy point

20:10

was about the heterogeneity of the

20:12

study populations. There was variability in

20:14

these different studies, you know, in

20:16

these more than a dozen single

20:18

center randomized control trials and the

20:20

subsequent multi-centered randomized control trials. There

20:22

is differences in patient demographics, settings

20:25

and interventions that may contribute to

20:27

the conflicting results. But you can

20:29

interpret this in a couple of

20:31

ways. You could say, well, you

20:33

know, there's a lot of heterogeneity,

20:35

but you could also say that

20:37

that's what we're looking for, external

20:39

validity. You know, like, does this

20:41

actually externally validate in lots of

20:43

different settings? So do you apply

20:45

it individually? If your single center

20:47

is identical, and you have the

20:50

identical population, the identical staff, the

20:52

identical resources. Okay, maybe you could

20:54

make a case that, yeah, this

20:56

single center should inform my care,

20:58

but most of us don't have

21:00

these identical populations and settings. Yeah,

21:02

I mean that you put your

21:04

finger on one reason that that

21:06

replication may not occur. I mean,

21:08

it's interesting when we talk about

21:10

single center RCTs. Single center studies

21:12

often have unique settings or expertise

21:15

that may not be generalizable till

21:17

the multi-center trial that's done subsequently.

21:19

They also have smaller sample sizes

21:21

most of the time, increasing the

21:23

risk of type one errors compared

21:25

to larger multi-center RCTs. So let's

21:27

talk about a few of these

21:29

if you're willing, Ken. Absolutely, let's

21:31

bring up some examples. You know,

21:33

here we're talking nerdy, but let's

21:35

talk to some clinicians here about

21:37

some clinical scenarios of how this

21:39

can be applied. Yeah, so we

21:42

mentioned type one error and if

21:44

I had a guess that's how

21:46

a study like intensive glycemic control

21:48

Actually managed to have a positive

21:50

single center RCT that was totally

21:52

disconfirmed in the subsequent Multisenter trial

21:54

and as someone who was in

21:56

the midst of critical care at

21:58

the time that came out it

22:00

was a real horror show because

22:02

you had getting hypoglycemic all the

22:04

time, some of them with brain

22:07

injuries, and all of us were

22:09

asking, what the hell are we

22:11

doing? It made no sense, but

22:13

it's not the kind of thing

22:15

you could just see the results

22:17

of in your own care. So

22:19

you had to rely on the

22:21

RCT, because these were long-term outcomes.

22:23

And then when the subsequent multi-center

22:25

came out, all of us are

22:27

like. Oh, thank you. We no

22:29

longer have to do this. My

22:32

guess, that's a type one error.

22:34

But there's other ways, too. We'll

22:36

just leave off nefarious people doing

22:38

misalignment of their data with reality.

22:40

Let's leave that off for a

22:42

second. Let's hope everyone's doing a

22:44

good job. There's other reasons. One

22:46

of them is hidden confounders. There's

22:48

something else going on in the

22:50

trial that's not mentioned in the

22:52

actual paper that's published. An example

22:54

of this, early goal-directed therapy, changed

22:57

the face of sepsis. In that

22:59

trial, it was... purported that the

23:01

differences between the two groups were

23:03

a whole set of interventions, a

23:05

bundle in the intervention group. But

23:07

there was one thing that wasn't

23:09

mentioned that really, I think, played

23:11

a big part in those study

23:13

results, which is the early goal-directed

23:15

therapy group had a fellow come

23:17

in from home and stay at

23:19

the bedside for six hours, only

23:21

caring for that one patient, versus

23:24

the other group that just got

23:26

standard care in a very, very

23:28

busy ED. but it was never

23:30

there and it was never part

23:32

of the subsequent replication. Now, Ken,

23:34

do you think that might change

23:36

the results of that study? I

23:38

think it might, you know, having,

23:40

having looked at the bundle and

23:42

having those multi-centered trials like a

23:44

rise and stuff, sort of unpack

23:46

it and find that the intervention

23:49

really didn't report the mortality benefit

23:51

they were hoping to find. If

23:53

you have... a whole bunch of

23:55

fluids, if you have a whole

23:57

bunch of pressers, you have source

23:59

control, you have this. How about

24:01

you add another clinician with one

24:03

to one care at the bedside

24:05

for six hours? I work short-staffed

24:07

almost every shift. Give me another

24:09

warm intelligent body. to help take

24:11

care of patients, I'm sure we

24:14

can improve the care of patients.

24:16

100%. Now, early goal directed therapy

24:18

was also interesting because it has

24:20

another possible reason for lack of

24:22

replication. And it's what I like

24:24

to call control group evolution, Ken.

24:26

And the idea here is that

24:28

by the time the multi-center is

24:30

done, the care that that single

24:32

center might have asked for might

24:34

have been incorporated into standard care.

24:36

And now especially if that subsequent

24:39

trial is using a pragmatic. control

24:41

group, meaning they get to just

24:43

do what they want versus the

24:45

intervention group, then that might actually

24:47

lead to a loss of replication

24:49

because people have already incorporated the

24:51

main tenets of that single center

24:53

RCT. Yeah, you can see that

24:55

happening because I do appreciate what

24:57

Dr. Rivers did with raising the

24:59

awareness of the importance of sepsis

25:01

and we're not just going to...

25:03

park an old person with maybe

25:06

an o cult problem on a

25:08

stretcher and a hallway, that could

25:10

be sepsis. So we really raised

25:12

the awareness and got the attention

25:14

and that deserves a shout out

25:16

for that. But then once we

25:18

became as a community more aware

25:20

and upped our practice and really

25:22

focused on the fundamentals of, hey,

25:24

this could be sepsis, you know,

25:26

so early recognition, early source control

25:28

and early antibiotics. that's our game

25:31

elevated and now you're going to

25:33

look at this bundle care and

25:35

say does this add anything to

25:37

our improved care in general and

25:39

failed to replicate in multiple multi-center

25:41

trials. Absolutely we can talk about

25:43

another one pre-oxy newer trial single-center

25:45

I'm sorry I take that back

25:47

multi-center RCT but I think it

25:49

makes a good point so I'm

25:51

going to mention this Ken even

25:53

though it's not perfectly filling the

25:56

rubric here but that multi-center RCT

25:58

looked good it was actually using

26:00

non-invasive positive pressure for pre-auctionation and

26:02

apnea period. But there was a

26:04

hidden confounder there, which was the

26:06

control group, which was supposed to

26:08

get ideal non-pressure. pre-oxy. They actually

26:10

did not force the control group

26:12

to use the auction flow meter

26:14

settings that would actually make that

26:16

happen. And my guess is if

26:18

they redo preoxy they would fix

26:21

that point in to the control

26:23

group making sure it was done

26:25

and I think there's a potential

26:27

preoxy will not replicate when redone.

26:29

So that one's a little off

26:31

topic but I think it's just

26:33

another example of these hitting compounders.

26:35

Well for hitting compounders it just

26:37

identifies... how important it is to

26:39

really think for a long time

26:41

about the methodology. The method, like

26:43

that's my favorite section of a

26:45

paper, go to the methods and

26:48

find out, will they be able

26:50

to answer the question they have

26:52

with the methods as described? Could

26:54

I replicate this? Do I see

26:56

any confounders? Do I see any

26:58

flaws or errors or limitations? And

27:00

so I think that's really, really

27:02

important because that, you know, like...

27:04

Prioxies a big trial, multi-centered trial

27:06

took a lot of money, a

27:08

lot of patients' time, a lot

27:10

of clinicians' time. You want to

27:13

get the quote truth, which is

27:15

the best point estimate of an

27:17

observed effect size, out of the

27:19

study that you're doing. And so

27:21

really focusing in on good methods

27:23

is so important. Yeah, I'll mention

27:25

one more, Ken, and this one

27:27

directly pertains to our clinical case.

27:29

And it goes under the category

27:31

of different skill sets or expertise

27:33

between the single center providers and

27:35

the multi-center providers. So let's talk

27:38

about beam versus boogie. The beam

27:40

study was a single center RCT

27:42

of boogie versus stylated tube in

27:44

one emergency department where they trained

27:46

boogie. very hard prior to the

27:48

RCT being done. They were fantastic

27:50

at it. And the rates of

27:52

first-pass success in that trial were

27:54

98 point something. I mean, just

27:56

unheard of rates of expertise when

27:58

they used the boogie. It was

28:00

subsequently studied in a multi-center RCT

28:03

that showed no benefit. And those

28:05

first-pass success rates in both groups

28:07

dropped to the low 80s. The

28:09

difference between these two trials was

28:11

in the first one remarkable expertise

28:13

with the boogie and in the

28:15

second subsequent multi-centered RCT, most of

28:17

the people performing intubations had never

28:19

touched a boogie before their one

28:21

time they intubated as a participant

28:23

in this study. So most of

28:25

the people had never touched a

28:27

boogie and they only contributed one.

28:30

intubation to the study. So they

28:32

were not gaining any expertise in

28:34

the midst of performing the during

28:36

the study time and that trial

28:38

was negative. But of interest is

28:40

I don't think that in any

28:42

way invalidates the results of that

28:44

initial RCT. It just means you

28:46

need to have the same milieu

28:48

as that single RCT occurred in

28:50

if you're going to get the

28:52

same results. Yeah, that gets back

28:55

to having a local champion and

28:57

the local culture of where you're

28:59

working. And so if you have

29:01

Dr. Bugee at your center and

29:03

he's he or she is training

29:05

all of these residents and advocating

29:07

for it and promoting it and

29:09

showing it and having rounds on

29:11

it and all of this kind

29:13

of stuff that goes on, people

29:15

are going to get probably better

29:17

at using a Bugee. And if

29:20

you don't and you apply it

29:22

to another center, the difference isn't

29:24

that. piece of gummed rubber. It's

29:26

the operator that's holding that gum

29:28

boogie and using that tool. So

29:30

train the tool user is probably

29:32

way more of a big impact

29:34

than the actual tool if you

29:36

just say, hey, here's this long

29:38

stick, use it to intubate people,

29:40

go. Absolutely. All right, the fifth

29:42

and final point is about guidelines.

29:44

I've said this many times. Guidelines

29:47

are two. Guide, our care. It's

29:49

right there in the word guidelines.

29:51

And again, this brings me back

29:53

to Princess Bride when they say,

29:55

Inconsievable. You can use it in

29:57

the word. I don't think it

29:59

means what you think it means.

30:01

Because people often use guidelines as

30:03

Godlines, thou shalt do this, thou

30:05

shalt not do that. And you

30:07

know, the research indicates that the

30:09

validity of guideline recommendations diminishes over

30:12

time. There was a study published

30:14

in the Canadian Medical Association journal,

30:16

analyzing what's the lifespan of clinical

30:18

guideline recommendations. may found approximately 90%

30:20

remained valid after one

30:22

year. However, the validity decreased about

30:25

80% after three years and down

30:27

to 78% after four years. Yeah,

30:29

these data suggest that a significant

30:31

proportion of recommendations may become

30:33

outdated within a few years

30:35

of publication. In the study

30:37

we're reviewing today of the

30:40

14 single center RCTs referenced

30:42

at least once in international

30:44

guidelines, six 43% have since...

30:46

been either removed or suggested

30:48

against in the most recent

30:50

versions of those guidelines. And

30:53

it's data like this that informs my

30:55

position that we should be skeptical of

30:57

the push to blindly follow guidelines when

30:59

we are pressured by organizations like the

31:02

American Heart Association and they have a

31:04

program called Get With the Guidelines. The

31:06

hair in the back of my neck

31:09

just goes up when I hear that

31:11

get with the guidelines. It's directive and

31:13

it's authoritative and it's authoritative and it's

31:15

authoritative. The recommendations when

31:18

you look at these are really

31:20

often not based on high quality

31:22

evidence. There's some, there's a review

31:25

of all the recommendations of about

31:27

a decade's worth of AHA guidelines

31:29

and showed that less than 10% were

31:32

based on level A data. So

31:34

my question always is how

31:36

closely should we adhere to

31:38

a specific recommendation? 25% of the time,

31:40

50% of the time, 75% of the

31:43

time, 100% of the time. Where's the

31:45

information that tells me that? Because I'm

31:47

taking care of an individual patient. And

31:50

we know that the EBM framework, the

31:52

literature is just one of three pillars.

31:54

We still need to use our clinical

31:56

judgment at the bedside, right? And then

31:59

ask the patient. because they're, you

32:01

know, my son always goes,

32:03

you're unique, just like everyone

32:05

else. That's what patients are

32:07

like. They're unique, individual patients.

32:09

They're unique, just like everyone

32:12

else. They're just, we need

32:14

to ask them about their values

32:16

and preferences. Well, that's enough

32:18

nerdiness there, Scotty. I want to

32:21

get to comment on the author's

32:23

conclusions and compare them to what

32:25

we felt. Sure. So we generally

32:27

agree. with the author's conclusions that

32:30

the results of single center RCTs

32:32

should be considered hypothesis generating and

32:34

should not contribute to clinical practice

32:36

guidelines. And how about an SGM bottom line?

32:39

Be skeptical of accepting the conclusion

32:41

of single center RCTs unless you

32:43

could precisely duplicate the conditions that

32:45

led to positive single center RCT

32:47

results. And can you resolve that case

32:49

that you presented at the beginning when

32:52

you had a decision to make about

32:54

the intubation? Sure. You choose to use a

32:56

boogie on your first pass because you

32:58

have trained extensively with the device and

33:00

believe you are more akin to the

33:02

bean trial clinicians than the boogie trial

33:04

clinicians. And so how are you going to

33:06

take this systematic review and apply it

33:08

clinically? Single-Center RCT reporting

33:10

a mortality benefit in critically ill

33:13

patients often does not replicate in

33:15

multi-center RCTs and should be considered

33:17

hypothesis generating rather than definitive. They

33:19

probably should only be used to

33:22

change clinical practice if you know

33:24

exactly what went into the intervention,

33:26

especially the behind-the-scenes stuff and

33:29

could precisely replicate those methods.

33:35

Okay, it's time to announce

33:38

the Keener contest winner,

33:40

and it was back-to-back wins

33:42

for Brian Caldwell. He knew

33:45

the opioid epidemic was declared

33:47

a public health emergency in

33:49

2017. Brian also knew that

33:52

Taylor Swift's favorite number

33:54

is 13. What's the question this

33:56

week? What is often cited as

33:59

the first? randomized control

34:01

trial. Yeah, and like most things

34:03

in science, there's a little bit

34:06

of debate around that, but we're

34:08

looking for the first randomized control

34:10

trial, not the first trial. That's

34:12

a little hint to people. So

34:15

if you think you know who

34:17

is credited with the first randomized

34:19

control trial, a modern sort of

34:21

version of that, then... Send an

34:24

email to the sjam@gmail.com and you

34:26

will be the first winner of

34:28

the Keener contest for 2025. I

34:30

look forward to seeing those fill

34:33

up my inbox. Scott it has

34:35

been an absolute pleasure having

34:37

you on the s gem. So much fun.

34:39

And I look forward to seeing you

34:41

in Spain my friend. Indeed. And

34:44

before you go, could you give us

34:46

the s gem tagline in your deepest

34:48

Scott wine? Remember

34:51

to be skeptical of anything you

34:53

were, even if you heard it

34:55

on a skeptic's guide,

34:57

to understand.