0:00

Welcome to the SEI podcast series. My

0:02

name is Tim Chick. I'm the

0:04

Applied Systems Group Technical Manager here

0:06

at the SEI. Today,

0:08

I'm pleased to bring you Aaron Ruffett

0:10

and Brent Fry, two of my

0:12

colleagues. We're here to talk about the platform independent

0:14

model and how it's evolved in the

0:16

way in which we do assessments and

0:18

evaluate programs to really help

0:20

put people on the improvement

0:23

path to success with DevSecOps. We

0:26

start every podcast by asking our

0:28

guest, what brought them to

0:31

the SEI and why are you here? Brent,

0:33

you want to go first? Sure.

0:35

So my

0:37

DevSecOps journey started about 30 years

0:39

ago before DevSecOps or DevOps was

0:41

even a term. I

0:44

was a systems administrator, database

0:46

administrator, and programmer.

0:51

I moved into

0:53

doing creating

0:55

programs that would

0:58

process security alerts and

1:00

try and weed out false

1:02

positives from their move to

1:04

another group that was doing

1:06

research and big data

1:09

analytics. And

1:12

then some of

1:14

my previous co -workers had said, hey,

1:16

we've got some interesting things going on

1:18

at CERT. So I came here and

1:20

started working on a... big data

1:22

analysis system for collecting

1:24

malware and processing and

1:27

analyzing it. And

1:29

now I'm taking all

1:31

those things that I've

1:33

learned over the years

1:36

and helping government sponsors

1:38

to improve their systems.

1:40

Aaron, how about you? I've

1:43

been at the SCI now for almost 15

1:45

years. Prior to that, I came from a traditional

1:47

software development background. I worked for small defense

1:49

contractors in my first job out of school. I

1:52

had to learn how to do that entire job, start

1:54

to finish. And

1:57

even at that time, it wasn't called DevOps, but

2:00

we started to try to dabble with

2:02

automation and tooling to support our

2:04

own development practices and try to implement

2:07

things like that in a small

2:09

shop from scratch with people who didn't

2:11

necessarily know. what they were

2:13

doing. So it was the opportunity

2:15

for me to learn a lot

2:17

and I'm glad I was able

2:19

to take that experience, come into

2:21

cert, work on larger systems in

2:23

an environment that has allowed me

2:25

to foster more of a collaborative working

2:31

relationship with my sponsors and larger

2:33

teams and just learn and grow. I've

2:36

kind of transitioned more away from

2:38

standard software development now into supporting programs

2:40

to write better software. Right.

2:42

Right. And I think the years that you've

2:44

also been here many years like Aaron has

2:46

really been able to keep up with the

2:49

modern technology that constantly evolve. We're not really

2:51

stuck in that, you know, this the way

2:53

we've always done things, right? This is one

2:55

the benefits of being in FFRDC. And

2:57

so that's obviously built into your

2:59

experience with DevSecOps, which is relatively

3:01

new -ish compared to more traditional

3:03

development. worlds, right? So

3:06

what is DevSecOps, Erin, in

3:08

your mind, and why do organizations,

3:10

especially government organizations, really

3:12

struggle in adopting and evaluating and

3:14

just embracing it? That's

3:17

a good question. DevSecOps

3:19

oftentimes gets conflated with

3:21

agile, and

3:23

it's not

3:25

surprising that you usually go

3:28

hand in hand. But particularly

3:30

within the government space, oftentimes

3:32

we're dealing with large

3:34

programs, individuals who come from

3:36

more traditional programs, maybe not

3:38

software -centric programs, who are

3:41

just learning what is DevOps,

3:43

what is Agile, what is

3:45

cloud, what are containers, what

3:47

are all of these pieces and how do they fit

3:49

together. And oftentimes one

3:51

challenge is just getting all of that

3:53

sorted out. for our mission partners and

3:55

really laying out what DevOps

3:57

is. So to answer your

3:59

questions, a long way of answering your question, it's

4:02

really about automating delivery of

4:04

value. So DevOps is about automation. You

4:07

don't necessarily need to be

4:09

doing an agile development methodology, which

4:11

is working closely with your

4:13

users, doing quick turn, iterative software

4:15

development and delivery. You can

4:18

be doing more traditional type development

4:20

with longer cycles. But

4:22

what you need to be doing is having

4:24

developers work closely with operators. That's really

4:26

what DevOps is, is marrying those two together.

4:28

And it works best in organizations where

4:30

that is actually the case. One

4:33

challenge I've seen is programs

4:35

trying to shoehorn it into

4:37

projects where it necessarily fit. Maybe

4:40

Agile or some of the building blocks

4:42

will work for them, but try to take

4:44

DevOps and apply it to... Programs that

4:46

it's not a good fit is one challenge

4:48

area. It's like near their

4:50

stove pipe. They haven't broke down some

4:52

of the social technical aspect. It's not

4:54

just automation, right? It's it's automating things

4:56

that you might have done manually or through

4:58

discussions, but you want to automate

5:00

it to take out the human error perspective.

5:02

But that that implies that we've

5:05

worked out the process. We know how it

5:07

works so we can automate it. That's

5:09

definitely a prerequisite for automation is, first

5:11

of all, actually doing what you want to

5:13

do and found that it is effective. That

5:16

starts with building an organization except

5:19

breaking down stovepipes. DevOps and Agile

5:21

are about breaking down stovepipes and

5:23

getting the people who need to

5:25

collaborate together actually collaborating. A

5:27

big component of that is leadership buy -in.

5:30

Leadership needs to understand exactly what they're

5:32

getting. And

5:34

they're the ones

5:36

who could affect things like organizational.

5:38

Lay out and bring the right people

5:40

in and other areas skills Having

5:42

the right people the right experience and

5:44

the right skills to actually be

5:46

able to succeed And then having a

5:48

risk taking culture as opposed to

5:51

a risk averse culture DevOps is a

5:53

new way of working a lot

5:55

of organizations particularly in the government are

5:57

familiar with waterfall and are comfortable

5:59

with it DevOps is unfamiliar and

6:01

you when married with agile

6:03

you're working in uncertain quick

6:06

turn situations. There's a lot of risk involved in

6:08

that and not everyone is prepared to take that.

6:11

Brad, what do you think is challenging people from

6:13

the adoption? Well, to add on to what Aaron

6:15

was saying, getting

6:17

the feedback quickly

6:20

is often a

6:22

challenge, whether it's

6:24

within the various

6:27

teams, the development

6:29

and the operations

6:32

teams, or back

6:34

to uh, the

6:36

customer and the customer in,

6:38

especially in government scenarios, the

6:40

customer is not always the,

6:42

um, the user of

6:44

the system. And so what

6:47

the customer who is specifying

6:49

what, uh, they think is

6:51

needed might not actually match

6:53

what the user needs and

6:55

making sure that value proposition,

6:57

right? Uh, you know,

6:59

that both values of the

7:01

customer and the users are

7:04

taken into account. In

7:06

this case, the customer being the

7:08

government is that the program office,

7:10

the funding of the work to

7:12

be done versus the Department of

7:14

Defense, it's the warfighters, usually the

7:17

end user. They have to

7:19

coordinate at the very beginning in terms of

7:21

a needed capability. The program office creates a capability

7:23

that the warfighter uses, but

7:25

if something's wrong, that coordination, we

7:27

deliver something that the warfighter can't

7:29

use or it's not optimal. That's

7:31

what we're trying to avoid. So

7:36

You mentioned Aaron a little

7:38

bit like cloud and you know

7:40

more modern knowledge. Is that

7:42

required to do just like ops?

7:44

So I have to be

7:47

doing cloud and containers and Docker

7:49

to be able to to

7:51

embrace DevSecOps? No, and that's that's

7:53

oftentimes a people assume that

7:55

Particularly the DoD. There is the

7:57

DevSecOps DoD enterprise DevSecOps initiative,

7:59

which is very cloud

8:01

and container focused. And

8:03

so that's the lens that the DoD

8:06

right now tends to be looking at DevSecOps

8:08

through. But you can do DevSecOps

8:10

in situations where, like I said earlier, you

8:12

have developers and operators who need to talk

8:14

to each other and work and collaborate with

8:16

each other. That's really

8:18

the prerequisite. And then the next piece is

8:20

automation. Once you have

8:22

processes that are effective, automating

8:24

them in order to shorten

8:26

delivery cycles, improve quality and

8:28

improve value. delivered

8:30

ultimately to the warfighter. That's

8:32

what you need. You

8:34

could do DevSecOps in situations

8:37

where you're not necessarily

8:39

doing containers. You can do

8:41

traditional metal. And

8:43

I'm delivering software directly to

8:45

pair metal, for instance. No containers,

8:47

no Kubernetes, no VMs

8:49

or anything like that. But the key

8:51

is, as I have developers who

8:53

are pushing code, goes through a pipeline,

8:56

ends up on a system that

8:58

then the same team or a team

9:00

that Virtually or physically sits next

9:02

to them is operating is giving them

9:04

feedback on those are the critical

9:06

elements as to what you need in

9:08

order to do this. Okay, so

9:10

the other question is super and ask

9:12

you like the whole part of

9:14

the podcast we're talking about assessments, right?

9:17

Why do people ask for assessments? Why

9:19

do people want to evaluate their pipelines? Well,

9:22

there's lots of different reasons. Most

9:24

often they're trying to do

9:26

a baseline to figure out

9:29

where their system is at

9:31

the moment so that they

9:33

know how to make it

9:35

better, how to

9:37

make it more mature,

9:40

add capabilities, find

9:42

out what's missing. Another

9:45

reason is they're looking

9:47

for us to find problem

9:49

points. One thing

9:51

we've done with one of

9:53

our sponsors was we actually

9:55

went through their development pipeline

9:57

as if we were a

10:00

developer team and tried to

10:02

find out what What the

10:04

pain points were in there

10:06

that weren't necessarily technical, right?

10:09

They were okay. I'm

10:11

gonna be a developer give me a

10:13

count I'm gonna build this thing like you

10:15

know an example are of using your

10:17

system, but I'm gonna use your technology your

10:19

pipeline from start to end, and

10:21

we're going to tell you what the experience

10:23

is from a developer. Right. So

10:25

they think it should take a week

10:27

or two weeks to do this, and we

10:29

find out, well, no. We

10:32

ran into this problem, and so that took

10:34

three weeks, and then something else happened. So

10:37

we have a day

10:39

in the life of a

10:42

developer, a time in

10:44

the life of the developer

10:46

to figure out what where

10:49

the problems are in there. And

10:52

we would not necessarily know that just

10:54

by saying, okay, what tools do you

10:56

use for this? What framework are you

10:58

using here? And

11:00

so that's something that

11:02

while they might be getting

11:05

feedback from those groups,

11:07

we're an independent group. And

11:09

so we don't have

11:11

a stake in... in that

11:13

we're not trying to

11:15

hype things up. We're just

11:17

laying it out like

11:19

it is. Right. Which I

11:21

think is important because

11:23

like that independence, you

11:25

know, I'm not trying to sell you anything, you

11:27

know, but I'm also not being distracted by the

11:29

next shiny thing. Right. If I a lot of

11:31

organizations, they're like, well, I want to try this

11:33

new technology. I want to try this new tool

11:35

because they've sold, they've, they've bought into the commercial

11:37

or whatever they saw some podcast or they saw

11:39

some webinar of someone talking about this great tool

11:41

and how, you know, it saved world hunger for

11:44

them. Right. And so I want to try

11:46

that here. Well, you could try

11:48

that here, but that might not be

11:50

your biggest pain point, right? That might

11:52

not be the place where it's costing,

11:54

you know, your throughput of your pipeline, right?

11:56

And you have limited resources to

11:58

have that evaluation, someone coming, okay, yes,

12:01

I agree that that would be helpful,

12:03

but this is really what's costing you,

12:05

you know, a fortune, right?

12:07

This other thing over here. The other thing would

12:09

be bonus. It would benefit you, but not as

12:11

much as this other thing over here, right? And

12:13

it's not always the most shiny coin. It could

12:15

be like the dullest, most boring thing in the

12:17

world, but really has an impact, right?

12:21

Right, so we can provide a

12:23

set of requirements to get to

12:25

a certain level of capability and

12:27

maturity for what they have, and

12:30

maybe that shiny thing is on

12:32

that path, and maybe it's not.

12:34

We can help. guide the decisions

12:36

that way if that's something that

12:38

they're interested in. So what is

12:41

capability maturity? So

12:43

when you're starting

12:46

out creating your

12:48

pipeline from start

12:50

to finish for

12:52

how to create

12:54

your software, At

13:01

the low levels of maturity,

13:03

you're probably going to have

13:05

a lot of manual processes

13:07

or scripts that you're creating

13:09

and doing things bit by

13:11

bit. And

13:14

as your

13:16

capability improves and

13:18

matures, you're

13:20

automating more of those things. You

13:23

might be bringing in

13:25

additional tools to improve

13:27

the security state. you

13:31

might have multiple

13:33

tools that are

13:35

checking source code

13:38

and for static

13:40

issues or dynamic

13:42

issues. And

13:45

so you're adding tools to

13:48

the mix to improve the

13:50

quality of your final code.

13:53

And as you do that, That's how

13:55

we're saying that the capability is maturing. We've

13:58

taken the process and we've actually

14:00

defined that. We've developed the DevSecOps platform

14:02

independent model, which has over 200

14:04

requirements in it. It's broken down into

14:06

10 different capabilities. We

14:08

really broke out that level one through four.

14:12

When I started building

14:14

that, we really didn't want to

14:16

do that level one through four. We just

14:18

wanted to model DevSecOps. The

14:20

problem with that is, If

14:23

I build this model, it's

14:25

overwhelming. If you're standing up

14:27

a pipeline from scratch, where do you begin? Do

14:30

I just study this shiny object or

14:33

this cool tool or really what's the

14:35

holistic basic stuff I have to do?

14:38

And so if you aren't overwhelmed by the

14:40

200 plus requirements, you have the level one. It's

14:43

just good core software engineering. If

14:46

you're not doing these things, something's going to

14:48

bite you because these are just best practices.

14:50

I don't give you to do it manually.

14:52

Right. And you don't have to start with

14:54

do all the level one things before you

14:57

start on the level two things. You're going

14:59

to have various capability maturity levels for different

15:01

things. But but

15:03

you can get a good

15:05

idea for how mature

15:07

your system is by how

15:09

many of those the

15:11

level one things you've done.

15:14

And then once you get most of

15:16

the way there. You can focus

15:18

on level two, level three. But really,

15:20

it's a matter of we're not

15:22

telling you what you should do next.

15:25

But we're pointing out these are the types

15:27

of things that you can do to help

15:29

mature your model, mature your

15:31

system and help you decide

15:33

on where you might want

15:35

to focus to improve next. So

15:38

what are your thoughts, Aaron? I've

15:42

seen, in my experience, two types of

15:44

organizations who would benefit from something like

15:46

our platform independent model. The first type

15:48

of organization, someone who's new, building a

15:50

new program. They just want to know

15:52

what to do. The platform independent

15:54

model is good at laying out,

15:56

here are the capability areas

15:58

that you need to focus on.

16:00

I mentioned earlier that mature

16:02

DevOps is about automation, but that's

16:04

the last step along a

16:06

path. you're not going to start

16:09

off by automating anything until

16:11

you know that that's what you

16:13

want to automate. And

16:15

so enacting manual steps in

16:17

a journey towards DevOps maturity

16:19

is very important because an

16:21

organization wants to at least

16:24

start to do everything broadly

16:26

correctly and get set out

16:28

on the right path and

16:30

then use something like the

16:32

platform independent model to help

16:34

mature their actual

16:37

practices, adopting more

16:39

tools, optimizing their techniques

16:41

and processes. Right, because if

16:43

I start with tools too early, I

16:45

end up in the wrong process. Is

16:47

it the right tool? Is it the

16:49

right process? By the wrong tool. Those

16:51

questions are really important and it's some

16:53

trial and error that an organization needs

16:55

to go through in order to find

16:57

the right answer. I think almost every

16:59

developer I've ever met has felt the

17:02

experience of being forced to use the

17:04

wrong tool in their development environment. They

17:06

go, well, management bought this tool that

17:08

you have to use, which doesn't actually

17:10

support their process and support their approach.

17:12

Worst case scenario, you support the tech stack

17:14

that you're doing the development on. Know

17:17

what you want to do before you

17:20

go buy the product. Know what you want

17:22

to do before you... These tools are

17:24

very flexible. And it's better to start small.

17:28

Start with the minimum set of things

17:30

that you need to do to

17:32

get your job done on the first

17:34

day. especially programs that are starting

17:36

out, rarely is there

17:38

a big bang approach to

17:40

everything where they're going to start

17:42

off and implementing a humongous

17:44

system and all the pieces and

17:46

all at once. That's

17:48

a path to failure. You're

17:51

going to start small and figure

17:53

out what works for you. You can

17:55

have multiple small teams trying to

17:57

figure things out, but then kind of

17:59

coalesce around what is working for

18:01

you and then look to mature. So

18:04

you can use the PIM to start with and

18:06

figure out these are the things we need to

18:08

do at a high level. And then

18:10

you can come back to it later, and this

18:12

is the second type of organization, are those that

18:14

are looking to improve. We have practices in place,

18:16

but maybe we have some challenges. Maybe we have

18:18

some bottlenecks. I'm not quite sure where they're at.

18:21

Help us try to figure out, in a

18:23

neutral and biased way, as Brent mentioned, help

18:25

us figure out what the problems are, identify

18:28

them, and solve them. And then we can

18:30

help them mature, get to those level two,

18:32

three, and four. That is

18:34

what is exhibited of very mature

18:36

practices. To some it's like treating

18:38

your pipeline, you treat your product,

18:40

right? Start with a minimal viable

18:42

product, and then you expand the

18:44

capability to improve your throughput and

18:46

improve your efficiencies, just like you

18:48

would try to improve your product.

18:50

Precisely, and in DevOps, your capability

18:52

and your capability is part of

18:54

your capability. It is all part

18:56

of the same thing. Your

18:58

development pipeline, your tool stack that goes

19:00

into delivering what ends up in

19:02

the hands of the warfighter, it has to

19:04

be part of the same singular whole. It's

19:06

not two separate things or multiple separate things.

19:08

It has to be treated as all of

19:10

one thing. And we keep using developers and

19:12

operators, but there is that sec part, that

19:14

security piece, right? And dev sec ops. Yeah,

19:16

we are certain we shouldn't do more of

19:19

those. And so, right. So the part is

19:21

holistically, you need to think about security, right?

19:23

You got to secure both your pipeline and

19:25

your product. a traditional engineering perspective, you

19:27

just worry about, know, you build

19:29

your product and build a mode around

19:31

it, right? And that doesn't work

19:33

with this ecosystem of your tools, really

19:35

connecting your development environment and your

19:38

production environment, right? And they're intertwining, right?

19:41

How does the SEC work into some of those

19:43

things? I'm sure the DevSec

19:45

has. How's that perspective always included as

19:47

well? Any thoughts? So,

19:53

It needs to be

19:55

holistic and baked in.

19:57

So DevOps, developers and

19:59

operators working closely together. Security

20:02

has to be something that is thought

20:04

about at every step along the way.

20:06

You mentioned that we need to secure

20:08

the pipeline against attacks. We learned in

20:10

SolarWinds that if you attack the pipeline,

20:12

you can attack the end system in

20:14

very insidious and hard to detect ways.

20:18

So security needs to start at that

20:20

very beginning. And

20:23

not trusting... I don't want to

20:25

kind of throw zero trust in

20:27

there because it's not a zero

20:29

trust conversation, but not trusting any

20:31

parts of your process implicitly. Assume

20:33

that you're being attacked. I think

20:35

about how an attacker would attack

20:37

your system at all levels at

20:39

injecting code into the code into

20:41

your source code repository or supply

20:43

chain or attacking a tool. I

20:45

attack a compiler and well now

20:47

I can now inject binary code

20:50

in that's not detected in static

20:52

code analysis for instance. There's a

20:54

lot of different attack vectors that

20:56

you need to think about and

20:58

you need to look at each

21:00

one individually and see how am

21:02

I going to attack this part

21:04

of my process. Well,

21:06

mitigate that issue. And then mitigate that. Right. Yes.

21:09

So I think you've probably done the most

21:11

assessments out of the three of us

21:13

here. You've done it both before we had

21:15

this PIM as an evaluation as well

21:17

as after using it. What's the

21:19

difference? Well,

21:22

the nice thing about the

21:24

PIM, you mentioned it's got like

21:26

230 or so requirements. And

21:30

before the

21:32

PIM, we started

21:34

off with a list

21:36

of hundreds of questions

21:39

Like almost a hundred

21:41

pages worth of questions

21:43

for questionnaires and trying

21:45

to do that We

21:47

wanted to try and

21:49

get our assessments down

21:51

so we could do

21:53

them within a day

21:55

and a half And

21:57

you know asking 300

21:59

questions or more and

22:01

so Part

22:03

of that was distilling

22:05

it down into a

22:07

couple dozen core topics

22:10

of interest. Which would

22:12

be made of our

22:14

10 capabilities became that,

22:16

right? Right, so

22:18

the 10 capabilities each

22:20

make up three

22:22

to eight of these

22:24

topics. we

22:33

try and make it more of a

22:35

conversation. And

22:37

so when we're talking,

22:39

we're not necessarily

22:42

limiting ourselves to just

22:44

like the one

22:46

question at hand. If

22:48

they happen to mention something and we

22:50

know, oh, it's on that list of topics

22:52

of interest, then we can make a

22:54

note of that. And it's a whole lot

22:57

easier to do that when you've got

22:59

a smaller number of things rather than flipping

23:01

through hundred pages of

23:03

questions and saying, oh,

23:05

we'll get to that on

23:07

page 42 and do

23:09

things. But

23:12

one thing that the

23:14

PIM provides us, once

23:16

we do have all

23:18

those questions answered, is

23:21

we have a

23:23

better feel for where

23:25

they are at

23:27

the maturity level that we talked about. So

23:29

we didn't have any type of concept of

23:31

maturity or where you are on this path

23:33

to perfection, right? Right. As a subject matter

23:35

expert, I could have a feel for how

23:37

good they were, but how

23:40

to translate that into

23:42

an objective assessment was more

23:44

difficult. We could say

23:46

now, oh, you're not doing

23:48

these maturity level three

23:51

or four things. You're

23:53

focused on the maturity level one and

23:55

two things. And

23:57

so or just like well why why you focus

23:59

this level four thing when you haven't figured

24:01

out this level one thing, right? Yeah,

24:03

don't worry what right? So about you just

24:05

have that rational conversation with this I

24:07

like I call it's an authoritative source where

24:09

before it was just a bunch of

24:11

subject matter experts giving you their opinion We're

24:13

here. It's like no really like I

24:15

can tell you what requirement you're not meeting

24:17

now if you think I'm wrong in

24:19

my assessment Well, then show me the evidence

24:22

prove me that you are doing this requirement. You

24:25

can't have that definitive of a conversation when

24:27

you present the findings when it was just, oh,

24:29

I just asked you a bunch of questions

24:31

and you gave me your answers. All

24:34

that we do more than just ask questions.

24:36

We look at documentation, we say, show us

24:38

if we can, we do a day in the

24:40

life. So we try to be more holistic

24:42

than just interviews. So part

24:44

of the assessment, we'll send out

24:46

a pre -visit questionnaire. which is

24:48

going to focus on the what,

24:50

the where, the how many kinds

24:52

of questions, things that

24:54

more than likely the development

24:57

and operations folks already have. And

24:59

hopefully the security folks also have

25:01

it so they know where all

25:03

the pieces are. What tools are

25:06

you are? What type of, are

25:08

a Linux based infrastructure, your Windows

25:10

based shop, like really basic stuff?

25:12

Right. Finding out what tools

25:14

you're using, which If we were to ask,

25:16

that's going to take a lot of time.

25:19

We want to reduce that

25:21

because we don't need a

25:23

dozen experts in the system

25:25

that we're evaluating standing around

25:27

saying, oh yes, we use

25:29

Linux. We're using Docker. We're

25:31

using whatever other technology they

25:33

happen to be using when

25:35

we could just get these.

25:37

ahead of time, and then

25:39

we know that we can

25:41

integrate that into our questions.

25:43

Really ask more informed questions,

25:45

stuff that would be hard

25:47

to to glean out of

25:49

a document or, you know,

25:51

a questionnaire. Right. Right. And

25:53

so once we have those, that pre

25:55

-visit questionnaire, those responses, then we can

25:57

go back to the customer and say, okay,

26:00

this is what we see. Are

26:03

there anything, we

26:05

notice that you don't have answers

26:07

to this, are those

26:10

topics that you're that are interesting

26:12

to you at all because it's

26:14

quite possible that some of the

26:16

things that are in the PIM

26:18

are things that are not of

26:20

interest or value to the customer.

26:22

We've had folks who are not

26:25

at all interested in model -based

26:27

systems engineering. That's something that we have

26:29

in our PIM. It's not

26:33

necessarily required to do any of the work, but

26:35

it has its uses. But

26:37

if the customer is not interested in

26:39

them, then we can just cut out

26:41

those questions. We can tailor those requirements

26:44

out of the assessment. To me, I

26:46

think my favorite part is the not

26:48

applicable category. Because

26:50

we're not saying you need to do all 230 requirements.

26:52

We're saying do the ones that make sense, but

26:54

consciously do the ones that you don't want to do.

26:57

Don't not do it because you didn't think of

26:59

it. Don't do it because it doesn't make any

27:01

business sense for you. Consciously evaluate

27:03

that requirement go, that

27:05

just doesn't apply to us. And

27:07

we're totally okay with that. As long as you

27:09

explain why it doesn't apply to you. It's

27:13

the ones that do apply to you that you're

27:15

not doing that you should pause and go, well, why

27:17

am I not doing that requirement? If

27:19

you haven't gotten to it yet, that's

27:21

fine. You

27:23

can put that on your path to

27:25

do it. Sharon,

27:31

you've done both types of assessments, pre and

27:33

post, having the PIM. What are your thoughts?

27:37

The latest assessment I did last

27:39

year was greatly benefited from the

27:41

PIM. Which I think was the

27:43

first time you used it, was

27:46

it? It was the first time

27:48

I used it. It was good

27:50

to have at hand that set

27:52

of 200 plus requirements broken down

27:54

within capability areas. Because

27:56

what it allowed me, as Brent mentioned, we

27:58

can start to hone in on things during

28:00

an assessment. Yes, they've said a lot of

28:02

things about these things, but haven't said a

28:04

lot of things about these things. And it

28:06

allows us to discover the things that aren't

28:09

being said. Because when you ask a person,

28:11

what are you doing? or what are your

28:13

processes, you're focusing them, or you're

28:15

biasing their answers, and you're going to get

28:17

the things that they are familiar with. But

28:19

if you don't directly ask for the things

28:21

that nobody is there talking about, then you

28:23

might have a blind spot in your assessment.

28:26

You might, you may miss it. So as

28:28

an assessor, it allows me to make sure

28:30

that I focus on everything, at least get

28:32

an answer. Like I said, non -netbookable is

28:34

an answer, and it's valid. We

28:36

will validate whether their answer is

28:38

good or not. as

28:41

we look through the lens of

28:43

what their mission is, because that's

28:45

one key thing is DevSecOps works

28:47

for the intended purpose and the

28:49

intended environment. And so we

28:51

don't just want to take someone's answer for

28:53

it, saying, well, we don't think that's important.

28:55

We can look at it and say, well,

28:57

your mission, because of your types of mission

28:59

and our experience in similar types of missions,

29:01

we actually think it should be important for

29:03

you. Allow us to have a conversation around

29:05

those topics as well. But

29:07

the previous assessments that I've done

29:09

go back many years, at

29:12

least 10 years. And in that

29:14

time, it was very much with the wild west. It

29:17

would just bring in subject matter experts, throw

29:19

them into an environment and say, what

29:21

do you see? What don't you see?

29:24

And it comes down to the skill

29:26

of that assessment team. So

29:28

it's not repeatable. It's not repeatable at

29:30

all. We really would just go in with

29:33

a set of questions, maybe a concept, and then

29:35

we just go. And we

29:37

find what we find. And

29:39

so the quality of the assessment

29:41

couldn't be high, but with a

29:43

lot higher effort. As

29:46

Brent said, we want to be able to get

29:48

assessments down to a much smaller time box and still

29:50

cover the same material. And with the pin, we

29:52

can do that because we're able to get more methodical

29:54

approach to things. So

29:56

we're very focused on assessments and

29:58

DevSecOps in general. really

30:02

the modern term for just good

30:04

software engineering, practices, tools, and techniques. So

30:07

what are you two working on next

30:10

three, four months that we can bring

30:12

you back out here and talk some

30:14

more about your research and your work?

30:17

Erin, like to go first. So one challenge

30:19

area that I'm seeing with my

30:21

mission partners is there's now a lot

30:23

of guidance. There's the software acquisition

30:25

pathway, which is in the news right

30:27

now in DoD. the

30:30

new administration is really pushing its use

30:32

for software centric systems. But it's only

30:34

been around for a few years. DevSecOps

30:37

has been in government for maybe

30:39

a decade, a decade and a half.

30:41

Agile has been around for about

30:43

30 years. It's marrying all

30:45

three of these things together. And

30:47

how, putting myself into

30:50

the shoes of a

30:52

program manager, how do they

30:54

take all of these things together and operationalize

30:56

them? Where is that template of I

30:58

have to pull all of these things together

31:00

and execute a program? How

31:03

do I do that? What is

31:05

a good configuration, quote unquote, for

31:07

a combination of these tools and

31:09

techniques that work for me? There's

31:11

no one size fits all. It's

31:13

not one size fits all. And

31:15

it really needs to be tailored

31:17

to the mission that the system

31:19

is going to work in. And

31:22

because there's no one size fits

31:24

all, there's no No template that

31:26

that will instruct a program manager.

31:28

Hey, I just go down through

31:30

this checklist and I will do

31:32

good DevSecOps and I'll have good

31:34

outcomes. That doesn't exist. So I

31:36

really want to work towards, can

31:38

we put actionable guidance based on

31:41

our experience supporting these programs that

31:43

is broken down along the lines

31:45

of maybe there's not one template,

31:47

but maybe there's a good set

31:49

of questions or Recommendations

31:51

typically comes out of the come out

31:53

of assessments. Okay, we found these things. What's

31:55

your path to good? Thinking

31:58

about that before a program needs to come

32:00

to us. Typically they come to us when

32:02

they're having challenges. Let's help them before they

32:04

have those challenges and get them set off

32:06

on their right foot. So one area I'm

32:08

looking in is like a template for a

32:10

good software development plan that comes from guidance,

32:12

these high level guidance. down to

32:14

actual actionable steps that says here's a

32:16

strategy and a plan that maybe a

32:18

program could use as part of an

32:20

RFP package when they're looking for someone

32:23

like a system coordinator or system integrator

32:25

or that first contracting with an organization

32:27

who's going to build their capability and

32:29

say this is the plan that we

32:31

are going to work towards as a

32:33

program. We the government are going to

32:35

own this. This is how

32:37

we want to work and this is how we

32:39

expect you to work for us. And that sets

32:41

you up for things like metrics and measurement and

32:44

value assessment and things down the road. Which

32:46

is what actually was one of the,

32:48

we talked about just doing assessments to

32:50

the DevSecOps platform independent model. But the

32:52

other part is these requirements can be

32:54

used to define what DevSecOps is. Right

32:57

from from a contractual from a program

32:59

office perspective, right? Correct. Program offices don't

33:01

normally build anything, right? They acquire capabilities

33:03

either organically by, you know, getting an

33:05

engineer in group together or they have

33:07

a contractor. You contract it all out

33:09

and they just deliver a capability back

33:12

to the government, right? Um,

33:14

and so the question is, but what is

33:16

DevSecOps, right? Because I think way down

33:18

the path of building the model wasn't just

33:20

for assessments. It was. I want

33:22

to do threat analysis. I want to

33:24

do cybersecurity type of stuff. And you

33:26

helped me do that back in the

33:28

day. And the problem was I could

33:30

do a web search for DevSecOps. And

33:32

I got one of two things. I

33:35

got the really high

33:37

level theoretical stuff. Or

33:39

I got a platform specific

33:41

solution. Use this tool

33:43

and I'll solve world hunger. There's

33:46

nothing in between. There was

33:48

no good formal definition of what

33:50

DevSecOps should be. And so

33:52

the PIM kind of helps. It doesn't go as

33:54

far as what you're saying, too. But it does

33:56

at least give you the requirements that one could

33:58

say, you know, contractor A, build

34:00

me a development pipeline for my capability.

34:03

And I have a way to say, did you actually

34:05

deliver what I contracted you to do? So there

34:07

is some acquisition things you could use with the PIM

34:10

as well. So Brent, how

34:12

about you? What do you think you could bring you

34:14

back here and have another conversation about? So

34:16

one thing I'm working on now

34:18

is related to reliability engineering. So

34:21

we get, you know, with

34:23

hardware, you know that there's,

34:26

you know, meantime between failures

34:28

and how likely something is

34:30

to stay up. With

34:33

a lot of

34:35

the containerization and

34:37

virtualization cloud stuff,

34:40

the focus is, you

34:43

know, we know that things are going to

34:45

fail. So the

34:47

goal is to make

34:49

sure that they

34:51

recover quickly and seamlessly

34:53

so that the

34:55

customer does not notice

34:57

any downtime or

34:59

minimal downtime. But

35:01

getting back to one of the early

35:03

questions about, does it

35:05

have to be on using containers

35:07

or cloud? If

35:09

you have bare

35:12

metal systems and

35:14

you can't do

35:16

the replication of

35:18

services, How can you

35:20

ensure reliability? And so

35:22

that's one thing that I'm working on and

35:24

investigating that. Awesome. So, Aaron, Brent,

35:26

thank you for your time. I appreciate it. For

35:29

the audience, we'll include

35:31

the links to the transcript, the resources

35:33

minted during the podcast. Finally,

35:36

a reminder to our audience

35:38

that our podcasts are available

35:40

on SoundCloud, Spotify, and Apple

35:42

Podcast, and the SEI's YouTube channel. If

35:45

you like what you see, And here today,

35:47

please give us a thumbs up and thank

35:49

you for joining us.