0:07

Hello , welcome to Microsoft Community Insights

0:09

Podcast , where we share insights from community

0:12

experts to stay up to date in Microsoft . I

0:14

am Nicholas and I'll be your host today . In this podcast

0:17

, we will dive into the world of container

0:19

security , but before we get started , I want

0:21

to remind you to follow us on social media so

0:23

you never miss an episode , and it will help

0:25

us reach more amazing people like yourself

0:27

. Today we have a special guest

0:29

called Josh Duffy . Could you please

0:32

introduce yourself , please ?

0:33

Absolutely . First , the intro . That's the longest five

0:35

seconds I've ever experienced

0:37

. Very exciting , I'm just kidding . Yeah

0:39

, so my name is Josh Duffy . I'm a senior cloud

0:41

advocate at Microsoft . My

0:45

background we were chatting a little bit about this

0:47

before we went live . Let's see here I'll give this a succinct

0:49

version , if I can . We can dive into more . So I wanted

0:51

to be a video game designer because

0:53

I played a lot of video games in high school , decided

0:56

I didn't want to have $100,000 of student

0:58

loan debt right starting life . So

1:00

I decided to go to a community college and

1:02

the only thing computer related they had was this computer networking

1:05

degree , an associate's degree , and it

1:07

turned out to be kind of an introduction to everything

1:09

. I had a networking class , a

1:11

full semester on printers , which was very painful I

1:13

still dislike printers to this day . But anyway

1:15

I landed a help desk job from there

1:18

, got really into like system

1:20

administration , worked my way up to

1:22

like a one man IT shop running

1:25

a small bank and racking servers , and then I kind of got into PowerShell

1:27

and so a lot of people have seen my name before

1:29

. It's probably because of the stuff that I used to do with PowerShell

1:31

. But anyway , that led me kind of

1:34

on the DevOps path . Previous to Microsoft I

1:36

was an SRE at Stack Overflow and then I

1:38

got recruited in by Mike Robbins

1:40

, who's familiar in the PowerShell community too

1:42

, to be a docs writer or a content

1:44

developer , I think is the technical term for

1:46

that , and I focused on Ansible and Terraform

1:48

. I had written a book on Ansible and

1:51

successfully implemented it in a couple enterprises and

1:53

stuff previously . I did the docs

1:55

writing for about 18 months and

1:57

then I got the opportunity to join the cloud

1:59

native team here at Microsoft in the DevRel

2:01

organization for advocacy work , which was

2:03

a good split for me between some of the content creation

2:06

and speaking which I enjoy doing , and I'd

2:08

done that a lot in my past . I ran a PowerShell user

2:10

group and other user groups before , spoken

2:13

at many things , different conferences , so it seemed

2:15

like a good fit . And then I landed somehow

2:18

in container security for

2:20

probably like the past two years or so . So that's

2:23

like I don't know how long have I been working Like

2:26

12 years , I don't know . 12-year snapshot right

2:28

there .

2:29

So before you land your job at Microsoft and

2:32

you are in Stack Overflow . Did you have experience

2:34

in Cloud Native or just be using

2:37

PowerShell to create like meetups

2:41

?

2:43

I wouldn't have put it under like cloud native , but

2:45

I definitely had a lot of cloud experience

2:47

. What's the introduction to

2:50

Azure ? So before Stack Overflow , the

2:52

company I worked for had AWS and Azure

2:54

, had to work in both clouds and we had on-prem so

2:56

we had to do all that . So I'd been working

2:58

in the cloud for a number of years

3:01

, probably before the term cloud native came

3:03

about and just I

3:05

don't know , it seemed like the things that I was naturally doing in

3:07

the cloud and then having

3:10

them change , I suppose , from on-prem

3:12

and stuff , and then it just became a term

3:14

. So it's one of those situations where you've

3:16

just been doing a thing for a while and then all of a sudden

3:18

, uh , the broader ecosystem comes up with

3:20

the term for what it is that you've been doing for

3:22

a while .

3:23

Yeah , because I remember watching some of your

3:25

episode on

3:28

Bash . You know Bash as

3:30

well as PowerShell . It's quite handy .

3:32

Yeah , I got forced into that once . I Very

3:35

interesting paradigm shift . So

3:37

outside Microsoft was Windows

3:40

System Administrator , powershell guru

3:43

doing DSC like even wrote my own

3:45

. Actually , dscv3 just came out and they

3:47

have the ability to do JSON representation . Back

3:49

in 2016 , I did a talk where

3:51

I wrote a module that abstracted DSC

3:53

into JSON , so that's pretty cool to

3:55

see that . I don't think they used any of my work or

3:57

anything like that , but just that idea transferred

4:00

. Yeah .

4:00

Okay , so today's session

4:02

is the world of container security

4:05

, so let's talk about the main

4:07

, like the introduction

4:09

. So what's container security and why

4:12

is it important ?

4:12

The area that I've been working in container

4:15

security is pretty broad , so I'll narrow it

4:17

down to the areas that I've been working in . I've

4:19

been working with a couple open source projects

4:21

Notation , which is a digital

4:23

signer you can think of it that way yeah

4:29

, and then they're ratify , which is an emission controller for Kubernetes , and then copacetic , so that

4:31

would be for the patching , and all those are related to the software supply

4:33

chain and containers . So container

4:35

security most people would think immediately

4:38

think chain guard that's really focused on the images

4:40

and make sure that those images in

4:42

the best possible way . That's a big part of

4:44

it . A lot of the work that I've been doing is on somewhat

4:47

on the other side . Right , okay , we have this

4:49

good image that we trust

4:51

or want to trust . How do we signal that trust

4:53

? And that's where the signing comes into play . And

4:55

then how do we know what's in here and that would be

4:57

something like an S-bomb and all of these are

5:00

. It's been an interesting thing

5:02

to watch in the industry because it's more

5:05

work and definitely not something that the average

5:07

developer wants is more work . But

5:09

there is a lot of pressure from especially

5:11

like government agencies and working contracts

5:14

with government agencies where they're now mandating

5:16

these types of things to improve the supply chain

5:18

, to avoid supply chain attacks

5:20

from happening . It's an evolving

5:22

ecosystem .

5:24

One of the important points is keeping

5:26

your images up to date and don't have

5:28

updated images , because it would lead to

5:30

having vulnerabilities in your code .

5:32

So yeah , I never writing

5:35

. So I wrote a series on Copasetic

5:37

, which is a patching tool , and

5:39

in writing that it just brought me back to

5:41

my first kind

5:43

of epiphany as I was learning Kubernetes and getting

5:46

into containers , when Docker was really

5:48

starting to come on the scene . And

5:50

one of my first thoughts I remember because I had battled Windows

5:53

updates like my entire career VM updates

5:55

and I was like , oh , finally I'm not going

5:57

to have to worry about this problem again . Here we are

5:59

, 10 years later . I'm having to worry about

6:01

, you know , vulnerabilities of my

6:03

images that I have to patch or

6:05

you can patch by updating the base image . We can

6:07

get into more detail with that later , but at the end

6:09

of the day there's still vulnerabilities in these

6:12

container images and systems . It didn't solve

6:14

the patching problems per se Exactly

6:17

. You know like we still have to have systems around updating

6:21

things to make them not vulnerable .

6:22

Okay , so the

6:25

free tool , Copasetica

6:27

, and the other one take it there . Cncf

6:30

approved yeah sandbox . Can

6:32

you use it via Helm

6:34

or is it like automated Ratify

6:36

? You can .

6:39

So that is the only of those three projects that

6:41

would sit on your cluster . The other two , notation

6:43

and Copasetic , are both CLI

6:46

tools that have GitHub Actions , so we

6:48

have to automate it .

6:50

So what are some of the use

6:52

cases of like impact

6:54

, use cases that you think both of the tools

6:56

do have ? Do you just have to use

6:59

all of them side

7:01

by side ?

7:02

You can use all of them . So here let me . Actually

7:05

let me just show you a repository . Would that work

7:07

? Yeah , here let me , let

7:10

me clean up my very messy desktop

7:13

.

7:13

Oh yeah , before I mention , I remember

7:15

the tool that we use A net

7:17

scaler , no , a new netter .

7:19

Oh , I've not heard of that one .

7:20

It's like a container scaler thing , it's like

7:22

a container-skeletal thing .

7:23

It's like an images thing . Does it have

7:25

a scanner ?

7:29

Yeah , but you have to pay for a license . It's not open source

7:31

, I think . Ah , okay , it may have a container-skeletal

7:33

license .

7:35

Share screen . Let's see . Oh

7:38

no , Is it going to make me have to drop ? I've

7:40

used this before . Let's see here Window

7:42

.

7:43

I want this one . I think it's sent

7:45

Yep .

7:46

All right , fantastic . So this is a little

7:48

outdated . I

7:51

will get to . If my build talk gets submitted , I will definitely take another pass at this , but

7:53

here this is kind of . This will give you the full picture here

7:55

. So if we look at this , this little graph here

7:57

will give you an idea . So

8:03

there's another tool I haven't actively contributed

8:05

to , but I use all the time , which is

8:07

Trivy . So Trivy is a vulnerability scanner . It's probably

8:09

you , might it might be using your tool

8:11

, might be using it .

8:12

I think you can scan

8:14

like infrastructure tool , like Terraform code as well

8:16

using Trivy oh yeah , yeah , yeah , misconfiguration

8:19

and secrets and stuff .

8:20

Sure , absolutely yeah

8:25

. So Tribu is just a vulnerability scanner and misconfiguration scanner , for lack of a

8:27

better word . I'm sure their bio will give a more accurate description

8:29

. But essentially , what that allows you to do is to generate

8:32

a report of , like what's vulnerable in

8:34

this image . So what you can do is you can hook

8:36

these all up with GitHub Actions , which I'll

8:38

show you in a second , and then the next step would be

8:40

wouldn't it be super cool if we

8:42

could remediate those vulnerabilities based on the

8:44

report ? And that's where Copasetic comes in . So Copasetic

8:47

will take care of OS level vulnerability patching

8:49

for container images and it can

8:51

ingest trivia results to do targeted

8:53

patching . It also has the ability to just patch

8:55

everything to the latest on your system If you want

8:58

to . You're feeling risky and you want to

9:00

, you want to go that route , you can , but

9:02

the trivia one will just I

9:05

found a Cve for libc , I'm going to update libc or whatever that might be , and

9:08

the report takes care of that , uh , and then ? So

9:10

now we have this image that we feel pretty good about . How do

9:12

we signal trust ? And that's

9:14

where the notary project or the notation

9:16

clr comes in , and you can more or less stamp

9:18

. You put a digital signature on that container

9:20

image , a notation , so another another

9:23

option would be Cosign from

9:25

Sigstore , so you could use that as well and that creates a

9:27

digital signature on the image and

9:29

that becomes important later on . But there's a distinction

9:32

between . Well , I think they can do both . If you

9:34

use the beta features for Cosign

9:36

, they both can use the referrers API

9:38

or it'll create , like another artifact

9:40

, a signature artifact on your registry , and

9:43

I think I even might have some packages that we

9:45

could see what that looks like on here

9:47

. We'll see if I do no , but maybe yosh

9:49

does so .

9:50

Have you used this on production environments

9:52

for , or is it just like poc

9:54

?

9:55

well , I know it's being used in production inside

9:57

microsoft , but the nature

9:59

of my role is I they don't let me in production very

10:01

often , right ? So they actually just

10:03

, uh , you kick me out of , like production

10:05

tenants and stuff , and that's just the nature

10:08

of my role in the advocacy . You know I'm not responsible

10:10

for any internal production systems

10:12

. I don't want the Wasm cloud

10:14

, it's all it's my cloud .

10:15

Yeah , so to update the image , you can just have

10:17

different base files for each of the

10:19

image user .

10:27

And then you can update it right via helm or cli . Yeah , uh , so there's different patching strategies

10:29

. If you want to one second and we'll go in a little segue about like

10:31

why would you even want to patch an image and not just

10:33

update the base image ? Because that's a valid , that's

10:35

a valid question . I wanted to show

10:37

signature real quick . Is

10:40

this the right one ? Now I want packages

10:42

and not there , so so this is a signature

10:44

here . This SHA-SIG , I

10:46

guess . Yep , and so that's what Cosign

10:48

does by default is it'll put the signature

10:50

on the repository Notation by

10:52

default , will make it through the refer , so

10:54

it won't show up here . It'll

10:59

actually be in , like the manifest metadata , if you were to like pull up latest and in here we would

11:01

have some information to the digest for the signature

11:04

and it can do that with . It works with , I think

11:06

, most registries , any OCI

11:08

compliant registry it works with .

11:10

So you have to modify the

11:12

image . If you want any custom image , make it

11:15

own .

11:15

It'll do it for you . Yeah , yep , so

11:21

the copacetic . You could overwrite the tag if you wanted , or you could

11:23

append , like attach , like a hyphen one to do a counter

11:25

, but we can go into that in a second . So we'll pause

11:27

right there , though , and just you know , like why would you

11:29

even consider patching container images when

11:31

, like , this is the build pipeline ? I could just

11:33

bump up , uh , the base image version

11:35

. When you can do that , then you know , I would

11:37

say patching , don't do patching . I think

11:40

what was it ? Chain guard just had like

11:42

something on their x profile of a big billboard

11:44

that they put up , that they had like patching something

11:46

, and they crossed it out and they said secure future

11:49

. I don't disagree . Like , if you have

11:51

the ability to use a better base image that doesn't

11:53

have CVEs , do that . The problem

11:55

comes when those vulnerabilities

11:57

are upstream and you have a dependency

11:59

on some other image and their

12:01

cycle is longer than yours

12:03

and they can't resolve that cve

12:06

. You're basically locked right , like I can't

12:08

upgrade a version because it's going to break

12:10

something or because there just is another latest

12:12

version yeah , not a lot of

12:14

the like couple seconds .

12:15

Like the upstream repo probably won't tell you the

12:18

vulnerability , like if you used to look at

12:20

a changelog , it probably won't say

12:22

just show the features . So you have to scan

12:24

it to see if that is very good .

12:26

Within the hash right image itself exactly

12:29

, exactly , yeah , and so it's most useful

12:31

when you've got an upstream dependency

12:33

that you just can't update the base image

12:35

for . How do you get rid of those vulnerabilities

12:37

, and copasetic is a good way to automate that

12:39

. Technically , it's super cool using build kit

12:41

to to break apart your container and patch it

12:44

and create a new page . It's pretty cool .

12:46

Have you used this for Azure

12:48

DevOps before , or is it just CLI ?

12:50

Oh yeah , it works in Azure DevOps , I believe

12:52

. What's the GitHub action equivalent for Azure

12:54

DevOps ? Is it just a task ?

12:57

Yeah , it's a task .

12:58

Yeah , so there's a copacetic task for it .

13:01

Okay , and

13:06

is it for the nodejs ? Now , the other one is trivia . So I know that is a trivia and then . Okay

13:08

, so a lot of them have to test . So is it

13:10

possible , just like , like , if you create

13:13

one , one base , one file , to

13:15

update image instead of updating lots

13:17

of image for each of the ?

13:19

the tool to have like a golden image . Is that what you're

13:21

referring to ? Yeah , uh , yeah

13:23

. So let's see here . Let's go to copasetics documentation , one of like . You have

13:25

a dot net image of .

13:25

Is that what you're referring to ? Yeah , yeah . So let's see here . Let's go to Copasetic's documentation

13:27

, one of like you have a NET image of NET and you can update that , then

13:30

it will pass the latest without and then

13:32

you can pass it with whatever image

13:34

you want , whether it's a patch image or

13:36

an actual NET , that is image I think

13:39

what you .

13:39

So you could do this , uh , dynamic

13:41

tagging where you basically

13:44

say , like here's , here's the base tag

13:46

that I'm patching , but then I'm always going to override

13:48

this version tag . So the patching you would want

13:50

to do per version . So there would be another

13:52

version of this . Like there would be another tag for

13:54

a point 1.25 , you would have another

13:56

one that is always patched . That's , that's one

13:58

tagging strategy that you could you could do . That'd

14:00

be the dynamic one . The other one is the

14:03

, the incremental one . So you

14:05

can just see there in the docs , you know , like the first

14:07

time it goes through and patches it's going to be hyphen one

14:09

. The second time will be hyphen two . The problem

14:11

with that is you , you have to go up and up

14:14

, you have to go back and update your manifests

14:16

and I'll have this ready . This would be cool to add

14:18

in . But I'm working on setting up a sample

14:20

repo that uses Dependabot to be

14:22

able to come back and say , hey , I detected a new patch

14:24

version of your image , just like it would

14:26

any other dependency update right , and

14:28

do a PR into the repo to say , hey

14:30

, let's update the image to a new

14:32

patched tag . The issue with

14:35

the other supply chain tools down the line is if

14:37

you're not using the tag

14:39

and you're using the digest . So , for

14:41

context and people that don't know , each tag has

14:43

its own digest . If you look back at this , the

14:46

SHA right here is the digest

14:48

, so you can actually reference container

14:50

images in your manifest by digest

14:52

. And that's the reason that's good is because

14:54

it's not mutable , meaning if

14:56

the image has changed in any way , it gets a new digest

14:59

and it won't pick it up , which

15:01

is inconvenient if you want that . But it also improves

15:04

the security , because now you know for

15:06

certain which exact tag and version

15:08

of the image you are going to have deployed . But

15:10

if copacetic patches changes

15:13

the tag and then that digest is actually used

15:15

by the signing tools . So the

15:17

other tool that I didn't get Okay .

15:19

so whenever you use the signing

15:21

tool , do you just get a new digest you

15:24

do yeah , okay , yeah

15:26

, I'm curious to know . Sure

15:28

, is there any like open

15:31

source tool that's equivalent to this

15:33

, that's like a web competitor , or is this

15:35

, is the one that's well known in microsoft

15:38

or in the cnc ? Uh ?

15:39

which tool .

15:40

Which tool ? Whether it's going to be copacetic

15:43

or notary

15:46

.

15:46

Notary . So the notation CLI . Its

15:48

counterpart would be cosine

15:50

. So the cosine CLI from Stixor . They

15:52

just , they , just . They just create the signatures

15:54

in a different way . The biggest . Here's what I would say

15:57

If you have a mature paykey

15:59

, public key infrastructure inside your company

16:01

like you're a large enterprise you have your own keys

16:03

. Notation is what you want to use . But if you don't

16:05

and you just want to have a signature , cosign actually

16:07

has something called keyless signing where it

16:09

will basically get rid of the private key so

16:11

it can't be used again , but then the public keys are made

16:13

available . I believe it was a cold record . So

16:16

that way you can verify the signature

16:18

but you don't have to maintain your own PKI infrastructure

16:21

. So that would be the big delineation , at

16:23

least in my mind . If you've got

16:25

a mature PKI system and you need to use your

16:27

own certificates , that's where notation

16:30

shines . And then cosign is on the other side

16:32

, where maybe you're doing more local development or

16:34

you just don't have the infrastructure , don't care to have

16:36

the infrastructure and you want signing , and

16:38

I'm sure they have offerings to kind of offload

16:40

the PKI infrastructure and they have , I'm sure they have

16:42

offerings to kind of offload the PayCat

16:45

infrastructure .

16:45

I'm just not super familiar with their product suite , so

16:50

I can see this improving by having a little dashboard , seeing

16:52

any outdated images , like when people like internal what developer or DevOps can monitor

16:54

, and then we can just like get

16:57

, we can just update the image

16:59

accordingly using automation , whether

17:01

it's GitHub or Azure DevOps Yep , there's something that can be built in like monitoring

17:03

on it if we just update the image accordingly using automation . Whether it's GitHub or Azure DevOps

17:05

Yep , there's something that can be built in like monitoring on it .

17:06

Kubescape might be doing something like that . This is a tool that popped

17:09

up on my radar recently . I haven't dove

17:11

into it yet , but it does use . It integrates

17:13

with Copasetic .

17:14

Okay . So this might actually be

17:16

kind of what you're envisioning you

17:21

probably can interface into like a GUI , right ? Yeah , when people like a front-end GUI . Okay

17:23

that's quite nice . It's easier for

17:25

like developers or anyone to

17:28

have login and concede vulnerability and

17:30

patch it themselves by running the automation

17:32

tool .

17:33

Yep , that would be . It's hard for me sometimes

17:35

to put on my S3 platform engineering hat

17:37

and want to build those types of things , but those

17:39

are lots of time . This is more putting

17:41

together open source tools , something . Yeah

17:44

, a product could definitely sit on top and use all of these

17:46

things together and make it a lot more seamless , and I think that's honestly

17:48

what needs to happen to make it

17:50

more palatable for companies and

17:52

organizations . This is a lot of additional work

17:54

, but this is what it would be doing under the hood . There

17:57

are a number of features inside Azure . One

17:59

would be the continuous patching for ACR

18:01

, so it uses ACR tasks to do the copacetic

18:03

stuff for you and then it takes it out of your

18:05

workflow and makes it more of a platform

18:07

checkbox type task .

18:09

Yeah , because normally we're trying to make everything

18:12

self-service so people can just run a pipeline

18:14

and then it will update the image . So we're just

18:16

putting the YAML and submit to PR

18:18

.

18:19

So whether it's looking at the dashboard to

18:21

see vulnerability or not , yeah , this would

18:23

all take care of it inside of a pipeline , but this

18:25

would require all the dev teams to kind of adopt a

18:27

similar workflow or a templated workflow . Right

18:30

, yeah , so the other end of it

18:32

.

18:32

Oh , go ahead . What's the result you get

18:34

at the end ? So do you get , like a

18:36

result of something that's outdated

18:38

, images with

18:44

SAP vulnerability and then someone can actually check whether it's an actual vulnerability or not

18:47

?

18:47

I don't know of anything that there's

18:49

. I think there's some Azure features that

18:52

would be able to do continuous scanning

18:54

on your registry and then present you with diagnostic

18:57

information there . I haven't looked into it too much

18:59

. That's what I'd imagine it would be going

19:01

Right now . It's point and click in the workflow

19:03

. Trivia runs inside that report

19:05

. You would see your vulnerability list . It's not displayed on any

19:07

kind of dashboard . It's displayed here . There are

19:09

flags with trivia that you can use to stop your workflow

19:12

and fail it if a certain number of CVEs

19:14

exist , even after you patch or

19:16

during the scan .

19:18

But yeah , there's no dashboard , at least

19:20

in my current map markup

19:22

there we go and you say you can

19:24

, this can be integrated with github

19:27

dependent bot or something that does the

19:29

pr scanning for vulnerabilities and

19:31

stuff , so you don't really have to scan it yourself

19:33

. You can get pr . That's been raised and just

19:35

so .

19:36

It's just reviewing really yeah , that's the

19:38

kind of the end to end for if you're just wanting

19:41

to focus on GitHub workflows

19:43

, and Dependabot obviously

19:45

works for Azure DevOps too .

19:48

Because I think I use something similar

19:50

called Renovate . It does image

19:54

for versions , but I think

19:56

it probably can integrate with this . It's

19:58

similar to Dependabot in a way Okay

20:01

cool , great

20:06

with this . It's similar to the pentabot in

20:08

a way okay , cool . So have you got any ? Do you know any examples

20:10

where this is used in companies , or it's just microsoft at the moment ? You know where

20:12

?

20:12

alibaba has been pretty good , uh , with

20:15

adoption on ratify . They've got some solutions

20:17

. So , yeah , let me go to the deploy phase . Um

20:19

, a little bit and just about that , and then

20:21

we can hang on to more customer

20:24

adoption . So , if we look at the bottom

20:26

part , so it assumed like we've got a signed

20:28

image on our registry and now we want to deploy

20:30

to our cluster . How do we put some

20:32

teeth to all this work that we did

20:34

during the build step right To produce

20:37

this artifact that we trust ? So ratifies

20:39

and gatekeeper combined , so you deploy

20:41

them together with a helm chart , and

20:46

what that allows you to do is do admission control for digital signatures or other verification policies

20:49

, so you can define some kind of rego policy . And in this example it's looking

20:51

for does a digital signature

20:53

exist for this public certificate

20:56

, right ? And so if I can't validate the signature with

20:58

my public key , that signature either doesn't

21:00

exist or is invalid , and I'm not going to let the

21:02

pod run on this cluster . And

21:04

so there's a lot of good use cases there , trying

21:07

to think what are the other customers ? By name

21:09

or if I should say names , but

21:12

yeah , there are definitely customers using it . A

21:14

lot of these are actually being baked into features

21:16

right Like features inside ACR features

21:18

inside AKS or whatever it might be . For

21:21

example , like with Ratify and Azure

21:23

, you can define an Azure policy that converts

21:25

to a Ratify configuration , so

21:27

that way you don't have to apply another kubectl

21:29

thing . It can just be any AKS cluster that

21:32

gets deployed to this resource group has to have

21:34

signed images , and then that translates

21:36

down to Ratify .

21:37

Yeah , I guess the good thing about it is

21:39

that it's a security aspect as

21:41

well , because it constantly gets updated

21:44

as well with the images , and

21:46

it's being maintained by the CNCF

21:49

, so that's a good side about

21:51

it .

21:52

Yeah , all those projects are . I don't know about Trivi , but Copasetic

21:54

, notation and Ratifier are all CNCF

21:56

projects .

21:57

Okay , brilliant . Do

22:00

you have a picture of what it will look like as a

22:02

result when you run it ?

22:04

Yeah , I think I have some successful

22:06

runs , or maybe I don't . I deleted them , I

22:08

don't know . We'll find out .

22:09

No worries , so it'll just give you a

22:11

trivia . I know trivia

22:13

just gives you SVE , whether it's

22:16

infrastructure code or any code in

22:18

general of the list , but

22:20

for the rest it just gives you the hash . So

22:23

it just trivies that it's bring up right Most

22:25

of the results .

22:26

Well , here let's just look at the workflow , we'll see . So

22:29

this is a little update . I'll show you a more recent one for

22:31

people that are more on the platform engineering side , because

22:34

a lot of this is being done on the

22:36

developer side , which might be hard , and

22:38

maybe you just care about patching the images

22:40

in your registry and I'll talk about a continuous patching

22:43

workflow here in a second

22:45

for at least GHCR . So

22:47

yeah , if we just let me bump this up , the

22:50

general workflow is we're going to log into Azure , log

22:52

into ACR , we're going to build our

22:54

app , right , and then we scan it

22:56

. So we scan with the trivia action

22:58

that's going to output this patchjson and

23:00

then we're going to patch it . So this can be

23:03

replaced . There's a GitHub action for this

23:05

. Now this can be replaced . I'll

23:07

show you that in the next one . We're going to patch the image

23:09

. So I'm just outputting what the digest is going

23:11

to be and then push the patch image . So

23:13

I'm going to push the patched image after

23:15

Copasetic patches it . So Copasetic is

23:17

going to patch it . It's going to create a new image

23:20

locally with the taking

23:22

prefix like hyphen one or whatever taking

23:24

strategy you're going to use , and then I'm capturing

23:26

that digest at the most optimal time

23:28

, which is right after it's built . I'm going to capture

23:30

it . So

23:37

there's no chance that I'm I'm grabbing the wrong digest and someone put another layer

23:39

on this image with something malicious in there and I didn't know . So I'm going to grab that digest right there and that's

23:42

where notation comes in . Notation has a , an action as well

23:44

, so I'm just setting it up and then I'm signing it

23:46

and then that puts a signature on the

23:48

refer API that can be validated with ratified

23:50

later . And then so if you looked at , I think I

23:52

have terraform code in here . This sets up the

23:54

aks cluster and

23:57

then I'm using it in the demo I actually set

23:59

up ratify . So it's not , it's not part of my

24:01

infrastructure as code , because it's part of the demo script

24:04

, but all this you can walk through as a demo . This

24:06

demo sh goes through every single

24:08

step in the workshop , so that's from

24:10

like a pipeline perspective . But let's

24:12

say that you wanted to like what would it look like to

24:14

continuously patch a

24:16

registry ? And so I started this repo

24:18

and I walked through this in a blog post series

24:21

for people that are interested .

24:22

Okay , it looks like . Oh , go ahead . This

24:24

would go through like ACR , like

24:27

for your , and it's patched all

24:29

the images to see if they're using the

24:31

correct images or not , if they have

24:33

any vulnerability .

24:34

Yeah , to see if they're using the correct image or not , if they have any vulnerability , yeah . So

24:36

there's a continuous patching . Private preview for

24:39

the AZ-CLI that basically

24:41

does just this . I just did it for GHCR

24:43

and I wrote a little . It gave me an excuse to write

24:46

some code which I can show you in a second .

24:48

Does this look at images that's , for example

24:50

, older than three years or three months

24:52

, or something ?

24:53

It'll look at everything on your GHCR

24:55

repo , everything . So essentially what it'll

24:57

do . If we look at this little flow chart , it sets

24:59

up , it gets your images from GHCR , it outputs

25:01

a matrix and then it loops through the matrix

25:04

and patches . So let's just look

25:06

at the workflow real quick . So here's the setup

25:08

right here . Make it

25:10

a little bigger . I

25:14

wrote a little go CLI . I turned it into a CLI instead of just a go file , which we can see in another

25:16

branch . There's actually quite a bit of logic in there to detect

25:18

the tags . So the advanced logic

25:21

that I put in there would basically calculating

25:24

the tagging strategy . So I'm only going

25:26

to grab the latest . If , like , net monitor

25:28

has a hyphen one , I'm only

25:30

going to patch hyphen one and not zero , because

25:32

zero came before hyphen one , and

25:38

then if I patch both , they'd be at equal patch level . You should , you know . Then why not just flatten

25:40

it down to one tag ? So there's a little bit of strategy there . And then I'm calculating

25:42

the incremental on that and that

25:44

looks like let me open a new tab . The

25:47

output for that CLI improved

25:49

looks like this . I called it contiguous . I

25:51

thought it was clever . Maybe it does . I don't have a screenshot

25:53

maybe , but essentially you would do this list

25:55

and then it would output a

25:58

JSON of here's your current tag

26:00

and here's your next tag . And then GitHub

26:02

Actions is pretty cool because you can just

26:04

use from JSON to generate your

26:06

matrix , yeah , and then basically

26:08

go through . So for each image in the matrix

26:11

that I identify that needs a patch , I'm going

26:13

to go through and generate a trivia report . I'm

26:15

going to check the vulnerability count . So if the vulnerability

26:17

counts zero , I'm not going to bother patching , because

26:19

Copasetic is just going to fail and say

26:21

I couldn't patch anything . This patch tag is taken

26:23

care of and then the new version is CLI . But this is

26:25

just some bash Again . This is why I've had

26:28

to learn more . Bash is because of GitHub

26:30

Actions , to be honest . This calculates the next

26:32

patch tag based on a tag , then

26:34

it goes through and runs Copasetic logs

26:36

into GHCR and then pushes the patch tag

26:38

. So the next part of this workflow would be like okay , I've

26:40

got all these new patched images on my workflow

26:43

. How do I automate that final step , which would

26:45

be the PRs , into the code repos

26:47

that are using those images ?

26:54

And that's where Dependable will come in . So that's next on my list to tackle . Okay , yeah , it's quite

26:56

good that you can automate this and

26:59

probably can make it a self-service for

27:01

anyone or developers to run .

27:03

Just add the images , yep , and

27:05

so here's an example , here's the setup , and

27:08

I only had one

27:10

image , but two tags that needed to be patched

27:12

, right , so let me

27:15

shrink it a little bit so you can see , but

27:17

then , yeah , it just goes through . This one didn't

27:19

even need to be patched , so it skipped it .

27:23

So this is probably good . You can probably build in

27:25

this with some alerts as well . When someone's

27:27

image has been outdated for like

27:29

one month , you can alert you so someone

27:31

can patch it .

27:32

Yeah , you could this continuous patching

27:34

thing . I have it on an on-demand trigger , but you

27:37

could a cron job , like whatever you wanted

27:39

to do , just kind of show you like what's possible with

27:41

it . Like I said , cubescapes doing something with

27:43

it , integrating with copacetic to make

27:45

this more full-fledged . This was just to give an idea

27:47

of , like what you could build with it and a lot

27:49

of people are doing with that . I I think , yeah

27:52

, in ACR they're working on the continuous patching

27:54

thing , solving a lot of these problems as you dig into it

27:57

. There's issues , like I mentioned , with the signing , like how

27:59

do we make sure that the signatures are still valid

28:01

? How do we update ? It's funny just

28:03

last year I contributed

28:05

the logic for Ratify to support

28:07

inversions of keys . So basically it'll

28:09

look at your , your keyball instance and

28:12

if you have more versions that you've created for

28:14

a key , it'll update , auto update the cache

28:17

, which solves the next problem that the continuous

28:19

patching does like . Okay , what if I signed it with

28:21

a new key ? Do I have to go then update

28:23

my verification engine ? The answer is no

28:25

, it'll . It'll dynamically pull um

28:27

. So that was kind of cool to see that that work

28:30

get used , okay so , yeah

28:32

, it looks like it's quite a good tool

28:34

to quite batches .

28:36

Is there any ? So , to get this set up

28:38

and running , is there any challenges that

28:40

you have to do ? I take it one of the challenges

28:42

you have to know more batches

28:44

right or just use your task well

28:47

.

28:47

So the way that I've set it up , if you I'm tomorrow's

28:50

on my to-do list , tomorrow is to take the

28:52

. So if you want to use ghcr , that's what I scope . Mine to you tomorrow is to take the . So if

28:54

you want to use GHCR that's what I scoped mine to you could actually just take

28:56

my actions and plug them in and it would

28:58

work . The only thing that you'd have to figure

29:01

out if you're not using GHCR is

29:03

figuring out what tags you want in your matrix

29:05

, and you could manually define that . So if we

29:07

go back and look at this one , I had an example . I

29:10

had an example of using just a static matrix

29:12

. So if you knew the

29:15

tag names that you wanted , you could just define your matrix

29:17

like this , right ? But I wanted

29:19

to be a little bit more pragmatic , and

29:21

so then that required me to go in write

29:23

my own logic against GHCR . So

29:27

that would be . The biggest gap right now is you need some mechanism to feed into

29:29

the copacetic workflow what images

29:31

you care about . So if you want , you can use

29:33

my Go code . That's fine .

29:35

Yeah , it's open source anyway , so it's good . So

29:38

do you reckon that the tool that

29:40

you show us today is the future

29:42

of container security , like containerized

29:44

?

29:44

I would imagine it's kind of the foundation . So

29:46

I think they're like what you were talking about earlier . These

29:48

will be wrapped into other tools

29:50

that probably cost a little bit of money . That just

29:53

make it easier , right . That's kind of the trade-off with open

29:55

source . But if you want to build your own type of solution , all

29:57

the tools are there to improve the supply

29:59

chain and again in the CNCF , so

30:01

like they're stable and you know that they've got good investment

30:04

in them , they're being used internally for

30:06

Microsoft . They're being used internally at other big

30:09

competitors too . One interesting thing AWS

30:12

and Microsoft are both working on Notation

30:14

, so they're collaborating . We're

30:16

collaborating on that one . But yeah

30:18

, so the tools are there to put it in there if

30:20

you want to go the open source route . But I'd

30:23

imagine there's going to be some cool tools that kind of wrap

30:25

all this and make cool button

30:28

Right .

30:29

Make everything a button . The only good

30:31

side about it is that this is free . It's open source

30:33

. Instead of other tools that you have

30:35

to pay it , you know to

30:37

use this stuff enterprise tool . That's more

30:39

of a better

30:49

suited . So that's about okay . So , as this episode is coming

30:51

to end , we always like to get to know our guests . What do you normally do

30:53

in ? Your spare time , josh , like your job

30:55

as an advocate on your job

30:58

and then learning about security

31:01

in containers .

31:02

Yeah , well , the last few years . So I just

31:04

my wife and I welcomed our third child

31:06

a year ago . He just turned one last month

31:08

. So the kids keep me pretty busy

31:11

, so my spare time isn't as fruitful as it

31:13

used to be , but I like to read , so I do a

31:15

fair amount of reading . Lately

31:19

it's been I've actually been revisiting some old books PowerShell

31:21

books , no , actually more like productivity-type books Deep

31:23

Work by Cal Newport , I think it was released in 2016

31:27

, 2018 .

31:30

And then UltraLearn . I just got it right here on my desk I can show you . Okay , like

31:32

you read 50 a book , but not kindle .

31:33

I have a kindle um , I use the kindle

31:35

to evaluate whether a book's

31:37

useful or not . So like I'll download

31:39

the sample , I'll burn through the sample on my kindle

31:42

, uh . But I do prefer a physical book

31:44

, um , even technical books , which are more annoying

31:46

to do because you have to . But

31:49

anyway , yeah , I like the physical book

31:51

. So I I don't know , I'm , I'm all kind

31:53

of over the place . I read a little bit of Rust

31:55

, worked with a colleague on some Rust recently

31:57

.

31:59

That's not the language right Rust .

32:01

No , yeah , the language Rust , yeah .

32:03

Oh , okay , yeah , it's not very

32:05

popular . So I don't think if you

32:08

use Copilot to tell me about Rust

32:10

, I don't think Copilot would know what's Rust

32:12

it but Rust . I don't think a lot of people would know

32:14

what's Rust . It's never helpful . Yeah , yeah , Rust

32:16

is quite hard . It's like Ruby . It's quite an outdated

32:19

language as well .

32:20

So it's probably more people know Ruby than Rust . To be honest , I think I would have had an easier

32:22

time picking up Rust , moving from PowerShell

32:25

as my primary language than

32:28

Go was . Just because of all

32:30

, it is still static , but

32:32

it has a lot of niceties to language Once

32:34

you understand . They're very cryptic . At first they just look

32:36

like hieroglyphics , but they're actually very useful

32:38

shorthand macros that

32:41

Go doesn't have . That makes

32:43

it made the transition a little hard for me , but I

32:45

do really enjoy Go still . It's made me learn

32:47

things at a much deeper level , that's for sure .

32:49

So I've remembered speaking

32:52

to you before this goes live

32:54

and you said you're going to the bill

32:57

in May .

32:58

Potentially . I have a CFP submitted

33:00

, so I'll have word back . I

33:02

don't know , maybe in the next month or so , so

33:05

I might be there .

33:06

Yeah , so it's going to be the same session as

33:08

this talk .

33:08

Yeah , yeah , it's this one right here , yep .

33:10

Okay , nice , so yeah , so

33:16

if anyone wants to interesting to meet josh , he might be in bill so you can catch him there

33:19

. Absolutely okay , uh , thanks for joining josh . So

33:21

yeah , as normal . Stay tuned

33:23

and see which is going to be

33:25

on youtube and spotify .

33:27

Thank you , bye thanks for having me .