0:00

You just realized your business

0:02

needed to hire someone yesterday. How

0:04

can you find amazing candidates fast?

0:06

Easy. Just use Indeed. Stop struggling to

0:09

get your job post seen on

0:11

other job sites. With Indeed's sponsored

0:13

jobs, your post jumps to the top

0:15

of the page for your relevant

0:17

candidates, so you can reach the

0:19

people you want faster. According

0:21

to Indeed data, sponsored jobs

0:23

posted directly on Indeed

0:26

have 45 % more applications

0:28

than non -sponsored jobs. Don't wait

0:30

any longer. Speed up your hiring right

0:32

now with Indeed. And listeners

0:34

of this show will get

0:36

a $75 sponsored job

0:38

credit to get your jobs

0:40

more visibility at Indeed .com/P

0:42

-O -D -K -A -T -Z -13. Just

0:45

go to Indeed .com/P -O -D -K -A

0:47

-T -Z -13 right now. And

0:49

support our show by saying

0:51

you heard about Indeed on

0:53

this podcast. Terms and conditions

0:55

apply. Hiring Indeed is all

0:57

you need. Hey,

1:03

folks, welcome back to another episode

1:05

of the Ruby Rogues podcast. This

1:07

week on our panel, we have

1:09

Ayush Noatia. Hello, hello. I'm

1:12

Charles Max Wood from Tophead Devs. This

1:15

week, we have a special guest.

1:17

We have David Henner. David, you

1:19

gave a talk at Rails World

1:21

in September talking about performance

1:24

and And so

1:26

we reached out because we thought, you

1:28

know, hey, it's interesting to talk

1:30

about. You're on the Zendesk Ruby scaling

1:32

team. So it sounds like you're

1:34

working on, I guess, a

1:36

non -trivial application. I don't know if

1:39

it's just the traffic, but it seems

1:41

like it's an involved app. So

1:43

yeah, do you want to just give,

1:45

I guess, any more introduction that

1:47

you want us to have and then

1:49

we can dive in and talk

1:51

about how to do performance with Ruby?

1:54

Yeah, I gave that

1:56

talk and the ironic part of that. I

1:58

gave that same while I gave. They've almost

2:00

the same talk within Zendesk and it

2:02

was originally an hour and a

2:04

half and then at Zendesk I had

2:07

to break it down to 45

2:09

minutes and then I thought I was

2:11

done because I thought it was

2:13

a 45 minute talk for Rails world.

2:15

I found out was a 30

2:17

minute talk. So I've got a lot

2:19

more content but it was about

2:22

30 minutes and that's normal probably but

2:24

wow 30 minutes it's not much

2:26

time. It's so funny to me because

2:28

I talk to people who haven't

2:30

spoken at a conference or haven't you

2:32

know given a talk in other

2:34

settings and you know so for example

2:37

at church a lot of times

2:39

you know they'll have us give like

2:41

a 15 minute talk on a

2:43

topic but it's like man 15 minute

2:45

talks are hard and they're like

2:47

what you only have to come up

2:49

with this much content and I'm

2:52

like I always come up with an

2:54

hours worth of content and have

2:56

to cut it down. There's

2:58

always more I could put in

3:01

it and so it's knowing what

3:03

to take out more than knowing

3:05

what to put in and then

3:07

turning that into a I guess

3:09

a cohesive narrative and yeah anyway.

3:11

Having flow is the hardest part.

3:13

It's like oh my god I

3:15

need to have this and the

3:18

original flow is so perfect that

3:20

I have actually tweaked down and

3:22

at my company then that's like

3:24

yeah I got this and it

3:26

wasn't until like two months before

3:28

the conference talk when I had

3:30

to submit my papers oh 30

3:33

minutes oh oh no let's get

3:35

that now. I

3:37

think it turned out fairly well though

3:39

at the end of the day. I was

3:41

the last talk on the last day

3:43

right before Aaron Patterson talked though and I

3:45

will say that is the worst time

3:47

slot to have like you're worried about your

3:49

talk the whole time and first talk

3:51

that I actually paid attention to I was

3:53

still kind of a little amped up

3:55

and that was Aaron's say goodbye speech basically

3:58

it's like right. Yeah.

4:02

So

4:05

yeah, I've been there too.

4:08

I want to just dive in.

4:11

So, you know, what's kind of

4:13

the low -hanging fruit in the Rails

4:15

performance world? I think the big

4:17

thing that people like, at

4:19

least I don't hear people talking about and

4:21

it's like, everybody focuses on, first off, a

4:23

lot of people focus on really hard things.

4:26

And it's like, hey, let's change it to

4:28

a microservice. It's like, oh, yeah, that might

4:30

make it faster, but probably is just going

4:32

to hold a bunch of crud. But the

4:34

big thing people don't advertise

4:37

in the community is just

4:39

observability and like trying to

4:41

actually like, find the things

4:43

you need through, let's say,

4:46

I use Datadog as end

4:48

desk, but everybody can also

4:50

complains about the cost of the

4:52

observability. And it's like, you can

4:54

actually make your costs lower by

4:56

turning on software switches. You don't

4:58

need the observability 100 % of

5:00

the time, for example. And you

5:03

can actually have like tricks around

5:05

like the observability layer just literally

5:07

going out and like, okay, if

5:09

we monitor something and something goes

5:11

awry, turn on the observability layer,

5:13

but don't turn it on all

5:15

the time. Because if you do,

5:17

you're going to be paying half

5:19

of your infrastructure costs to Datadog. So

5:22

I started the talk with that.

5:24

And that was, I hope a

5:26

lot of people got valuable as

5:28

in dollar

5:31

figures, stuff out of

5:33

that. But then like, hang

5:36

on, because I have a question about that. Because,

5:39

you know, I've

5:41

used Sentry, I've used

5:44

Rollbar, I've used

5:46

New Relic, I'm trying

5:48

to think of all the different ones

5:50

that I've used over the years. I mean, most

5:52

of them, they sponsored the show. And so I

5:54

feel like I have to, I

5:56

have to feel good about it being a good

5:58

product before I go, okay. you can sponsor the

6:01

show and I can go out and say,

6:03

you should try this one. And so I've

6:05

tried a whole bunch of them, but

6:07

most of them you just install the gem and

6:09

then it just kind of, you

6:11

know, behind the scenes does the work.

6:14

And so my question there is, how

6:17

do you put a switcher on that? Do you

6:19

just put it into your initializer and then... So,

6:22

Lloyd, or how do you handle it? In

6:24

the talk I gave the actual

6:26

code, but there is a general,

6:28

there is a general like, hey,

6:30

it's on or off. And we

6:33

always add general thing on. So

6:35

there's like the general, but then

6:37

let's say in the traces there,

6:39

you get a stack trace. And

6:41

the default stack trace I believe is just one

6:43

bar, nothing extraordinary, but

6:45

you can actually turn a whole

6:47

bunch of traces on within that trace.

6:50

Oh, okay. Break down to see

6:52

everything that's going on. And then you

6:54

can actually identify, oh, this part

6:56

of the code right here is slow.

6:58

So by turning that on and

7:00

off, like all of it on or

7:02

all of it off, if you

7:04

try to have it all on, all

7:06

it off, I don't know what

7:08

our costs at Datadog would be, but

7:11

I'd be willing to let Datadog

7:13

would even say, hey, stop, you're

7:16

overloading us, right? But

7:20

no, that's the big thing

7:23

is like turning, not the

7:25

base layer on, it's all

7:27

the other little things, they're

7:29

optional trace. And then

7:31

one of those optional traces are where

7:33

the valuable data is though. Like it's

7:35

the stuff that you can figure within

7:37

your own application that gives you way

7:40

more insights. Like, hey, I want to

7:42

put a trace around this certain

7:44

method. Cause I think it's slow. And then

7:46

sometimes you like do that and you're like,

7:48

oh, that's fast. Where's the

7:50

slow part? But if you put

7:52

that around enough things, then

7:54

we basically are main application, at

7:56

least the ticketing side of

7:59

the application. We've we got the

8:01

observability layer like just

8:04

it's covering almost everything. So

8:06

if a certain customer starts

8:08

to have slow response times,

8:11

we just turn it on really. Oh,

8:13

we can focus on this part

8:15

of the code and make customer

8:17

faster. I got you. That makes

8:19

sense. But let's see, what what

8:21

am I doing now? Right now, I'm

8:23

doing something that's not in my talk,

8:26

it wasn't my talk, but is also.

8:28

Interesting, I think if you're a

8:30

big company or if you're a

8:32

small company, you might go, well, why

8:34

is that hard? And that would be,

8:36

we're changing the default scope for our

8:38

main ticketing object. And the reason why

8:41

we're doing that, we're adding new columns

8:43

to the, and it just seems like,

8:45

okay, you just change the queries. Well,

8:47

you change every single query. You also

8:49

have to change every index from out

8:51

from under the hood. And you have

8:53

to actually never have downtime.

8:55

And adding index alone. on

8:58

our ticketing tables, they're just

9:00

so huge. You can't just say,

9:02

hey, uh, DBA's, add this, they'll

9:04

say no to you. So... You can lock

9:06

that table for, I don't know,

9:08

a few days. Exactly. You can't do

9:11

that. So it is a juggling act

9:13

that, luckily, I don't have to deal

9:15

with the DBA side of it. I

9:18

don't know what the problems are,

9:20

and essentially... When we move from

9:22

Aurora 2 to Aurora 3, it

9:24

did two things. First off, it

9:26

made some problems bigger, so it

9:29

made this a direction we needed

9:31

to go. But also, have you

9:33

heard of invisible indices? No. So

9:35

invisible indices is

9:38

kind of what sounds like. Essentially,

9:40

you have an index, and a

9:42

lot of times you're too scared

9:45

to delete any index. Because

9:47

if you delete an index. it's

9:49

gone and maybe you actually affected

9:51

your application. So invisible basically says

9:53

make it invisible. It's still there.

9:55

The sequel is still populating that index

9:58

to make it actually work. But

10:00

you can see are there performance

10:02

problems with it? So we have

10:05

not deleted an index for 15

10:07

years of our application. And now

10:09

we're actually able to go in

10:11

and say, hey, if we actually

10:14

delete these indices, we can add

10:16

new ones because the memory profile

10:18

won't blow up. So that is

10:20

the saving grace of allowing us

10:23

to actually do this whole project.

10:25

Without that, we'd be too scared

10:27

because, well. Having

10:30

things break is scary. A

10:32

lot of money. So I'm

10:34

curious. I mean, I've deleted

10:36

indices off of databases that

10:38

I worked on, but I

10:40

haven't done it on like

10:42

very large tables. I've added

10:45

indices on very large tables

10:47

and watched the thing go.

10:49

Ah! But deleting them doesn't

10:51

seem to have that same

10:53

effect. So is there a

10:55

downside to deleting them other

10:58

than just some queries that

11:00

use those indices might slow

11:02

down? Right. And you don't

11:04

know what like, like when

11:06

your application is, I think

11:08

we have over, I'm guessing

11:10

have over a thousand repos,

11:13

for example, you can't figure

11:15

out whether you have. beat

11:17

down every location or every

11:19

query that actually hits this.

11:21

There's actually a, I had

11:23

to go through every query

11:25

in our main API and

11:28

we separate our queries by

11:30

API queries and we separate

11:32

it by. You just realized

11:34

your business needed to hire

11:36

someone yesterday. How can you

11:38

find amazing candidates fast? Easy,

11:40

just use indeed. Stop struggling

11:42

to get your job post

11:44

seen on other job sites.

11:46

With Indeed sponsored jobs, your

11:48

post jumps to the top

11:50

of the page for your

11:52

relevant candidates. So you can

11:54

reach the people you want

11:56

faster. According to Indeed data,

11:58

sponsored jobs posted directly on

12:00

Indeed have... 45% more applications

12:02

than non-sponsored jobs. Don't wait

12:05

any longer. Speed up your

12:07

hiring right now with Indeed.

12:09

And listeners of this show

12:11

will get a $75 sponsored

12:13

job credit to get your

12:16

jobs more visibility at indeed.com

12:18

slash P-O-D-A-T-T-S-T-S-E-T-T-T-T-S-T-S-T-S-T-T-S-T-E-T-T-T-E-T-T-T-T-S-T-S-T-E-T-T-S-T-S-S-S-A-S-A-A-S-A-S-A-A-S-A-S-A-A-S-A-S-S-A-S-S-A-A-A-A-S-A-A-S-S-S-S-S-A-S-A-S-A-A-S-A-S-A-S

12:29

Hiring, indeed, is all you

12:31

need. Non-API queries. Let's just

12:34

that way. And the API

12:36

queries is a manageable list

12:38

of things. It's like a

12:40

couple hundred queries to the

12:42

ticketing table that actually happens.

12:44

You can wrap your head

12:46

around that. But the non-TAP

12:48

queries, I think it was

12:51

over 40,000 different unique queries.

12:53

Oh, wow. Go through each

12:55

query and say, would this

12:57

index work? So it is

12:59

complex. So what's your approach

13:01

to that then? So that

13:03

luckily, the things we care

13:05

more about are the API

13:08

queries. So we absolutely make

13:10

sure that those actually are

13:12

working. And the non-API queries,

13:14

we're actually moving a

13:16

lot of that stuff to

13:19

elastic search anyways. So our

13:21

long-term fix is actually all

13:23

those things. We're never actually

13:25

going to need a need a

13:27

query. And I don't know how

13:29

the magic of that works. Like,

13:31

oh wow, you don't even need

13:34

a query to get the data?

13:36

That's awesome. Okay, I'm guessing somebody's

13:38

looking at bin logs or something

13:40

like that. And they're throwing that

13:42

information right from the

13:44

bin logs into sequel. But I

13:47

have, that's magic to me. So. I

13:49

used, do you have anything you want

13:51

to jump in with? I haven't seen

13:53

your talk so I'm probably gonna

13:55

come at you with probably

13:58

some stupid questions really. What

14:00

kind of, like, where do you

14:02

normally find performance bottlenecks in like

14:04

apps you see? Because I know

14:07

you're primarily, you're working for Zenters,

14:09

you've probably worked on that product

14:11

for quite a while, but I'm

14:13

a freelance, so I look at

14:16

a lot of code bases and

14:18

what I see, and it's all

14:20

rails. What I see most of

14:23

the time is people just haven't

14:25

taught through their database design properly.

14:27

and the doing all these funky

14:29

joins or even worse getting stuff

14:32

out of the database and doing

14:34

things in memory that you could

14:36

quite easily do in the database

14:39

and it's like it's quite common

14:41

mistakes like that is where I

14:43

usually find performance bottlenecks. What's your

14:45

experience in that regard? So the

14:48

worst areas are where they add

14:50

funky joins because sometimes you can't

14:52

solve those. very easily. The talk

14:54

was actually more focused on the

14:57

coding aspect, like just simple things

14:59

that people do that they should

15:01

know about that they just don't

15:04

do. So, for example, just memorization

15:06

and little different memorization techniques. And

15:08

one of the cool things I

15:10

brought up at Zendes that we

15:13

do is we have what I,

15:15

it's kind of like a singleton,

15:17

but it's not, so our account

15:19

object. If you get an account,

15:22

it doesn't matter how you get

15:24

that account. In the request cycle,

15:26

you only need one account object.

15:29

So if you call it tickets.account,

15:31

or if you call user.account, or

15:33

if you call just get me

15:35

the account through this method called

15:38

fetch account that we have, any

15:40

way you get the account, it

15:42

is the same exact object. So

15:44

one of the cool things you

15:47

can do based on that. So

15:49

if you get the same object,

15:51

if you get the same object,

15:54

if you memorize your whatever you're

15:56

trying to memorize. one object. You

15:58

can instead of actually have

16:00

have a request where,

16:02

hey, listen, I'm going to

16:04

memorize it on this object, but

16:06

then someplace else in the

16:09

application, you want to use that,

16:11

but you can't get to

16:13

that original object. We can actually

16:15

memorize it on this God

16:17

object. And wherever we call that

16:19

method again, it is memorized.

16:21

So that was, I thought, a

16:23

really cool way of actually

16:25

going about, like, memorizing something, having

16:27

that God object, and then

16:29

you don't have to worry about,

16:31

okay, the presenter we're calling it this

16:33

way, and the controller we're calling it this

16:35

way, and the models we're calling it

16:37

this way, but we cannot tie those spaghetti

16:39

pieces of code together. But because we

16:41

have that God object, we can memorize it

16:43

in one place and always access the

16:45

same data throughout, and then never

16:47

have to worry about. That was a

16:49

cool thing that actually, well, that was

16:51

in the talk, but really, What are you

16:53

using for that, are you using current

16:56

attributes or something else? As far as making

16:58

it a God object, yeah, I wish I

17:00

knew that was well before I came

17:02

into it. But I think we're basically,

17:04

we created a method called fetch account.

17:07

And whenever we call

17:09

a fetch account outside

17:11

there, and then

17:13

in probably active

17:15

record, we override

17:17

the account method in active

17:19

record, not even every one

17:21

of our models has an

17:23

account method. So I'm guessing

17:25

we override that and just

17:27

memorize the account in an instant variable,

17:29

and it always goes back to that

17:31

incident. That's just a guess. I've not

17:33

actually looked at that code because I

17:35

haven't needed to. And I was actually

17:37

asked that at Rails, but I should actually

17:39

really get into it. But the person who

17:41

knows that best just left that desk.

17:44

So I'll have to dive into code

17:46

instead of ask questions. Yeah,

17:49

I thought the current attribute is probably a bit

17:51

too new. So when it came in, I got

17:53

a Rails 5 or a Rails 6 or something

17:55

like that. You just realized

17:57

your business needed to hire

17:59

someone? Yeah. How can you find Amazing Candidates

18:01

Fast? Easy. Just use Indeed. Stop

18:03

struggling to get your job post

18:06

seen on other job sites. With

18:08

Indeed sponsored jobs, your post jumps

18:10

to the top of the page

18:12

for your relevant candidates. So you

18:14

can reach the people you want

18:17

faster. According to Indeed Data,

18:19

Sponsored Jobs posted directly on

18:21

Indeed have 45% more applications

18:24

than non-sponsored jobs. Don't wait

18:26

any longer. Speed up your hiring

18:28

right now with Indeed. And

18:30

listeners of this show

18:32

will get a $75

18:34

sponsored job credit to

18:37

get your jobs more

18:39

visibility at indeed.com/P-O-D-K-A-T-Z-13. Just

18:41

go to indeed.com/P-O-D-K-A-T-Z-13 right now

18:43

and support our show by

18:46

saying you heard about indeed

18:48

on this podcast. Terms and

18:51

conditions apply. Hiring, indeed,

18:53

is all you need. Did

18:55

we just upgrade it to, did

18:57

we hit rails eight yet? Rail

19:00

seven. Either way, like it's

19:02

so good to be closer to

19:04

the edge, or definitely on

19:06

rail seven for main application.

19:09

We're on rails for, for

19:11

like six years or something

19:13

like that. It was so

19:15

good to be closer to

19:17

the edge. Yeah, I've

19:20

experienced that too for a long

19:22

time. I was working on like

19:24

rails. four or five apps and

19:26

yeah the last year or so I've

19:28

been able to work on rail seven

19:30

and rails eight apps now and what's

19:33

fun is you get to go back

19:35

now and figure out all the stuff

19:37

that's been added in since so you

19:39

know I got into like stimulus

19:42

and turbo and then figured out

19:44

that oh look turbo actually will

19:46

work through your what's it action

19:48

cable channels and so you can

19:51

you can push updates when you have

19:53

sort of out a band Anyway, it's

19:55

just all this stuff that it's like,

19:57

okay, I don't now need a full

19:59

cycle to... get these features to

20:01

work. So being at a bigger

20:03

company, we started on rails one

20:06

or something like that. Upgrading through,

20:08

there's obviously monkey patches like throughout

20:11

the years because well, at the

20:13

time, rails didn't have certain things

20:15

and now they do. And now

20:18

taking those monkey patches away is

20:20

hard, which also means that really

20:22

adding the newest features of rails

20:25

is hard. In fact, almost impossible.

20:27

But what we can do is,

20:30

like, for example, is, like, yeah,

20:32

this week, we upgraded to Rail's

20:34

3-3, and all of a sudden,

20:37

the performance got a lot better.

20:39

But also, like, let's say, in

20:42

Ruby. Yeah. Oh, Ruby 3-3, sorry,

20:44

Rail's 3-3, sorry, Rail's 3-3, yeah.

20:46

I do remember upgrading to Rail's

20:49

3 from 2, that was hard.

20:51

That was brutal. That was brutal.

20:53

I don't think I'm going to

20:56

remember what I was saying. Oh,

20:58

you did a ton of work

21:01

on that. Yeah, it was. It

21:03

was brutal. But where was I

21:05

going with this? Oh, I forgot

21:08

my train of thought. You upgraded

21:10

to Ruby 3. Upgraded Ruby 3.

21:13

I don't think I'm going to

21:15

remember what I was thinking about.

21:17

Oh, and like for example, with

21:20

Ruby 3, we had in our

21:22

application, we were very scared because

21:24

we allowed, we had and still

21:27

do allow customers to add their

21:29

own reg excess. Well, if you

21:32

had your own rejects, you can

21:34

really slow down and stop your,

21:36

you can just kill an application

21:39

if you allow the end users.

21:41

So we had some weird way

21:44

of actually making it so they

21:46

couldn't just kill our system. But

21:48

it spun up a process and

21:51

that spinning up a process took

21:53

about a half second to a

21:55

second, let's just say to actually

21:58

spin up. So it would run.

22:00

and it would make sure that

22:03

RegX would never actually just be

22:05

infinitely running. It would time it

22:07

out at a certain point. But

22:10

doing that was, well, added a

22:12

half second every request, for

22:15

example. Ruby 3 allows you

22:17

to actually, by default, have

22:19

a timeout in your RegX's.

22:21

So now we'll get that

22:23

extra process up. So now

22:25

if you're, most cases. You're

22:28

just running a redjects, it

22:30

takes a millisecond or two. Those

22:32

other cases that actually do time out,

22:34

those do time out, just by default,

22:36

with Ruby, we didn't have to

22:39

actually do this weird fancy trick

22:41

to actually make it happen. So

22:43

that was actually a very, very

22:45

big upgrade for us, because we've

22:47

had those redjects that allowed

22:50

people to put redjects as in

22:52

forever, and we've had to protect

22:54

against it forever, and that's just...

22:56

I think I would have, if

22:59

I was around way back

23:01

then, I'd probably have

23:03

said, no, don't let

23:06

them put their own

23:08

redexes and give them

23:10

an array of things that

23:12

they can do, don't give

23:14

them an infinite array.

23:17

Right. Yeah. But yeah, 90%

23:19

of our upgrades, it's

23:21

just looking at code

23:23

that's old and people

23:26

don't. When an application is

23:28

as big as it is, people

23:30

cannot wrap their head around

23:32

the whole ticket save request,

23:34

the cycle, because we have,

23:36

everything gets into the ticket.

23:38

There's triggers, there's not just,

23:40

every layer, and essentially people

23:42

can't wrap their head around

23:45

that. So once you put

23:47

traces around this, you find

23:49

blatant, like, obvious mistakes, and

23:51

you just tackle them one

23:53

at the time. And most of

23:55

time, it's. Cody mistakes. It's not,

23:57

and it's not even a mistake.

24:00

it was, I gotta get this feature out

24:02

fast. People don't think about performance when

24:04

you have to get it out within a

24:06

week. Right. But

24:09

it's been good actually, like

24:11

the amount that we've saved in

24:13

the ticket cycle and just

24:15

not even take it. Now we're

24:17

actually, since we did such

24:19

a good job, we've been given

24:21

more and more freedom to

24:23

actually work in random areas of

24:25

the code and just the

24:27

amount of improvements that we've made.

24:30

Just you don't think of

24:32

little coding things like, for example,

24:34

scopes versus conditional associations. Like

24:37

a scope, every time you

24:39

call a scope, Rails makes the

24:41

query. But if you have

24:43

a conditional association, it makes a

24:46

query in Rails by default,

24:48

has that query cached up. Just

24:51

making that switch will reduce

24:53

the number of queries and

24:55

sometimes those queries aren't very

24:57

free. So all of a

24:59

sudden you've just saved time

25:01

there. So many n plus

25:04

one queries were that. So

25:06

were you advocating for scopes

25:08

or for? Both.

25:10

Conditional. Most of the

25:12

time conditional queries, when

25:15

you're trying to tweak

25:17

the last bit of performance,

25:19

I would be advocating

25:21

for conditional associations. Scopes

25:24

are not evil. It's just,

25:26

sometimes people don't realize, I'm

25:28

using this in the scope

25:30

within a loop, for example.

25:35

And all a sudden I'm getting like, just

25:37

query after query after query, where if

25:39

you don't put it in a loop and

25:41

if you use a conditional association, that

25:43

loop makes one query and you're done. So

25:46

it automatically eliminates the n plus one

25:48

query is what you're saying. Yeah. Well,

25:52

yes. Yes, but n plus

25:54

one queries, I always think of

25:57

n plus one queries as

25:59

the things that you can add

26:01

just includes in. this goes away and yeah

26:03

generally yeah but this is an n plus one query that

26:05

generates itself a different

26:07

way yeah if it was to

26:10

do like some if I just

26:12

built like a new feature and

26:14

I just want to reassure myself

26:16

that I haven't made any stupid

26:19

performance mistakes or anything like that

26:21

again like the work I do it

26:23

tends to perform like performance doesn't

26:25

really come into it because I

26:28

tend to work with startups and

26:30

stuff. Like what, how do I just go

26:32

about like even testing like certain things

26:35

in my local dev environment in terms

26:37

of like what do I put in

26:39

my Rubica to check how long service

26:42

taking I don't even know that that's

26:44

how stupid this question is. It's going

26:46

to go back to the basics. You know,

26:48

we have this debate all the time

26:50

with internally. Because essentially the only way

26:53

to find out is to have our

26:55

huge customers, I won't name the customers,

26:57

but we have a few huge well

26:59

customers and you put it on those

27:02

and you're going to find the problem

27:04

very quickly. But then there's also the

27:06

aspect of a new feature, even when

27:09

you deploy it, looks great because nobody's

27:11

using it. Right. Then two years

27:13

later, somebody uses it in a

27:15

way that you don't anticipate. Reactive development

27:17

isn't terrible, I would say. At

27:19

the end of the day, your

27:21

objective when you first get out

27:24

is if you write the code

27:26

clean, you can clean it up

27:28

later. If you don't write the code

27:30

clean, cleaning it up later

27:32

is almost impossible. So I

27:34

would say for feature developers,

27:36

focus on keeping it clean,

27:38

because once it's clean and

27:40

it's very easy to read,

27:43

somebody else can maintain it

27:45

later. Then I would say that

27:47

that's the case for... like any

27:49

small startup. Just folks on

27:51

cleaning a lot of different

27:53

just review code and actually

27:55

have an old job I

27:58

had. There's a guy. he

28:00

hated me. He loved me, but

28:02

I was just, he hated me

28:04

because I would always review code

28:07

and I would be like a

28:09

stickler on making it really easy

28:11

to read and maintainable and stuff

28:13

like that. He was hated me.

28:16

And then he eventually left that

28:18

company and I left that company

28:20

and I have talking to old

28:22

people and I've heard him say,

28:25

I miss Dave. Oh my god,

28:27

I hated when he reviewed our

28:29

code, but now I'm at a

28:31

company that doesn't review code and

28:34

the crap that comes in here,

28:36

we can't maintain it. Yeah. Maintainability

28:38

is 90% of getting your speed

28:40

out of it, because if it's

28:43

easy to maintain, it's easy. You

28:45

don't want to add the best

28:47

trick for speed day one anyways.

28:50

Because if the code is never

28:52

used. Who cares if it runs

28:54

a few milliseconds slower? Who cares?

28:56

You care about the features that

28:59

all of a sudden get traction

29:01

to make performance improvements. And then

29:03

somebody, a team like mine, is

29:05

probably always required at big companies

29:08

because eventually you get big enough

29:10

where all of a sudden, hey,

29:12

10 years ago, people weren't paying

29:14

attention to performance. And now needs

29:17

to, then, well. Yeah. Well, to

29:19

your point on this, so we're

29:21

actually running into performance issues on

29:23

this application that I'm working on

29:26

now for client. And what it

29:28

does is it does this gigantic

29:30

calculation across a whole bunch of

29:33

transactions on a hedge fund. And

29:35

the people that wrote it, I

29:37

mean, it's impossible to read. And

29:39

so, you know, you're trying to

29:42

just parse through it and go,

29:44

okay, what, what is it doing?

29:46

And it's super hard, right? But

29:48

the thing is, is that it's

29:51

in a sidekick job. So it

29:53

gets kicked off and, you know,

29:55

it runs and, you know, if

29:57

it's a large. It can run

30:00

45 minutes an hour, two hours.

30:02

And, you know, I'm looking at it and

30:04

what we're, you know, we had the conversation

30:06

where we're saying, well, it'd be

30:08

really nice to break this up,

30:10

right, so that you can parallelize

30:12

some of the work because you

30:14

have multiple workers and so, you

30:16

know, that would make it faster. And

30:19

just getting to the point where

30:21

we can actually figure out where to

30:23

break it apart has been just, I

30:25

mean, it's been ugly. And, you know,

30:28

yeah, if it had been written in just

30:30

a maintainable way, like you're

30:32

talking about, where it's like, okay, I

30:34

see where this kind of breaks down

30:37

and, you know, where it kind of

30:39

goes into these methods and comes back

30:41

out, it'd be way easier to

30:43

do because it's like, okay, we

30:45

can turn these things into these

30:47

things over here, we can put

30:49

these safeguards in place to make

30:51

sure that it's not changing anything

30:53

it shouldn't, but yeah, it's gonna

30:56

be a major. overhaul, but it's like

30:58

the core of the app. And so it's

31:00

something that we're going to have to do.

31:02

And so it's just been interesting

31:04

to sit there and think about, okay,

31:06

yeah, if this had been written in

31:08

a maintainable, readable, understandable manner, right? We're

31:11

not even talking about how it executes

31:13

or how fast it isn't any other

31:15

way. It would have made the fixes

31:17

that we have to make to it

31:20

now is so much, so much easier. Yeah, and

31:22

90% of the problem is that

31:24

since it's all in one job,

31:26

it's, it does everything

31:29

serially, right? It does the

31:31

whole process in one

31:33

run without outsourcing any

31:35

of the work. Yep, everybody

31:38

has ran into that application

31:40

where they have a 26,000

31:42

line file and you're like,

31:44

what? Yeah, yeah. I don't

31:46

think this one's 26,000, but

31:48

it's a few thousand at

31:51

least. I'm thinking about a

31:53

very specific contract they

31:55

had a long time ago.

31:58

Oh, I'm sure. Yeah. Oh, God. of my

32:00

co-workers worked at that job too.

32:02

So if he listens to this, which

32:04

I'm sure you will, he'll get a

32:07

good laugh out of my comment

32:09

right there. Yeah. But it's interesting because

32:11

yeah, when we talk about performance, we

32:13

don't often talk about clean code,

32:15

maintainability, you know, and those kinds of

32:18

things because we're just talking about, yeah,

32:20

what are the tweaks we can make?

32:22

And then when we get into

32:24

it, this is the other thing that

32:27

I'm kind of pulling out of what

32:29

you've been talking about what you've

32:31

been talking about. a lot of times

32:33

the things that we're doing are not

32:36

just simple fixes right because you

32:38

know n plus one queries it's like

32:40

okay drop it includes on it right

32:42

you know you want to make

32:44

your query faster at an index right

32:47

and and most of the time the

32:49

simple fixes will get you a long

32:51

way but you know you get

32:53

too far down the line and yeah

32:56

all of a sudden it's I have

32:58

to restructure this to run a

33:00

different way or it's not going to

33:02

right in order to speed it up

33:05

or you know there's going to

33:07

be some some massive infrastructure changes in

33:09

order to make this faster or you

33:11

know I have to change the

33:14

way I'm doing cashing or something like

33:16

that where it's not just this straightforward

33:18

oh okay here's a shortcut you got

33:21

some speed out of it and

33:23

there are those I'm not saying they're

33:25

not but at a certain point you

33:27

get to that place where it's

33:29

like okay all the all the quick

33:32

fixes on this are done and it's

33:34

still not good enough so now

33:36

what? maintainable if you know if you

33:38

can approach it and understand it and

33:41

know how to change it it

33:43

it makes that job a lot easier

33:45

but it's it's also interesting because i've

33:47

worked contracts where it's like i'm not

33:50

i'm not going to pay it

33:52

right tests and i'm not going to

33:54

pay it a refactor and i'm not

33:56

going to pay it right and

33:58

and i'm looking at them and going

34:01

to go on. I'm going to have

34:03

to refactor or write tests in

34:05

order to understand what the hell what

34:07

this even does. so that I can

34:10

make it do what you want

34:12

or change what it does. Oh yeah,

34:14

yeah. Without test I wouldn't be able

34:16

to do any of my work. Right.

34:19

That is the key. And we

34:21

don't have perfect test. Nobody probably does.

34:23

But we have enough tests where all

34:25

of a sudden you change something

34:27

and you can feel have relative confidence.

34:30

And I think that is another thing

34:32

about performance work that you can't

34:34

be afraid to change things and break

34:36

it, which means you need to have

34:39

software switches everywhere. Where you turning

34:41

it on and you're seeing if you

34:43

broke, we've had extremely good luck within

34:45

my group of turning these switches on

34:48

and not breaking things. But if

34:50

without those switches and without the monitors

34:52

that you set up, you can't change

34:54

anything because of fear, you fear-driven

34:56

development. It's like Ford is talking about

34:59

before. It's like if you don't have

35:01

like the indices that we took

35:03

away, we didn't take it away because

35:05

we're scared. And was actually that before

35:08

the show started or during the

35:10

show that I talked about the indices,

35:12

forget. Anyway, I don't remember. Yeah. Anyways,

35:14

like, yeah, so anyways, fear of

35:16

driven development will like slow down performance

35:19

too because you don't, as you just

35:21

said, you don't have tests. You're too

35:23

scared to make any changes. I

35:25

think it will work and I don't

35:28

know, my first couple jobs didn't have

35:30

any tests. And then ever since

35:32

then, I've had several jobs where I

35:34

came in, like, no, we're writing. Yeah.

35:37

We were writing tests and people

35:39

like, how? It's like, well, let's learn.

35:41

Yeah. Well, that's the other piece on

35:43

the thing that I'm working on

35:45

is there aren't tests. And so it's

35:48

like, it's this mission critical thing that

35:50

does all the valuations for all of

35:52

the hedge funds and all of

35:55

your customers. God. And yeah, I mean,

35:57

I was just like, you, you want

35:59

a story. factor this but you know

36:02

but then I'm going putting a

36:04

test on this is it's ugly

36:06

yeah so ugly but the

36:08

flipside is is that making it

36:10

testable right writing the

36:12

test will force us refactor

36:15

some of it and when

36:17

you wrote code without test

36:19

typically methods all of a

36:21

sudden go from I like to

36:24

keep my methods within five lines

36:26

of code and ruby and job

36:28

I'm sure it would be different,

36:30

but five lines of code. No,

36:33

that's not a rule, but you

36:35

try to keep them small. Whereas

36:37

if somebody wrote the

36:39

original code without test,

36:42

usually they have 100, let's

36:44

just say 100 line methods

36:46

are not uncommon. Trying to

36:48

test the 100 line method

36:50

is like, I, there's too

36:53

many conditions, there's like seven

36:55

ifs here. What do I do? Yeah,

36:58

I've heard that I've heard that rule.

37:00

I've heard that rule all the way

37:02

up to like 10 lines of code.

37:04

Some people are a little bit more

37:06

rigid on it than others. My rule

37:09

of thumb is if I'm

37:11

talking to somebody and I'm explaining

37:13

how something works, and I try

37:15

in everything that kind of happens

37:17

at the same level at the

37:20

top level, right? It, it, it

37:22

retrieves the... Or maybe not even that,

37:24

right? That might be part of a

37:26

lower step. But usually it's, it makes

37:29

sure that these conditions are met, right?

37:31

And then if the conditions are right,

37:33

then it makes sure that, you know,

37:36

it locks the table, right, does this

37:38

other stuff in a transaction, right? So

37:40

there's this step, right? Maybe the transactions,

37:42

an implementation deal further down. But if

37:45

I can say, these are the five

37:47

steps to do a thing. Right, and they're

37:49

all mostly at the same conceptual level,

37:52

then that's what my method should look

37:54

like. And then underneath that, it's like,

37:56

okay, how do I do this stage

37:59

of the thing? right, and again, at

38:01

the conceptual level. And so once you

38:03

start getting lower into the code, when

38:05

I've written it, what you wind up

38:08

seeing is, is the bottom levels of

38:10

the methods, they'll have some things that

38:12

look like their implementation, right, where it's

38:14

like, I'm actually doing the query, and

38:17

then call this to do the rest

38:19

of the work, because I feel like

38:21

the retrieval and the work are at

38:23

the same level. But, you know, and

38:26

then at the very bottom, yeah, it's

38:28

just logic. And just being able to

38:30

approach that and say, okay, I see

38:32

what the process is, okay, I see

38:35

what the process for this part of

38:37

the process is, that's how I like

38:39

to approach it. So yeah, so sometimes

38:41

I wind up with a 20 or

38:44

30 line method. But at the end

38:46

of the day, I'm looking at it

38:48

and going, I understand, I can look

38:50

at this and know what it does.

38:53

I think if you can't look at

38:55

the method name and understand what a

38:57

piece of it does. It's time to

38:59

create another method name. Yeah, well that's

39:02

another one. When I first got started,

39:04

I kind of had this ethos from

39:06

rails, where it was like, you know,

39:08

they have convenient methods called like find

39:11

or, you know, update or things like

39:13

that. And I was like, okay, so

39:15

I've got to come up with this

39:17

short, you know, concise name. Nowadays is

39:20

like, updates the thing on the thing.

39:22

Because then you look at it and

39:24

you're like, oh, I know what this

39:26

does. I'll just make a

39:29

guess on how many methods there

39:31

are that we've created just on

39:33

our ticket object and it's got

39:35

to be At least 500. I

39:38

don't know whether that's five thousand

39:40

You can't create little things little

39:42

update Nothing and know what the

39:44

hell it's doing you need a

39:46

big well It's it's also interesting

39:49

because you're working on an application

39:51

that mostly focuses around the ticket

39:53

And so everything touches this one

39:55

core piece. Absolutely. and we created

39:58

several teams around it. Right. You

40:00

know, I've worked on other projects

40:02

that didn't have that, but then

40:04

they had like this core two

40:07

or three objects that were the

40:09

same thing. You know, I've almost

40:11

never seen an app where, you

40:13

know, everything spread across everything. The

40:15

only exception I can really think

40:18

of is the way the top

40:20

endevs is put together, but that's

40:22

because I deliver several different types

40:24

of media. And so, those kind

40:27

of get, between those and like

40:29

the membership and role management, you

40:31

know, it kind of gets spread

40:33

across all of that because it's,

40:36

do you have access to it

40:38

and then how do you deliver

40:40

it? And so, you know, I

40:42

get a wide swath of things

40:44

that people are going to be

40:47

generally touching. But yeah, I mean,

40:49

even within their little areas, right,

40:51

it's podcast episodes have a ton

40:53

of stuff on them, right? And

40:56

it's for that same reason, right?

40:58

Because within that little focused area,

41:00

I've got that core piece. And

41:02

so understanding that and then recognizing,

41:05

okay, so I've got to optimize

41:07

and make it as easy to

41:09

understand as possible around this thing.

41:11

There's some somebody else is going

41:13

to maintain it. Chris and yours.

41:16

That's right. But I'm the boss.

41:18

So they'll have to person quietly.

41:20

Granted, I worked at a company.

41:22

We'll go nameless that. Before I

41:25

got there, there's another topper that

41:27

only named us also. And they

41:29

didn't have, everything was mixture between

41:31

tabs and white space. There's always

41:34

extra white space at the end.

41:36

Nobody actually had any cleanup around

41:38

that. So I wanted to create

41:40

a script, basically made it. So

41:42

everything spaces, and there's no extra

41:45

white space at the end. So

41:47

you do a get blame in

41:49

that. And everybody thought I wrote

41:51

terrible code. I'm like, no, this

41:54

is not my code. Right. I

41:56

don't know what it is, but

41:58

get lame lies. Yeah, so yeah,

42:00

I want to bring it back to performance

42:03

a little bit. Kind of

42:05

hit the maintainability idea pretty

42:07

hard. And I think it's it's

42:10

a critical piece to this, right?

42:12

Because you're going to suffer

42:14

trying to make performance happen

42:16

with poorly maintained or hard

42:18

to maintain code. But when

42:20

you're, so you were talking about observability,

42:22

I want to go back

42:24

to that idea. Like, what are

42:27

you actually looking for? Are you looking

42:29

for kind of the, like new relic,

42:31

for example, we'll break it down

42:33

and say, you spent this much

42:35

time on your query and this

42:38

much time rendering the view and

42:40

this much time working in the controller

42:42

and this much time doing the other

42:44

things? Is that what you're looking at

42:47

when you see something slow? Or are

42:49

you looking at some other breakdown

42:51

or some other numbers? Or how do you

42:53

approach something like that? It was

42:55

just basically, this is how long

42:58

the presenter is taken, this is

43:00

how long the controller is taken,

43:02

and various little methods

43:05

throughout saying, say maybe

43:07

validations. When we started this

43:09

team, our first objective was,

43:12

hey, listen, we need observability

43:14

everywhere. So almost, I'm not

43:16

going to say every method,

43:18

but every validator, for example,

43:21

is wrapped in a data dog, optional

43:23

data dog. tag where you

43:25

can actually show really. So

43:27

literally every not every method but

43:29

a significant portion of the methods are

43:31

actually showing you can see and you

43:33

can narrow down exactly in the code

43:36

very fast because all of a sudden

43:38

it's like you turn it on you're

43:40

like there it is it's right there

43:42

and you might have like okay it's

43:44

within this section and there's three things

43:46

going on here of these three. And

43:48

there's sometimes, what happens is a lot

43:51

of times, it's one of the three,

43:53

we have the option of trying to

43:55

figure out where in the code the

43:57

slow part is, where we have the option

43:59

of just. deploying more code that will tell

44:01

us exactly where it is. We've

44:04

gotten to the point where there

44:06

is some empty gaps, especially within the

44:08

new features. When people add new

44:10

features, they're not thinking about adding traces

44:12

and stuff like that. But we

44:14

can really just go down and target

44:16

exactly the method that we're looking

44:18

for. We've had

44:20

things like you look at the code

44:22

and you're like, it looks fine, but you

44:25

don't see a loop, and then way

44:27

over here there's another loop, but that's in

44:29

there, and then the way over here

44:31

there's another loop, and that's in there too.

44:33

So you got, I think they call

44:35

that n -cubed or whatnot, and it's like

44:37

all of a sudden like, oh, obviously we

44:39

need to do something with this. And

44:41

then you add some, for that specific one,

44:44

I just added some memoization where all

44:46

of a sudden this last loop was just

44:48

literally just, it wasn't even a loop

44:50

anymore, it was just calling into a hash

44:52

table lookup, and this is a band

44:54

call. So all of a sudden that turned

44:56

into an n -cube, which is still not

44:58

good, but the performance of n -cube versus

45:00

n -squared or whatever is

45:03

huge. We

45:06

got a big enough performance gain there, where all

45:08

of a sudden I can run away from that

45:10

problem. Kick

45:14

it down the road a little bit.

45:16

Oh my God, some of the times you

45:18

see the code and you're like, it

45:20

is business decisions that people make too. Do

45:22

they make business decisions like, yeah, I

45:24

can see why the developer wrote it like

45:26

this, but why didn't the developer tell

45:28

you no? Like,

45:31

no, we can't do that. It's going to

45:33

not perform in five years. Yeah.

45:38

Sometimes you don't know though. At

45:40

the end of the day,

45:42

the one thing that I didn't

45:44

talk about today or I

45:46

might talk that I think is

45:48

really important is when you

45:50

first create an application and you

45:52

deploy it, you can always

45:54

set tight limits and the amount

45:56

of data that you have.

45:58

So product limits is the limits

46:00

we use. Hey, listen, you can only have 10

46:02

options in your dropdown. That's

46:04

obviously exaggeration, but say

46:07

you're allowed to put any 10, but

46:09

you only get 10. You can always

46:11

give more and eventually say,

46:13

I'll give you more, I'll give you more,

46:15

I'll give you more, I'll

46:18

give you more. But going

46:20

from, hey, you get unlimited,

46:22

I remember the first thing

46:24

I worked at at the

46:26

Zen desk was you can

46:28

have unlimited brands. Well, now

46:30

the limit is 3. 100

46:33

because we realize that unlimited

46:35

brands means almost well unlimited

46:37

complexity and you can't give

46:39

you 100,000 brands because and

46:41

that's 100,000 domains and that's

46:44

the complexity around that

46:46

is just too big so yeah

46:48

adding limits to your product

46:50

and as a developer I think

46:53

that's our responsibility to tell

46:55

the product people that There's

46:57

areas in the code you

46:59

go if we have a

47:01

hundred million of these what does

47:03

it? What's the performance? It's

47:06

Sometimes very obviously we wouldn't

47:08

be able to have a

47:10

hundred thousand these right when

47:12

we first developing the product

47:14

go back to your product

47:16

manager and say hey That feature if

47:18

somebody actually uses it and

47:21

uses it a lot It's not going

47:23

to work. And they might come back

47:25

saying, nobody would ever do that. So

47:27

what's the problem with actually having a

47:29

limit if nobody's ever going to

47:32

run it with? So I think that's

47:34

also very key. And that's not programming

47:36

per se, but I think 90, or

47:38

a huge percentage of our job as

47:40

a programmer is to go back to

47:42

the product managers and telling them, hey,

47:45

listen, three years from now, we wouldn't

47:47

want a drop down. We had

47:49

drop downs, for example that. had

47:52

over 100,000 options in them. Oh

47:54

wow. First off, the UI

47:56

isn't very good, but just

47:58

loading them up. And for

48:00

example, for this one, this one

48:02

was cool and it wasn't actually

48:05

a little talk because one of

48:07

the things that we did, well,

48:09

that customer that had 100,000 options,

48:12

they churn because, well, we were

48:14

too slow for them. But we

48:16

actually, we probably would be able

48:18

to support maybe even 100,000 than

48:21

the options now because what we

48:23

did is we changed our queries

48:25

and our original queries was grabbing

48:28

100,000 active record objects. And you

48:30

can't have 100,000 active record objects

48:32

without a serious time penalty. Right.

48:34

But what we do now is

48:37

we pluck for the data. So

48:39

we get the data fields that

48:41

we need, and we only need

48:44

like two or two fields out

48:46

of records. So you're not initializing

48:48

100,000 active record objects. So you

48:51

grab 100,000 options. Right. And that's

48:53

another reason why you have a

48:55

performance team, is all of a

48:57

sudden, this is very fast now,

49:00

and all of a sudden, that

49:02

customer that had 100,000 options that

49:04

churn five, six years ago, we

49:07

might be able to maintain them

49:09

now, but they're not coming back.

49:11

Right. And that's another reason why

49:13

you have a performance team. All

49:16

of a sudden, once you start

49:18

to have enough churn, people are

49:20

like, who customers are leaving because

49:23

of performance. Yeah. Do you have

49:25

any opinions around using things like

49:27

fibers or falcon in front of

49:29

rails to improve performance? Do you

49:32

have any experience with that? I

49:34

wish I did. No. Yeah, that's

49:36

good of the answer as I

49:39

can give you. Yeah. No, I

49:41

have not used fibers at all.

49:43

Yeah, I just started a new

49:45

side project and I put Falcon

49:48

and not, not because it's a

49:50

performance intensive project, it's just a

49:52

side project that lets me play

49:55

around with new stuff with no

49:57

risk. I was like, why not?

49:59

It'll be... fun. I did that

50:01

for, I worked for the U.S.C.

50:04

one time and the U.S.C. thought

50:06

they were going to get a

50:08

lot of different, but it

50:11

just, they thought day one

50:13

is going to be, bam,

50:15

it's just an e-commerce store

50:17

too, so it's not going

50:20

to get that much traffic,

50:22

but U.F.C. is like, but

50:24

we're U.S.C. it's like, yeah,

50:26

I'm just going to... tweak

50:29

every part of this e-commerce

50:31

app so you can go

50:33

as fast as possible. I

50:35

think they launched and had

50:38

almost no traffic, but that's,

50:40

it was fast for the no

50:42

traffic. Yeah, one thing that I'm

50:44

running into with Top End Devs

50:47

is the search takes like five

50:49

seconds to run. So search

50:51

is a very completely different

50:53

team for us, but for

50:55

us, it's the exact opposite.

50:58

So our search team is

51:00

better than my sequel.

51:02

Like whatever they're doing,

51:04

they've made it index

51:07

things quite fast. So

51:09

we've done, there's APIs

51:12

that sometimes you need,

51:14

they need it be like, no,

51:17

it knows current data, so

51:19

you were going after sequel.

51:22

But there's APIs where it's

51:24

like, hey, if it's delayed

51:27

by. 200 milliseconds. Is it

51:29

that big of a deal that we

51:31

don't return all the tickets? We

51:33

minus all the tickets except for

51:36

the one last one that they

51:38

put in? Right. And in those cases,

51:40

we've found the performance of

51:42

Alaska search is so much

51:44

faster. Like we'll get our

51:46

response times, we'll just drop

51:48

in half, for example. Yeah,

51:50

that's the thing that I've been looking

51:53

at lately is I put in PG

51:55

search. And yeah, it's

51:57

it's just you know.

51:59

doing a multi-search across all the

52:02

models. I mean it will give

52:04

you back a result, but I

52:06

had to tell the proxy server

52:08

that sits in front of it,

52:11

Kamal Proxy, not to time out

52:13

so fast in order to get

52:15

those results. And yeah, I've been

52:17

looking at possibly moving to Elastic

52:19

Search because I hear it's really

52:22

just stinking fast. I was using

52:24

another search engine Mealy Search, but

52:26

the problem I had there was

52:28

that they... I couldn't... I could

52:30

only search one type of object

52:33

at a time. And so I

52:35

had to go and say, okay,

52:37

give me all the episodes, give

52:39

me all the podcast. And it

52:41

was it was fairly fast, but

52:44

I hated making six queries to

52:46

it to get all the different

52:48

things, because I kind of want

52:50

just the most relevant thing to

52:52

show up first. And so, yeah,

52:55

that's where I'm looking at elastic

52:57

search, because I think elastic search

52:59

doesn't make you do that. Yeah,

53:01

we have a different team that

53:03

maintains that. So it's another place

53:06

where I just use elastic search

53:08

and I fast in our system,

53:10

but we have a, that's a

53:12

thing about being at a big

53:14

company. There's areas where you know

53:17

how to use it within your

53:19

company, but to set it up

53:21

or anything to be an expert

53:23

on it, it's like, oh, yeah.

53:25

That's where I miss the being

53:28

a startup. You get to be

53:30

an expert or at least. You

53:32

get to play with more tools.

53:34

Right. So are there any other

53:36

big secrets that you want to

53:39

share with us or thoughts before

53:41

we head into PICS? Nothing that's

53:43

coming into mind, but I mean,

53:45

as I said, we have... I

53:47

mean, I'm creating a whole new

53:50

talk, for example, I think I

53:52

told you before we started, it's

53:54

like how GraphQL is going to

53:56

speed up a whole bunch of

53:58

things for us, and then... Just

54:01

there's so many different areas that

54:03

I was not able to hit

54:05

that now I'm going to. I'm

54:07

submitting talks right now to actually

54:09

talk about those, but those are

54:11

not well thought out in my brain

54:14

yet, so I don't think I should

54:16

be talking about right now. Yeah, you

54:18

mentioned GraphQL and I have

54:20

to say my experience implementing

54:23

things in GraphQL has

54:25

been not my favorite thing

54:27

that I've ever done. But you're

54:29

talking about a performance

54:31

aspect of it. Yeah, I'm a

54:34

little curious what you're looking

54:36

at there. I know that with

54:38

GraphQL, the thing that people tend

54:41

to like is that you make

54:43

one query and you get all

54:45

the data you need in one

54:47

go. Is that where the speed

54:50

up comes from or is it

54:52

something else? No, no. So we

54:54

don't use Ruby every place. And

54:56

our GraphQ endpoints, we use Java.

54:59

And we're actually only using them

55:01

internally. We're not using them externally

55:03

yet. But internally, it hits our

55:05

Java server. Java does an

55:08

amazing job of paralyzing things.

55:10

So then, Java makes like 50,

55:12

well, I'm not 10 probably, but

55:14

let's say 50 different rest requests

55:16

to gain application, gets them all

55:18

back way faster than if it

55:20

was compounded off. And then. That

55:22

comes back to the Java app,

55:24

the Java app knows how to

55:26

configure and put it all in the way

55:29

it's supposed to put it back and

55:31

then serializes it and sent back to

55:33

the requester of the actual application.

55:35

And because of the parallelization

55:38

of that, it's so much faster. Because

55:40

rest requests, right now in our application,

55:42

we have something we call internally as

55:45

side loads. where you can basically say,

55:47

I want anything I want. It's almost

55:49

like a poor man's GraphQL. Well, if

55:52

all this sudden you're doing that serially,

55:54

it's just the response time just

55:56

here. So GraphQL is feeding us

55:58

up because it's basically. forcing us

56:01

to come back to true

56:03

restful responses that are really

56:05

fast. All right. Cool. Well,

56:07

let's go ahead and do

56:09

some picks. Before we get

56:11

to those, if people want

56:13

to follow you online, where

56:15

do they find you? I

56:17

think GitHub or LinkedIn is

56:19

where you'd find me. Like,

56:21

DR Henner is usually what

56:23

you find me. If you

56:25

look for DR Henner, GitHub

56:27

or DR Henner, I think

56:29

get LinkedIn. Even D.R. Henner

56:31

and my G-mail would work

56:33

too. Awesome. All right, well,

56:35

let's go ahead and do

56:37

some picks. Ayush, do you

56:39

have some picks for us?

56:42

Yeah, hang on one second.

56:44

I have broken broad for

56:46

my client, which is always

56:48

fun. I'm on it. I'll

56:50

be give me five minutes.

56:52

I'm on it. Yeah, perfect

56:54

timing isn't it? I shouldn't,

56:56

I shouldn't have merged up,

56:58

I merged up all the

57:00

questions just before coming into

57:02

the scholars shouldn't have done

57:04

that. Yeah, so I used,

57:06

when he got real quiet

57:08

over there, that's what was

57:10

going on, he's like, yeah.

57:12

Yeah, sorry about that. Very

57:14

picks. We'll start with a

57:16

very selfish pick, which is,

57:18

I just relaunched one of

57:20

my side projects called Scatagon.

57:22

Email. two and a half

57:25

years because of the joys

57:27

of sales tax and EU

57:29

VAT. I was using stripe

57:31

for payments on my account.

57:33

I was like, yeah, you

57:35

know, you can't do that

57:37

because you'll have to collect

57:39

VAT for everyone in the

57:41

EU. And I'm like, okay,

57:43

thanks for telling me that

57:45

now. So I moved payments

57:47

over to paddle, which took

57:49

a lot longer because I

57:51

got distracted by Heather stuff.

57:53

But it's back now, so

57:55

Scatagon. Email has relaunched it

57:57

last Tuesday. So that's one

57:59

pick. I usually do like a

58:01

music or a TV show pick so

58:04

I'll go to the TV show this

58:06

time. Severance on Apple TV season 2

58:08

just started. It's weird and I like

58:11

it I like it very much.

58:13

It's been a very strong start

58:15

I think it was quite a

58:17

cliffhanger ending in season one and

58:20

the show started off really

58:22

strong so if you haven't

58:24

watched it I'd recommend that.

58:26

Just getting the links

58:28

here for that. I've

58:31

got a couple of

58:33

picks I'm gonna put

58:35

out there So as most

58:37

of you probably know I

58:40

tend to do the board

58:42

game picks as part of

58:44

my picks so I'm going

58:47

to put out the game

58:49

is called Cascadero

58:51

and it's It's

58:54

got some elements that are similar

58:57

to other board games that I

58:59

played so there's kind of a

59:01

There's a track that you advance

59:04

up and you get rewards

59:06

for hitting certain milestones And you

59:08

also get points for hitting certain

59:10

milestones The way that you advance

59:13

up the track is you so

59:15

there's a kingdom and the the

59:17

king has died and so you're

59:19

basically you're out there connecting the

59:22

cities and things like that and

59:24

spreading the word on the new

59:26

king or whatever. And so you

59:29

put little horsemen down on

59:31

the board around these towns

59:33

and you're connecting up the

59:36

towns and you get bonus

59:38

points for say connecting two

59:40

towns to the same color.

59:42

So the first time you do

59:44

that for any given color,

59:46

there are five colors. Then

59:48

you get points. You advance

59:50

up the board. If you are the

59:52

first. person to that city and

59:54

it's not the first horseman

59:56

that you put down in that group

59:59

then you get point and if you're

1:00:01

the second third fourth I guess you

1:00:03

can do fourth because other people can

1:00:06

connect onto it as well right but

1:00:08

it's it's per group so when you

1:00:10

connect a group to a city and

1:00:12

you're the second or third or fourth

1:00:15

person you move your thing up the

1:00:17

advanced track or advanced up the track

1:00:19

of that color two spaces instead of

1:00:22

one and then if there's a herald

1:00:24

on the city then you add one

1:00:26

to whatever you get to advance it.

1:00:29

And then there's a bonus for having

1:00:31

connected to cities of all five colors

1:00:33

with any, you know, across any of

1:00:36

or all of your groups. And then

1:00:38

there are also point bonuses for getting

1:00:40

all five of your markers past the

1:00:42

first. It's kind of a finish line,

1:00:45

three on the second finish line or

1:00:47

getting being the first person to get

1:00:49

one all the way to the end

1:00:52

of one of the. lines and the

1:00:54

way that you win is you have

1:00:56

the most points of the people that

1:00:59

get their Their piece all the way

1:01:01

to the end of whatever color they're

1:01:03

playing So when I was playing I

1:01:06

was playing as red and so I

1:01:08

made sure I got my piece all

1:01:10

the way to the end of red

1:01:13

and then I tried to get it

1:01:15

all the way to the end of

1:01:17

yellow as well, but I didn't have

1:01:19

to do that And then one of

1:01:22

the other guys that I was playing

1:01:24

with got his all the way to

1:01:26

the end of blue, which was the

1:01:29

color he was playing. But he did

1:01:31

that on his last turn. And if

1:01:33

he hadn't gotten it there, even though

1:01:36

he had more points than I did,

1:01:38

because I got mine to the end

1:01:40

of the track, I won. And so

1:01:43

what your goal is, is you want

1:01:45

to be the person who gets it

1:01:47

all the way to the end and

1:01:49

has the most points. But the way

1:01:52

you end the game, is by getting

1:01:54

more than getting more than 50. Or

1:01:56

when somebody runs out of horsemen and

1:01:59

can't place another horseman, that's the other

1:02:01

way it ends. And that's the way

1:02:03

it ended for us because we had

1:02:06

never played it before. And so we

1:02:08

weren't rack up. the points the way

1:02:10

that we should have been. But anyway,

1:02:13

it was pretty fun. And then there,

1:02:15

like I said, there are different rewards

1:02:17

up the track that, you know, so

1:02:20

you can advance another marker up the

1:02:22

track or you can place another horseman

1:02:24

or you can move a horseman or

1:02:26

just stuff like that. So anyway, it

1:02:29

was pretty fun. Cascadero, it came out

1:02:31

in 2024, so it's a relatively new

1:02:33

game. And board game geek gives it

1:02:36

a weight of 2.53. which means that

1:02:38

if you're kind of a casual player,

1:02:40

this might be a little bit complicated,

1:02:43

but not so complicated that you're not

1:02:45

gonna enjoy it. I have to say

1:02:47

that the complicated part is just keeping

1:02:50

track of what you've connected with your

1:02:52

groups and then, you know, keeping track

1:02:54

of how to move stuff up the

1:02:56

track. And you get used to that

1:02:59

in the first 10 minutes or so.

1:03:01

And then I would probably wait it

1:03:03

once you understand those two things at

1:03:06

like 1.9. I mean it's it's just

1:03:08

those couple of concepts you have to

1:03:10

get used to and then you're good

1:03:13

to go. It took us about there

1:03:15

were three of us playing took us

1:03:17

what 45 minutes maybe an hour and

1:03:20

that was with us learning the game

1:03:22

so I would I would assume that

1:03:24

it probably takes you know if you

1:03:27

have four or five people playing you

1:03:29

know and take you 45 minutes. So

1:03:31

anyway it was fun. Definitely enjoyed it.

1:03:33

Definitely enjoyed it. Definitely enjoyed it. One

1:03:36

of my buddies is teaching the hot

1:03:38

games at SaltCon in March, which is

1:03:40

the big board game convention up in

1:03:43

Layton, Utah. And yeah, anyway, so that

1:03:45

was one of the games he had

1:03:47

to learn. We'll probably pick up another

1:03:50

one later. But yeah, I really enjoyed

1:03:52

it, so I'm going to pick that.

1:03:54

My wife and I for TV shows.

1:03:57

We just started watching season two of

1:03:59

the night agent. And we enjoyed the

1:04:01

first season a lot, so I'll pick

1:04:03

that. that as well. Let me see

1:04:06

if I can get a link for

1:04:08

that. And then I am starting to

1:04:10

put together meetups and stuff for Ruby

1:04:13

Geniuses. So if you go to Rubygeniuses.com,

1:04:15

I kind of, so it took me

1:04:17

a little longer to get it launched

1:04:20

than I wanted to. And the reason

1:04:22

essentially is that the stuff that I'm

1:04:24

doing for Top End Devs as far

1:04:27

as like delivering courses and podcasts and

1:04:29

stuff like that. I've been talking to

1:04:31

other people who want to use the

1:04:34

system I'm using. And so rather than

1:04:36

do the smart thing and just deploy

1:04:38

them a copy, I made it a

1:04:40

multi-tenant app so I can charge them

1:04:43

for it. So that took a bit

1:04:45

of work to kind of get everything

1:04:47

figured out and learn how to do

1:04:50

multi-tenancy with Stripe Connect on, you know,

1:04:52

on my app and make sure I'm

1:04:54

getting all the right. web hooks and

1:04:57

all that stuff, but it's basically done.

1:04:59

So if you want to go sign

1:05:01

up, I'm going to change the price

1:05:04

today. I'm going to move it down

1:05:06

to $47 just because I'm still pretty

1:05:08

happy about the inauguration of Donald Trump

1:05:10

and he's president 47. But yeah, so

1:05:13

if you want to go and sign

1:05:15

up for $47, it will go back

1:05:17

up to $97 in a few weeks.

1:05:20

And that will give you access to,

1:05:22

I'm essentially, I'm putting together videos kind

1:05:24

of like Railscastsas was. or go rails

1:05:27

or drifting ruby. My focus is gonna

1:05:29

be, essentially I'm gonna say, here's my

1:05:31

project, and then you're gonna get a

1:05:34

video of me working on the project

1:05:36

every week. And it might be interspersed

1:05:38

with, and this is something that I

1:05:41

fixed on top endevs, right? So if

1:05:43

I get elastic search working for the

1:05:45

search, for example, I might work that

1:05:47

in. The first app I'm building is

1:05:50

actually to help track bills through the

1:05:52

Utah legislature. And so it's going to

1:05:54

be hitting their APIs and working around

1:05:57

the gaping holes in their API. because

1:05:59

they have gaping holes in their APIs.

1:06:01

So there may be some screen scraping

1:06:04

and things like that to get all

1:06:06

the data in and then yeah put

1:06:08

together the alerts and stuff like that. So

1:06:10

you know just you know you get a notice

1:06:12

that says hey your bill went to committee

1:06:15

or you know your bill had its second

1:06:17

reading and so if it has a

1:06:19

third reading and passes then it passes

1:06:21

you know just kind of working through

1:06:23

all that stuff and then helping

1:06:25

people connect with their legislators. Because

1:06:28

I feel like if we have more people

1:06:30

involved in the legislative process

1:06:32

and talking to their legislators,

1:06:35

regardless of which side of the aisle

1:06:37

you're on, I think we generally get

1:06:39

better legislation. Because I think

1:06:41

most of the things when we're talking

1:06:44

about roads and schools and things like

1:06:46

that, we generally agree on what we

1:06:48

want, whether you're right or left. And

1:06:50

so what I want is I want

1:06:53

moms and dads talking to the legislators

1:06:55

and saying... You're pushing this education bill,

1:06:57

but that's not what we want. You

1:06:59

know, or things like that. So that's

1:07:01

what I'm going to be working through

1:07:03

and just showing people how to do

1:07:06

that. It's probably going to have some

1:07:08

premium features too that involve AI

1:07:10

and doing bill summaries for AI. I'm

1:07:12

already doing this for another project.

1:07:15

And so yeah, you're going to get

1:07:17

kind of a wide breadth of things

1:07:19

out of this. Also creating groups so

1:07:21

people can discuss bills together. I

1:07:24

want to give organizations an option

1:07:26

for putting together their slate of

1:07:28

bills and where they stand on

1:07:30

it and give them some way of posting

1:07:32

that on their websites. So those are

1:07:35

kind of the features that I'm going

1:07:37

to be putting in. So, you know,

1:07:39

embedable stuff. So anyway, if you're looking

1:07:41

to learn any of those things, then

1:07:43

you want to sign up for that.

1:07:45

We're also doing the meetups. and we'll

1:07:47

be going through some of the technologies

1:07:49

maybe a little more in depth. So

1:07:51

where in the videos I'll be showing

1:07:54

you, I'm building this feature using turbo,

1:07:56

I may go through, hey, here are

1:07:58

all the features that turbo off. in

1:08:00

this way or Kamal or I've

1:08:02

had people ask me for all

1:08:04

kinds of stuff. If you connect

1:08:07

with me on LinkedIn too I'll

1:08:09

actually get on a call with

1:08:12

you and ask you what you

1:08:14

want to learn. So anyway I've

1:08:16

been way more long-winded on that

1:08:19

than I want but yeah so

1:08:21

that's all going to be coming

1:08:23

out through Rubygeniuses.com and yeah so

1:08:26

hopefully that's in the wheelhouse of

1:08:28

things you want but yeah we're

1:08:31

going to be doing meetups and

1:08:33

tutorials and courses and the whole

1:08:35

nine yards and you get access

1:08:38

to all of that through the

1:08:40

membership. So anyway, David, what are

1:08:42

your picks? Yes, so I have

1:08:45

a co-worker actually that I work

1:08:47

side-by-side with that's publishing a book

1:08:50

soon and it's called Rail Scales.

1:08:52

He actually gives more talks about

1:08:54

scaling rails than I do, but

1:08:57

Christian Palantos is his author's name.

1:08:59

book name Ralph Scales. I'm actually

1:09:01

very interested in to see what

1:09:04

it looks like and I can't

1:09:06

wait to get myself a copy.

1:09:09

I hope I get myself a

1:09:11

free copy. Yeah. Become a technical

1:09:13

reviewer they give those out for

1:09:16

free. Well I mean he works

1:09:18

with me and he's yeah I'll

1:09:20

tell him I gave him a

1:09:23

shout out. There you go. What

1:09:25

else? I actually have one other

1:09:28

co-worker that he wrote a an

1:09:30

article on AWS about scaling Zendesk.

1:09:32

And if you do a search

1:09:35

for the guy's names Anatole, AWS,

1:09:37

Zendesk performance, so Anatole, AWS, Zendes

1:09:39

performance, you will find his article

1:09:42

in Google, and he talks about

1:09:44

some of the things that we've

1:09:47

been doing much more on the

1:09:49

infrastructure side. He's a much more

1:09:51

of an infrastructure than a Ruby

1:09:54

guy. But I work side by

1:09:56

side of that guy. He's absolutely

1:09:58

brilliant. Also, if anybody's

1:10:01

hiring product manager or managers in

1:10:03

general, I know several that got

1:10:05

laid off, so feel free to

1:10:07

ping me on LinkedIn because I

1:10:09

feel like there's some good people

1:10:11

out there that I know that

1:10:14

I'd love to connect to other

1:10:16

people. And lastly, you guys probably

1:10:18

don't know Zwift, but Zwift is

1:10:20

essentially an interactive game that you

1:10:22

connect to a bicycle, a bicycle

1:10:24

trainer. And I use it. I

1:10:27

absolutely love it. I mean, every

1:10:29

night, like I do races sometimes

1:10:31

and the last few nights I've

1:10:33

been doing races. I am not

1:10:35

a racer. I am not that

1:10:37

fast. But at the end of

1:10:40

the day, I'll still go in

1:10:42

and do the races. And I

1:10:44

think the category to use, I'm

1:10:46

taught. I'm the best of the

1:10:48

worst category. Oh, there you go.

1:10:50

So as soon as I get

1:10:53

new with the next category, I'll

1:10:55

get destroyed. Yeah, I got a

1:10:57

membership to Zwift when I bought

1:10:59

my bike trainer because I have

1:11:01

a Wahoo trainer and Is it

1:11:03

the bike or okay? I got

1:11:06

the kicker bike Okay, yeah, I've

1:11:08

only used it once or twice

1:11:10

in fact, I think it expired

1:11:12

but Yeah, it's cool because it's

1:11:14

yeah, it's almost like well I

1:11:16

guess it's not almost like but

1:11:19

it's it's kind of fun to

1:11:21

just ride your bike and bike

1:11:23

and yeah If you're getting harder

1:11:25

up the hills and all that

1:11:27

stuff and exactly and then you're

1:11:30

in Utah So you you suffer

1:11:32

from the coldness and you don't

1:11:34

want to be biking 20 years.

1:11:36

Oh, yeah I don't train outside.

1:11:38

I'm training for a triathlon now

1:11:40

this year I want to do

1:11:43

either an Iron man or half

1:11:45

Iron man I haven't decided I

1:11:47

do that with you, but I

1:11:49

think my knees would say no

1:11:51

on the running like I got

1:11:53

an MRI scheduled for next week

1:11:56

for my shoulder so I'm breaking

1:11:58

down Yeah, then last year. Oh,

1:12:00

I watched my dad fall apart and I

1:12:02

just I feel like I need to keep moving so

1:12:04

I don't so The last

1:12:06

day I was going to give this

1:12:08

my last thing to saying go bills

1:12:11

because I'm hoping the for the Super

1:12:13

Bowl, but Well, I still say go

1:12:15

bills because I love them Okay,

1:12:18

so have you decided if you're rooting

1:12:20

for the Eagles or the Chiefs? Have

1:12:23

I decided what they're

1:12:26

watching the Fair enough

1:12:28

um You

1:12:31

know, I don't hate the Chiefs, but at

1:12:33

the same time I got a route

1:12:35

for the Eagles just because I

1:12:39

don't want the Chiefs to win three

1:12:42

times in a row Like they're

1:12:44

not bad. They're not like Patriots where

1:12:46

I hated them But still

1:12:48

I don't want to win three times a row, and

1:12:50

I think that their fan base understands that too Yeah,

1:12:53

I like both teams, so I'm

1:12:55

not sure which way I'm swinging

1:12:57

yet, but Say quite

1:13:00

a good deal. What was that? Take

1:13:02

one was amazing this year. All right

1:13:04

Well, I'm gonna go ahead and end

1:13:06

the stream, but thanks for coming. Thank

1:13:08

you and until next time folks max

1:13:10

out