0:00

You're listening to

0:00

The Catalyst by Soft Choice, a

0:03

podcast about unleashing the full

0:03

potential in people and technology.

0:07

I'm your host, Heather Haskin. Imagine waking up in a world where

0:14

machines don't just assist us.

0:18

They think, create, and

0:18

even make decisions.

0:21

That world isn't decades away. It's happening right now.

0:25

In just a few years, AI has crossed

0:25

a threshold with platforms like

0:29

ChatGPT, moving it from an emerging

0:29

technology To a transformative

0:34

force, reshaping industries,

0:34

creativity, and society itself.

0:39

But how did we get here? In today's episode, we're tracing AI's

0:41

recent history, uncovering the pivotal

0:45

moments that shaped the industry. We'll explore the breakthroughs

0:47

that reignited deep learning,

0:50

the rise of neural networks, and

0:50

the key milestones that brought

0:54

generative AI into the mainstream,

0:54

revolutionizing the way we do business.

0:59

And the AI market isn't slowing down. With innovations, like China's DeepSeek,

1:01

shaking up the industry, what's next?

1:07

So how did AI go from niche research

1:07

to a force redefining our world?

1:13

Today we're joined by Gil Press, a

1:13

senior contributor at Forbes magazine.

1:18

Gil covers emerging technologies,

1:18

startups, big data, and the history

1:22

and future of artificial intelligence.

1:26

So, Gil, I'm really excited to

1:26

be able to interview you today.

1:29

I looked into your background a little

1:29

bit, and it seems like you have quite

1:32

an extensive bird's eye view on the

1:32

history of generative AI, and so that

1:36

brings to light so many questions.

1:38

As we think about AI today, it's

1:38

such a buzzword, but there seems

1:42

to be something different about

1:42

the moment that we're in right now.

1:47

And it makes me wonder,

1:47

have we turned a corner?

1:50

What's changed? And maybe talking about that history

1:51

will help us understand that.

1:54

So I'd love to hear from

1:54

you what your thoughts are.

1:57

What's changed recently is

1:57

that, uh, even at AI has been

2:03

around for at least 70 years.

2:05

So the research on the right, some,

2:05

some implementation of, of, uh,

2:10

various AI programs recently, both

2:10

the scope and the reach of AI.

2:18

have expanded significantly.

2:21

And now we have AI programs

2:21

that are expanding the menu

2:26

of what computers can do. For example, generate text or

2:28

generate images, the reach of AI

2:34

in the sense of, uh, I'm managing

2:34

to reach millions of computer users

2:41

around the world, mostly because of

2:41

the very recent success of ChatGPT

2:46

that's what really is happening now.

2:49

It's been a long, long journey, and

2:49

especially over the last 25 to 30 years,

2:57

in terms of very specific advancements

2:57

in what is called deep learning.

3:04

It's interesting to hear

3:04

about some of the changes that have

3:06

happened over time, Gil, and I'm really

3:06

excited to talk with you about some

3:09

of them because there's some things

3:09

that I hadn't really learned before.

3:13

So today we're looking at the

3:13

recent history of AI going back

3:15

to the turn of the 21st century.

3:17

An important paper was

3:17

written in the year 2000.

3:21

And I'd love to hear about that

3:21

paper from your perspective.

3:25

Why was it important? Why did it renew interest

3:27

in deep learning?

3:30

And maybe what was the initial impact in

3:30

the early 2000s after it was published?

3:35

That year there was another

3:35

advancement in deep learning in a sense

3:39

that there were three leading computer

3:39

scientists that insisted for many years

3:46

that this is the right approach to AI.

3:48

And now they had a lot of challenges

3:48

to that and a lot of skepticism.

3:53

The three were Geoffrey Hinton,

3:53

Jan LeCun, and Joshua Bengio.

3:58

And Bengio and his team published in

3:58

the year 2000, a paper on training

4:04

the computer to understand language,

4:04

text processing, language processing.

4:10

And the breakthrough there was the

4:10

idea that there will be a lot of

4:15

examples that mix together phrases with

4:15

similar words, but not similar meaning.

4:24

We humans understand it.

4:26

Computers at that time could not

4:26

understand it at all, got confused,

4:30

and then devised a model that

4:30

overcame that particular challenge.

4:35

This was basically the launching pad.

4:38

For what today we call large language

4:38

models, the ability of the computer

4:44

to understand context, to understand

4:44

the different meaning of words,

4:51

even if they're the same words.

4:53

So we see the next important

4:53

moment in AI between 2007 and 2009.

4:59

What happened then? How were data and processing power

5:00

factors in the technology of the day?

5:05

Why was deep learning a step

5:05

forward from machine learning?

5:09

It's actually an easy question. Deep learning is part of machine

5:11

learning, but there are a lot of

5:14

approaches to machine learning.

5:17

Uh, the advantage of deep learning

5:17

can be summarized in two words.

5:25

big data. So most of the traditional and very

5:26

successful machine learning approaches

5:32

relied on analyzing small sets of data.

5:35

Deep learning is different in the

5:35

sense that it shows its advantages, it

5:40

shows its benefits, specifically when

5:40

we deal with lots and lots of data.

5:46

And by the year 2005 2010, we had lots

5:46

and lots of data because of the invention

5:55

of the World Wide Web in the early 1990s.

6:00

By the way, what we see today in

6:00

this sudden expansion and triumph

6:05

of modern AI is very similar to what

6:05

happened When the World Wide Web was

6:10

invented, it was basically a piece of

6:10

software that was installed on top of

6:18

an existing worldwide network that was

6:18

established 30 years before the internet.

6:27

So today we see this triumph of

6:27

modern and I also based on not a

6:32

lot of work for many, many decades.

6:35

But specifically in around 2005 to 2010,

6:35

and specifically one AI researcher,

6:44

Fei Fei Li, came up with the idea that

6:44

if we have all these images on the

6:50

internet, and all of these images are

6:50

labeled, annotated, because people put

6:56

a picture of dog and they said, this

6:56

is my dog, Sasha, which is my dog.

7:02

But then you see it's, they are

7:02

identified, the images identified.

7:05

That's very important when you do

7:05

deep learning, because otherwise

7:09

you have to do it by hand. Each image, because you train the computer

7:12

and you say, this is an example of a cat.

7:17

So, Fai Fei Li and others. started collecting, scraping the

7:20

internet for all of these labeled

7:24

images, and they put together a

7:24

database they called ImageNet.

7:30

By 2010, they announced an annual

7:30

event, an annual competition, in which

7:37

AI researchers could submit their

7:37

program, specifically image recognition

7:42

programs, to compete with each other. Also, at that time, we had another very

7:44

important element, as I mentioned before,

7:50

the challenge for deep learning for many

7:50

years was the lack of computer power.

7:56

When you deal with lots and

7:56

lots of data, you need a lot

7:59

of computer power to analyze. A company by the name of NVIDIA, already

8:02

in 1993, was established to develop what

8:10

they called graphic Processing unit.

8:13

GPUs. Graphic processing unit, originally,

8:16

when NVIDIA was established,

8:21

were used for computer games.

8:23

So all of this came together, the GPUs,

8:23

big data, and deep learning, advanced

8:32

algorithm, to create a perfect storm.

8:37

A perfect storm that hit us in 2012.

8:47

It's interesting that the

8:47

computers that gamers are using, this

8:51

GPUs, they're nicer video cards, seem

8:51

to somehow advance our deep learning.

8:57

So it's so cool to see how

8:57

different channels of technology are

9:00

converging together during this time.

9:03

So when we come to 2010 and

9:03

we're thinking about ImageNet,

9:08

you mentioned that competition. Why was image and computer

9:11

vision harder to do?

9:16

Identifying images was hard to

9:16

do simply because other approaches

9:20

other than deep learning did not

9:20

use, for example, lots of data.

9:24

Deep learning itself had a challenge

9:24

in doing what it's doing best, which is

9:28

analyzing lots of data simply because

9:28

it didn't have enough computer power.

9:34

But when you put all the three together,

9:34

and that happens in 2012 in, in

9:40

various settings, it's bad the most. Impressive one, the one that influenced

9:42

the later developments was the ImageNet

9:48

competition at the end of 2012.

9:51

And in this competition, Geoffrey

9:51

Hinton and two of his PhD students

9:57

submitted their program, which was

9:57

a deep learning program, but based,

10:02

this is the first time, based on GPUs.

10:05

Which means basically they

10:05

processed whatever the images

10:09

from ImageNet very, very quickly.

10:12

And most important, it was more

10:12

accurate than all the other competitors.

10:17

Its error rate at that time,

10:17

at that particular competition,

10:21

was 15 percent error rate.

10:23

The second best program in that

10:23

competition had an error rate of 26%.

10:30

Using a different approach. Machine learning, but different approach.

10:33

So, um, at least the academic

10:33

world completely changed its mind

10:39

about the benefit of deep learning.

10:43

It was huge excitement

10:43

in the academic world.

10:46

Then the venture capitalists paid

10:46

attention and other investors.

10:51

The government started to pay attention. And we were off to the races of modern AI.

10:56

It makes me wonder what is

10:56

going on in academia right now that

11:00

I should be paying attention to, that

11:00

we'll hear about in a few years time

11:04

when investors make those big moves.

11:07

So as we think about that change

11:07

with ImageNet and these, these big

11:11

updates back in 2012, what did we

11:11

see the big industry companies do?

11:17

Google, I don't even know if

11:17

it was AWS or Amazon back then,

11:20

but what are the big names doing

11:20

with all this back at that time?

11:24

This is a very good question. And I actually omitted mentioning

11:25

big tech in my previous answer.

11:31

It was the venture capitalists, it

11:31

was academics, and most important,

11:35

actually, it was big tech, the Googles

11:35

of the world, because at that time

11:41

there was a very interesting shift from

11:41

a focus of this research in academia.

11:48

And for 70 years, the focus of, of

11:48

AI research was mostly in academia.

11:54

There were periods where there was

11:54

more of a business interest, especially

11:57

with expert systems, so called expert

11:57

systems in the seventies and eighties.

12:02

But mostly it was a, an academic focus.

12:06

But by 2012, we had Google as a

12:06

good example of a company that has

12:13

always, since its inception, almost

12:13

modeled itself after a university.

12:18

Publish or perish was a motto, not

12:18

just in academia, but also at Google.

12:25

They immediately, from the beginning, not

12:25

only invested a lot in research, of course

12:30

around search technology mostly, but also

12:30

about deploying information technology.

12:36

They did a lot of internal innovations.

12:38

So then you see, there

12:38

was already a huge shift.

12:41

And certainly around 2012, it was

12:41

established that to attract talent,

12:47

a company like Google, like Facebook,

12:47

like Amazon, pay a little bit less

12:52

attention even to the implementation,

12:52

to application, the business value of

12:56

that research, but also allowed them

12:56

to build their own reputation just

13:02

like academics do, by publishing. First of all, a lot of academics, the

13:05

ones who had already experience in

13:09

deep learning or other approaches to

13:09

AI, left academia and went to Google.

13:14

Yann LeCun was a good example for a

13:14

long time, being a tenured professor

13:20

at New York University, but he became

13:20

the chief AI scientist for Facebook.

13:26

Many of them just left academia

13:26

and went to work for a lot more

13:29

money for these big tech companies.

13:31

I'm starting to wonder

13:31

when we're getting to the

13:34

start of the modern AI boom.

13:36

So as we talk about 2012,

13:36

are we getting there yet?

13:40

The ImageNet competition of 2012

13:40

is the start of the modern AI boom.

13:47

The big, I think, event that made it more

13:47

accessible Was in 2016 when a program

13:57

from Google, its AI unit at the time,

13:57

DeepMind a program that managed to beat a

14:05

champion of go a game that was considered

14:05

to be a much, much bigger challenge for

14:11

AI programs than chess, for example.

14:15

But the AlphaGo program from Google gained

14:15

a lot of attention, a lot of headlines.

14:22

And I think this is where the

14:22

average newspaper reader found

14:28

out for the first time about this,

14:28

what was basically deep learning.

14:33

People stopped using, by that time, deep

14:33

learning as a label and started using AI.

14:45

Going back to 2014, a couple

14:45

years later, a man named Ian Goodfellow.

14:51

had a big idea that changed the AI game.

14:53

I'd love to hear more about that. What did he do?

14:55

What did neural networks talking

14:55

to each other allow AI to do?

15:02

So this is when we have the birth

15:02

of yet another buzzword, generative AI.

15:07

This is the first time that he

15:07

and some other people came up

15:10

with the idea that if you have two

15:10

models compete with each other.

15:16

that will help the program generate new

15:16

data, generate images, generate text,

15:26

generate eventually videos, and so on.

15:29

A very powerful idea that was taken by

15:29

a lot of other people, a lot of other

15:35

researchers, a lot of other companies. This was in 2014, but the real

15:37

big milestone, and maybe the last

15:43

milestone we have in terms of the

15:43

progress of thinking and developing

15:48

deep learning, what we now call AI. It came in 2017, and there are a

15:51

number of, uh, researchers at Google

15:58

published a paper titled Attention is

15:58

All You Need, in which they suggested

16:04

a new type of architecture, a new way

16:04

to design the deep learning model.

16:09

that allowed the program to

16:09

understand the context of

16:17

whatever was written in the text. Context meaning it could read, so to

16:19

speak, a whole paragraph and understand

16:26

the connection and the relations between

16:26

words far apart from one another.

16:31

Before that, it was really words one after

16:31

another that the program could analyze.

16:37

Now, with this new, actually simplified

16:37

architecture, This was the flourishing

16:42

of large language models started because

16:42

other big tech researchers, other

16:49

entities started competing with each

16:49

other by releasing better and better

16:55

large language models, more accurate.

16:58

Performing better and so on.

17:00

Is this the moment that AI

17:00

entered the mainstream with ChatGPT?

17:04

That's what it feels like to me.

17:06

Yeah. When ChatGPT was released in November

17:07

of 2022, it became within two months the

17:16

most popular consumer application ever.

17:18

I think a hundred million

17:18

users within two months.

17:21

It suddenly reached a very wide audience.

17:25

People got excited about it. People got upset about it.

17:29

Recently, we had surprising

17:29

development from a Chinese company,

17:36

Chinese AI company, DeepSeek.

17:38

It managed to do the same with much less

17:38

computer power and smaller amount of data.

17:46

So DeepSeek is a Chinese company.

17:49

It's also open source. So how does that impact the AI industry?

17:55

Is there controversy about it being

17:55

open source and what is the larger

17:59

business world doing with that?

18:00

Yeah. So open source is not something that Deep

18:01

Seek brought to the table a few years ago.

18:08

Facebook, which is investing

18:08

enormous amount of resources in

18:14

modern AI, made the decision to open

18:14

source everything that it's doing.

18:19

So open source is a very important

18:19

trend or approach to business within AI.

18:26

And it's been, before DeepSeek,

18:26

it's been, let's say, a threat.

18:31

to those companies like OpenAI that

18:31

keep the nature of the models, the

18:38

innards of the models to themselves.

18:41

When we look at the future,

18:41

though, what do you think we're going to

18:44

see in the next few years with AI models?

18:47

And how do you think that will

18:47

change the way that we do business?

18:51

As you know, prediction is

18:51

difficult, especially about the future.

18:55

I tend to focus on the history,

18:55

but history gives us some

18:57

indication of where are we going.

19:00

I do think that we will see steady and

19:00

slow improvement in the accuracy and

19:06

the performance of deep learning models.

19:08

And maybe even more important, we

19:08

will see steady and very, very slow

19:14

adoption by businesses of whatever

19:14

AI can do for them to save money

19:22

and to generate new revenue streams.

19:25

Well, that's the big

19:25

question right now is how businesses

19:28

can utilize generative AI to

19:28

bring ROI to their business.

19:32

So I'm excited to see how. The world changes with all of these

19:35

incredible technology changes.

19:40

And I appreciate having

19:40

you here on The Catalyst.

19:43

Thank you for having me.

19:47

That wraps up our journey

19:47

through the modern history of AI,

19:50

from its quiet resurgence in the

19:50

early 2000s, to the breakthroughs that

19:54

brought us today's AI driven world.

19:56

We've seen how deep learning,

19:56

neural networks, and large language

20:00

models have shaped industries and

20:00

transformed the way we work, create,

20:04

and interact with technology. But this is just the beginning.

20:07

AI is evolving faster than ever,

20:07

and the next big shift could

20:11

be right around the corner. Thank you to Gilpress for coming

20:14

on the show to share his insights.

20:17

This is the last episode of the season. We'll be taking a short break

20:19

and we'll be back soon with more

20:22

episodes on everything tech and AI.

20:25

For The Catalyst, I'm Heather Haskin.

20:29

The Catalyst is brought to you by

20:29

SoftChoice, a leading North American

20:32

technology solutions provider. It is written and produced by

20:34

Angela Cope, Philippe Dimas,

20:38

and Brayden Banks in partnership

20:38

with Pilgrim Content Marketing.