Selected Papers on Computer Science (Don Knuth) and the current state of Deep Learning

This is the most accessible book from Don Knuth. Although I was published
nearly 20 years ago, it still is a classic computer science book. In Amazon,
there is an interesting comment about the book from Peter Norvig (Director
of Google Research). The major topic in the book is the origination of
computer science in the period of which the discipline is a new thing. Through
chapters, we can see there were many debates and struggles among scientists
about whether if computer science is truely a “science” and not a branch of
mathematics [Chapter 1, 2, 3].

This makes me think about the current state of Deep Learning. Maybe 5 or 10
years later, Deep Learning will become a separate discipline as Computer
Science segragated from mathematics several decades ago.

There are especially interesting chapters in the book which I can describe
as below:

Chapter 0: a general overview about Computer Science.

Chapter 1: Computer Science and its relaton to mathematics: the difference
between modern mathematics and computer science. Besides, the author also mentioned
the analysis of the classic algorithm: hashing.

Chapter 2 and 3: The overview about algorithms as well as the approach by
which the author solves algorithmic problems. Although the chapter is published
long time ago (1976-77) which maybe quite new at that time, in my opinion these
mentioned algorithm, namely the shortest paths, searching and
combinatoric optimizations, become classic research in CS nowadays.

Chapter 6-9: Theory and Practice: the whole book is distilled into these chapters.
Historic text actively disscuss the most important aspects in CS.

Chapter 11-13: The history of Computer Science: From ancient civilization uses
algorithms to solve practice problems to the very first analysis of John von
Neumann about merge sort.

History of Computer Science and Deep Learning

While reading the book, I had the feeling that the period of 1950-1975 perhaps
exploded into computer science research, which is pretty much similar to Deep
Learning nowadays. That was the time we there are few universities opened
Computer Science department and people still debated about the name of this
science that whether its name is “Computer Science”, “Information Technology”,
or “Information Processing”. That is also the time when people thought that
the problem which can only solve on $\mathcal{0}(N^2)$ actually can be solved
in $\matchcal{O}(N \lg N)$, the time when there is not any mathematical tools
for analyzing algorithms.

How about Deep Learning?

Nowadays, people remain skeptical about Deep Learning in many aspects. Somebody
compared it to alchemy because of the lack of rigorous analysis and mathematical fundamentals.
However, according to the currect development of Deep Learning and avoiding
the media hype, I am confidient about the future of Deep Learning. At least,
in my opinion, it will become an interdiscipline and completely transform
into “chemistry”. In retrospect, perhaps Computer Science was compared to
“alchemy” in the period of 1940-50 due to its lack of rigor.

Study about History of Computer Science

I was fascinated by the analysises in detail about “ancient” algorithms as well
as the elaborate way to discuss the draft of merge sort from John von Neumann.
IMO, this really is the way people should study about the history of Computer
Science. It is not about memorizing who and when algorithms were created, it
is about the motivation and approach which the inventor tackled the problem
as well as analyzing the methods in the circumstances which the technology is
limited. Through these insights, we will respect the contribution from algorithms
and their authors. In addition, we can gain more methods, approached for another
problems.

Don Knuth is especially interested in this particular subject, there are
many videos and documents in which he discussed exhaustively: