Superintelligence: Paths, Dangers, Strategies
Nick Bostrom
Oxford University Press (September 2014)

This is what could be “the most important and most daunting challenge humanity has ever faced”

Note: Ten years after publication, the insights in this book are even more valuable than they were then. Bravo!

John H. Flavell was probably the first to use the term metacognition when suggesting that it “refers to one’s knowledge concerning one’s own cognitive processes or anything related to them, e.g., the learning-relevant properties of information or data. For example, I am engaging in metacognition if I notice that I am having more trouble learning A than B; if it strikes me that I should double-check C before accepting it as fact.” That was in 1976.

As I began to read Nick Bostrom’s brilliant book, I was again reminded of Flavell’s research. What does the term “superintelligence” mean? According to Bostrom, “We can tentatively define a superintelligence as any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest.” Bostrom focuses on three different forms of superintelligence and asserts that they are essentially equivalent: Speed superintelligence, a system that can do all that a human intellect can do, but much faster; Collective superintelligence, a system composed of a large number of smaller intellects such that the system’s overall performance across many very general domains vastly outstrips that of any current cognitive system; and Quality superintelligence, a system that is at least as fast as a human mind and vastly qualitatively smarter.

He could well be right that the development of superintelligence – by human beings — could be “quite possibly the most important and most daunting challenge humanity has ever faced. And – whether we succeed or fail – it is probably the last challenge we will ever face.” To his credit, he duly acknowledges the possibility that many of the points made in the book could be wrong.” It is also likely that there are considerations of critical importance that I fail to take into full account, thereby invalidating some or all of my conclusions.

These are among the several dozen passages of greatest interest and value to me, also listed to suggest the scope of Bostrom’s coverage in Chapters 1-7

o Seasons of hope and despair (Pages 5-11)
o Opinions about the future of machine intelligence (18-21)
o Artificial intelligence (23-30)
o Whole brain emulation (30-36)
o Biological cognition (36-44)

o Brain-computer interfaces (44-48)
o Forms of Superintelligence (52-57)
o Recalcitrance (66-73)
o Will the forerunner get a decisive strategic advantage (79-82)
o From decisive strategic advantage to singleton (87-90)

o Functionalities and superpowers (92-95)
o An AI takeover scenario (95-99)
o The relation between intelligence and motivation (105-108)
o Instrumental convergence (109-114)

I commend Bostrom on his skillful use of reader-friendly devices such as Figures (14), Tables (13), Boxes (13), summaries and synopses, and extensively annotated notes (Pages 261-304). These devices will facilitate, indeed expedite frequent review of key material later.

As of now, today, no one knows with certainty to what extent (if any) superintelligence will eventually be able to do everything that human intellect can do…and do it better and faster. Humans design systems and, as Bostrom suggests, humans are beginning to design systems that can also design systems. My own crystal ball imploded long ago so I have no predictions to offer. I do have a few articles of faith that I presume to share now. First, I believe that instruments of artificial intelligence (AI) will never replace human beings but, over time, they will become increasingly more valuable collaborators insofar as the whats and hows are concerned. Second, I believe that human beings will always be much better qualified to rank priorities and determine the whys.

Finally, and of greatest importance to me, I believe that only human beings possess a soul that can be nourished by “a compassionate and jubilant use of humanity’s cosmic endowment.”