AI singularity, and the transmission of intelligence

Question

I was quite taken with David Deutsch's dismissal of the fear of the AI general intelligence singularity, based on the idea that any true intelligences will in some sense start with hybrid human-machine intelligences. I pictured human-machine cyborgs, or simulated human brains like the Human Brain Project is attempting.

But I just encountered Eliezer Yudkowsky's strong dismissal of this view:

• I don't think that humans and machines "merging" is a likely source for the first superhuman intelligences. It took a century after the first cars before we could even begin to put a robotic exoskeleton on a horse, and a real car would still be faster than that.

• I don't expect the first strong AIs to be based on algorithms discovered by way of neuroscience any more than the first airplanes looked like birds.

• I don't think that nano-info-bio "convergence" is probable, inevitable, well-defined, or desirable.

• I think extrapolating a Moore's Law graph of technological progress past the point where you say it predicts smarter-than-human AI is just plain weird. Smarter-than-human AI breaks your graphs.

• The only key technological threshold I care about is the one where AI, which is to say AI software, becomes capable of strong self-improvement. We have no graph of progress toward this threshold and no idea where it lies (except that it should not be high above the human level because humans can do computer science), so it can't be timed by a graph, nor known to be near, nor known to be far. (Ignorance implies a wide credibility interval, not being certain that something is far away.)

• I think outcomes are not good by default - I think outcomes can be made good, but this will require hard work that key actors may not have immediate incentives to do. Telling people that we're on a default trajectory to great and wonderful times is false. From

This throws the spotlight for me, on what would is involved in transmission of human intelligence, between individuals and to any AI. It has to be transmission in some sense. For instance with https://en.m.wikipedia.org/wiki/AlphaZero even though it was able to adapt from Go to Chess, this involves some intelligence about game design and aims. A container for calculations, in the same way that a human baby's brain and it's engagement with energy flows is a container, which must develop considerably to redefine the game.

Is there anything beyond speculation, about how much a true general AI intelligence would need language, training, and cultural transmission, or whether it could somehow become 'self-arising' - could it in some sense become an entire new evolutionary tree, developing itself as evolution developed us?

Perhaps there is a parallel issue about how much descendents resemble antecedents, how intelligence is preserved and transmitted, and how much we can be self-defining. There seems a tension between individual and community intelligence..

Answer 1

A true general intelligence needs information which may be included in itself when created or it may receive this from data it analyzes. It must have information but it may receive basic principles when it was created so it can start to self improve. In other words, the algorithm itself can contain the needed information for a machine to self improve.

Intelligence is transmitted as information. It depends on the actual implementation how that happens. It is like saying it depends on the language of the machine.

Answer 2

This seems related to the nature vs. nurture debate for human beings -- does our behavior come from our genes (nature) or the environment (nurture)? Clearly both play a role, but their relative importance remains an issue of debate. One might further divide nurture into nurture from human culture (direct instruction from parents and others, observing what other humans do, reading books, etc.) and other (learning to move your body, playing with sand to learn about physics, etc.). One can ask the same question for other animals.

For the first artificial general intelligence (AGI), we can frame the question the same way -- how much of its behavior will be directly encoded (nature), how much will result from learning from human beings through teaching and demonstration (nurture/culture), and how much from its own observations of and experiments on the physical world (nurture/other). I don't know how to answer this question exactly. But as far as current AI research goes, it has become much more learning (nurture) oriented than it was at one point, and the idea of learning from human demonstration is also very popular, especially in robotics (as a quick web search will reveal).

Answer 3

"When Kasparov was defeated back in 1997, he didn’t give up the game. A year later, he returned to competitive play with a new format: advanced, or centaur, chess. In advanced chess, humans partner, rather than compete, with machines. And it rapidly became clear that something very interesting resulted from this approach. While even a mid-level chess computer can today wipe the floor with most grandmasters, an average player paired with an average computer is capable of beating the most sophisticated supercomputer – and the play that results from this combination of ways of thinking has revolutionised the game." https://www.theguardian.com/books/2018/jun/15/rise-of-the-machines-has-technology-evolved-beyond-our-control-