I find that easy to believe. I’m a very minor genius, hardly enough of one to be worthy of the name. Even that is enough to cause me regular problems in spite of my ever-present mammalian human nature along with the rest of the people on the planet.

Like what? I have a communication issues with people a lot of the time. I make assumptions about intelligence that just don’t work. I assume people are well-read. I assume I’m well-read. Even when I correct myself ,I find there are issues with communicating, and these are other human beings I’m trying to communicate with.

AIs will have far less in common with any of us than I ever had on my worst days, or even the geniuses who design them. I get the ideas of sex, going to the bathroom, eating. I get it viscerally, and so do the rest of us humans on the planet. AIs can’t. They can extrapolate it, make notes about it, put it on their nanosecond calendars as to why someone is talking to is taking a huge vacation. They can’t truly get it, the same way we humans can’t really get what being a slug is like.

Communications problems kill people daily. The bigger the communication problem, the more dangerous and deadly.

So in my human opinion, even the most benign AIs are likely to be dangerous and deadly to us simply because they’re not human and could only communicate to us in ways limited by their lack of humanity.

Does that mean I think we shouldn’t do it? No. I hope I’m wrong, or AIs are kind enough to treat us well. Since we’re working to create AIs anyway, I may live to find out. It will be interesting.