AI 灾难概率评估现实性分析

Emad (@EMostaque) 转发了 6529 (@punk6529) 的帖子： 1/ On p (doom) tl;dr a) Everyone is making up the numbers b) nobody knows anything (least of all the experts), c) don't worry about it d) there is nothing you can do to stop it e) most things you can do if you "worry" will end up being deeply counter-productive to your own interests f) on the margin, there are a couple of things you may consider 2/ p (doom) is the probability that the AGIs/ASIs kill us all People in the AI space have a lot of estimates about this, ranging from very low to absolute certainty that it is going to happen. You see them quoted in the news like this "Famed AI safety researcher / economist thinks that there is a 33% we are going to die but a 66% chance we will live in luxury techno-communism" 3/ I am here to reassure you that the number is definitely between 0% and 100% but other than that, all the other numbers are completely made up. 4/ True ASI is a gap in intelligence relative to us that is larger than you have from your favorite pet. Your pet may love you, may cuddle with you, may have some type of understanding that you are their "human" but has no useful views on LLMs or NFTs 5/ Likewise, if ASI is real which is what all these p (doom) scenarios assume, then a human economist's assessment (circa 2026) of p(doom) is about as useful as your cat's view on how Saylor is going to wiggle himself out of his empire of vibe-coded financial instruments. If your analyst insists otherwise, you can just tell them "well, it seems like you are not a true believer in ASI, if you think you can model an uber-intelligence of 600 IQ with your puny 140 IQ" 6/ Separately, people have estimates about which models may be dangerous to bring into the world. This is actually more useful because it allows you to see how well predictions age. Famously, Dario thought GPT-2 was "too dangerous" to release to the world and the OpenAI Board got all worked about the safety of GPT-4 class models and fired/unfired Sam broadly, putatively for that reason. I have no doubt these are all very smart people who know a lot about AI and mean very well, but obviously in retrospect that idea that GPT-2 and GPT-4 were some super danger to the world was, well, "extremely not true" 7/ Again, this is not a criticism. I sympathize with the people who are ushering in perhaps the most important technology in human history. It is a large responsibility, obviously risk is not zero, and they are naturally at some level worried at a personal, human level about the consequences if things go wrong. I am not sure I would want that level of responsibility. And because they are mathematically oriented people, one way to deal with this anxiety is to try to tame it with numbers and formulas. It is a math whiz's way to get comfortable with uncertainty, a form of coping mechanism. It is fine, just don't take any of the specifics seriously. 8/ I have been very tech-forward in general for a long time, I use $50K of inference each month and so naturally people ask me about AI. Mostly they ask me "6529, aren't you worried?" My answer is: "and what good would it do if I was worried?" Will Xi get a report that I am not sleeping so well in my (air-conditioned, thank you very much) southern European bedroom and change China's industrial strategy? Will Elon notice that I am stressed and cancel the data centers in space? No, none of this will happen. All that will happen is that I will be less happy, less productive and less of a contributor to my friends, family and society at large. 8/ This is the most important societal technology in human history, for both economic and military purposes. Nobody is going to stop working on it, nobody can risk stopping unilaterally and the dynamics of testing/verifying make it hard / impossible to enforce a global ban. 9/ If you get worried, you will do many things that are counter-productive to your own interests, such as: a) lobbying against data centers in your country b) lobbying for regulations to limit AI in your country c) not using AI yourself in your life 10/ By analogy, will be doing the following: It is 1850, some European has shown up in your country with a gun and you are still at the "bow and arrow" stage of military technology. You naturally get alarmed by guns (they suck and are scary) and decide to: a) ban production of guns in your country, b) ban Europeans from selling you guns and c) ban yourself from learning how to use a gun. 11/ Obviously all of this is deeply against your interests, but very welcome by the European with a gun. I remember when GPT-4 came out and the Italian regulator threatened to ban OpenAI from selling it in Italy. And all I could think was "FFS, your real risk is that the US government will ban OpenAI from selling you GPT 8; you should be lobbying for an access agreement, not a banning agreement" Turns out I was wrong and this has happened with GPT 5.6 12/ If you think that ASI/AGI is scary (like guns) and your geopolitical opponents have ASI/AGI, I regret to inform you that the correct move is to learn as quickly as possible how to use, deploy and ideally make your own ASI (the latter now out of reach for most countries) 13/ "6529, this is defeatist, if the people just unite, we will elect officials that will put ASI in a proper regulatory framework" Yes, we will build on humanity's long-standing success in getting everyone to turn in their knives, guns, tanks, fighter jets, ballistic missiles, drones and nuclear weapons to the 3rd grade schoolteacher by recess. You go ahead and let the Pentagon and PLA to turn in their AIs 14/ Also, since we mentioned regulation, your cat and your dog and your goldfish are pleased to inform you that they have developed a new regulatory framework for DeFi and look forward to Uniswap coming under their control and jurisdiction 15/ "Wait, do you think there should be no regulations on AI" - no, I have no doubt the Pentagon and NSA will have plenty of discussions with the frontier model providers on a variety of topics above my pay grade. What I am saying is that you specifically are unlikely to have the security clearance to productively contribute to those discussions, unless that you are currently involved in fighting nuclear proliferation and bioweapons. 16/ What I mean is this: I see that >50% of people in Europe and some very large % elsewhere are going to read these p (doom) type essays and also a bunch of gross mathematical errors about how much water data centers use or whatever and conclude that they specifically should not use AI and they should discourage their co-citizens from doing the same Which is very counter productive for them and the organizations they work for and their countries and will do literally absolutely nothing to change p (doom). 16/ You see, I don't know what p (doom) is but I can tell you what p (someone-is-going-to-compete-with-you-with-AI) is and that p is 100%. That someone may be a personal "competitor" or a "company" or a "nation-state" 17/ As with all major technology change (and if ASI is real, this is the biggest one of all), going early may be a big win, going fast follower may be ok and going late could lead to "extinction" History has not been kind to companies or countries that "missed" an industrial revolution. The EU has missed the internet altogether and is no longer a techno-economic peer to the United States. 18/ If the hard case, the doomer case of ASI is real, then no non-ASI-enabled organizations and societies will survive. It is impossible to compete without ASI with a competitor with ASI. You have to have your own and you have to believe in a world of balance of power. Should ASI be real, one sole ASI is extraordinarily dangerous to the world. A vast bazaar of ASIs may possibly generate some balance of power. 19/ We will find all this out in time, but in any case, it is hard to imagine that it is "your" personal / corporate / national interest to be the one without AI capabilities in that world. The very best case is that you will be a charity case in that world of gods-on-earth. The worst case is you will get a sympathetic paragraph in the future history books about how it was not very neighborly when the drone swarm disabled your national infrastructure and you had to accept very unpleasant terms to keep the lights on. 20/ I think my view is probably closest to OpenAI's view if I had to pick - that models should be deployed and diffused often, giving society the change to absorb change incrementally and as broadly as possible. And the open-source models are a good check on the ecoysystem. 21/ So, what can you do personally? Two things: a) Learn and use, as discussed above. It can only help you personally, there is no chance it will hurt you personally b) the number #1 medium-term risk from AGI/ASI is an engineered respiratory virus. that is this weekend's thread. hardening your life systems for that scenario is probably helpful on the margin 22/ tl;dr we are going to live the most interesting time in human history together, it is going to be a wild ride, it is better to be a proactive participant, and given all the above, you might as well decide to enjoy the ride 🫡😀