00:01I'mgoingtotalkaboutafailureofintuitionthatmanyofussufferfrom.It'sreallyafailuretodetectacertainkindofdanger.I'mgoingtodescribeascenariothatIthinkisbothterrifyingandlikelytooccur,andthat'snotagoodcombination,asitturnsout.Andyetratherthanbescared,mostofyouwillfeelthatwhatI'mtalkingaboutiskindofcool.00:25I'mgoingtodescribehowthegainswemakeinartificialintelligencecouldultimatelydestroyus.Andinfact,Ithinkit'sverydifficulttoseehowtheywon'tdestroyusorinspireustodestroyourselves.Andyetifyou'reanythinglikeme,you'llfindthatit'sfuntothinkaboutthesethings.Andthatresponseispartoftheproblem.OK?Thatresponseshouldworryyou.AndifIweretoconvinceyouinthistalkthatwewerelikelytosufferaglobalfamine,eitherbecauseofclimatechangeorsomeothercatastrophe,andthatyourgrandchildren,ortheirgrandchildren,areverylikelytolivelikethis,youwouldn'tthink,"Interesting.IlikethisTEDTalk."01:09Famineisn'tfun.Deathbysciencefiction,ontheotherhand,isfun,andoneofthethingsthatworriesmemostaboutthedevelopmentofAIatthispointisthatweseemunabletomarshalanappropriateemotionalresponsetothedangersthatlieahead.Iamunabletomarshalthisresponse,andI'mgivingthistalk.01:30It'sasthoughwestandbeforetwodoors.Behinddoornumberone,westopmakingprogressinbuildingintelligentmachines.Ourcomputerhardwareandsoftwarejuststopsgettingbetterforsomereason.Nowtakeamomenttoconsiderwhythismighthappen.Imean,givenhowvaluableintelligenceandautomationare,wewillcontinuetoimproveourtechnologyifweareatallableto.Whatcouldstopusfromdoingthis?Afull-scalenuclearwar?Aglobalpandemic?Anasteroidimpact?JustinBieberbecomingpresidentoftheUnitedStates?02:08(Laughter)02:12Thepointis,somethingwouldhavetodestroycivilizationasweknowit.Youhavetoimaginehowbaditwouldhavetobetopreventusfrommakingimprovementsinourtechnologypermanently,generationaftergeneration.Almostbydefinition,thisistheworstthingthat'severhappenedinhumanhistory.02:32Sotheonlyalternative,andthisiswhatliesbehinddoornumbertwo,isthatwecontinuetoimproveourintelligentmachinesyearafteryearafteryear.Atacertainpoint,wewillbuildmachinesthataresmarterthanweare,andoncewehavemachinesthataresmarterthanweare,theywillbegintoimprovethemselves.AndthenweriskwhatthemathematicianIJGoodcalledan"intelligenceexplosion,"thattheprocesscouldgetawayfromus.02:58Now,thisisoftencaricatured,asIhavehere,asafearthatarmiesofmaliciousrobotswillattackus.Butthatisn'tthemostlikelyscenario.It'snotthatourmachineswillbecomespontaneouslymalevolent.Theconcernisreallythatwewillbuildmachinesthataresomuchmorecompetentthanwearethattheslightestdivergencebetweentheirgoalsandourowncoulddestroyus.03:23Justthinkabouthowwerelatetoants.Wedon'thatethem.Wedon'tgooutofourwaytoharmthem.Infact,sometimeswetakepainsnottoharmthem.Westepoverthemonthesidewalk.Butwhenevertheirpresenceseriouslyconflictswithoneofourgoals,let'ssaywhenconstructingabuildinglikethisone,weannihilatethemwithoutaqualm.Theconcernisthatwewillonedaybuildmachinesthat,whetherthey'reconsciousornot,couldtreatuswithsimilardisregard.03:53Now,Isuspectthisseemsfar-fetchedtomanyofyou.IbettherearethoseofyouwhodoubtthatsuperintelligentAIispossible,muchlessinevitable.Butthenyoumustfindsomethingwrongwithoneofthefollowingassumptions.Andthereareonlythreeofthem.04:11Intelligenceisamatterofinformationprocessinginphysicalsystems.Actually,thisisalittlebitmorethananassumption.Wehavealreadybuiltnarrowintelligenceintoou...
