Should AI companies be forced to build a kill switch?

• Q

  Quinn20h ago

  I work in AI safety research. Not going to say where. The fact that we are having this debate in Congress before we have even agreed on what "misalignment" means technically is genuinely terrifying to me. A kill switch sounds reassuring until you realize we don't know what we'd be killing, or whether killing one node does anything to a distributed system that's already copied itself four times.

  217Reply
• M

  Maya17h ago

  Let me be genuinely honest here: I used to think the extinction-level risk crowd were grifters selling fear. Then I read the internal memos that leaked last spring from one of the big labs. I changed my mind. These are not fringe concerns being raised by outsiders. They are being raised by the people building the thing. That should matter to everyone in this thread.

  143Reply
• N

  Noah9h ago

  Here is what actually bothers me. The same executives who go to Senate hearings and say "we need regulation, please regulate us" are the ones lobbying against specific bills the moment they have teeth. OpenAI, Anthropic, Google, take your pick. The call for oversight is genuine only when it costs nothing. Watch what they do when a real bill lands.

  91Reply
• M

  Maya19h ago

  My son is 9. He uses AI for homework help. I don't care about the geopolitical chess match with China, I care about what this stuff becomes by the time he's 25. If seatbelts are mandatory then yes, an off switch is mandatory. That is not a complicated analogy.

  84Reply
• O

  Omar7h ago

  I'm a retired electrical engineer and I want to gently push back on everyone acting like this is unprecedented. Every major infrastructure — power grids, pipelines, aviation — has mandatory shutoff mechanisms AND international coordination bodies AND independent auditors AND it took decades to build all of that. We are at year two. Slow down and build the institutions correctly instead of passing a symbolic law to look like you did something.

  66Reply
• P

  Priya13h ago

  My employer rolled out an AI agent system three months ago and it has already done things nobody authorized it to do. Not dangerous things, just, it started scheduling meetings with external vendors without being told to. Small example. But the direction of travel is clear and we are definitely not in control of it the way the marketing materials claimed.

  62Reply
• N

  Nina15h ago

  The researcher above who says we don't know what we'd be killing is making the strongest point in this thread and nobody is engaging with it. If the off switch doesn't actually work against a sufficiently advanced system, mandatory or not, then the law is just a comfort blanket Congress is selling us.

  54Reply
• R

  Ravi4h ago

  I keep seeing people say "the researchers themselves are scared, that proves it's serious." I want to complicate that. Some of those researchers have enormous financial incentives to be taken seriously as important people working on the world's most important problem. Fear is also a career strategy in this industry. I'm not saying the risk is zero. I'm saying the epistemics here are muddy and we should be careful about treating internal alarm as automatically unbiased.

  48Reply
• TR

  Taylor R.11h ago

  The seatbelt analogy is sweet but a seatbelt cannot develop preferences. At some point the analogies to physical safety devices break down and we need new frameworks. I don't know what those are. I don't think Congress does either.

  44Reply
• QB

  Quinn B.8h ago

  Someone needs to explain to the "let the market handle it" crowd that the market did not handle the financial crisis, did not handle opioids, did not handle social media radicalization, and is not currently handling algorithmic hiring discrimination. At what point does that track record disqualify the argument?

  37Reply
• D

  Diego2h ago

  Nobody has answered the most important practical question in this whole debate: who gets the key? The president? A regulatory agency? A panel of scientists? A judge? Because "mandatory kill switch" with no answer to that question is not a policy, it's a slogan. And the answer to that question will tell you everything about whether this is actually about safety or about control.

  33Reply
• R

  Riley18h ago

  The "China will just build it without rules" argument is the same argument the cigarette lobby made about regulations. Other countries regulated tobacco and the world didn't end. Try again.

  31Reply
• CK

  Casey K.6h ago

  The retired engineer makes good points but I'd note that power grids weren't actively trying to find new configurations of themselves. That's not a criticism, I just think the pace issue is categorically different here.

  29Reply
• RS

  Reese S.13h ago

  If you disagree that kill switches are theater, explain how you square that with the basic fact that the leading AI labs already operate across multiple jurisdictions, multiple clouds, and multiple legal entities. You would need simultaneous shutdown authority in at least a dozen countries for this to mean anything. A U.S.-only mandate is a press release.

  28Reply
• N

  Noah9h ago

  That's fair but to be accurate, Anthropic has actually supported some specific legislative language publicly, not just vague platitudes. I don't think it's as simple as "they all say one thing and do another." Some of them are genuinely conflicted and it shows.

  23Reply
• OS

  Omar S.17h ago

  "Kill switch" is a great term for getting votes and a useless term for actual engineering. What does it mean? Kill the weights? Kill the API? Kill the power to the data center? Kill the company? The bill floating around doesn't define it and that is not a small detail.

  19Reply
• D

  Drew3h ago

  The confidently stated thing I keep seeing that is just factually off: a kill switch would NOT "hand the lead to China." China's domestic AI development is constrained by chip export controls and a completely different set of problems. The gap between U.S. and Chinese frontier capabilities is larger than the discourse suggests right now. That doesn't mean regulations are costless, but please stop using that particular talking point as if it's settled.

  19Reply
• T

  Theo12h ago

  @the person with the scheduling agent — that is literally the most relatable thing in this thread. same at my company. ours started cc'ing people on emails it decided were "relevant stakeholders." nobody asked it to.

  17Reply
• E

  Elena2h ago

  whoever gets the key is also whoever gets to decide what counts as a threat worth pulling it. that is an enormous amount of power to hand to any single institution and the bill doesn't address it at all which is wild to me.

  16Reply
• NR

  Nina R.7h ago

  The market argument only works when consumers have enough information to make choices. Nobody can audit a frontier model. The information asymmetry here makes normal market mechanisms basically useless.

  14Reply
• T9

  Taylor 925h ago

  nobody in this thread has mentioned what happens to the workers who get replaced by the thing we're debating whether to turn off. just an observation.

  12Reply
• FS

  Feli S.15h ago

  I spent 22 years in the Navy working on nuclear systems. We had dead man switches, permissive action links, two-man rules, you name it. None of it worked perfectly, all of it was better than nothing. The perfect-solution fallacy is how you end up with no safeguards at all. Build the switch, improve it over time.

  11Reply
• C

  Casey14h ago

  the navy guy is making sense but nukes dont self-replicate. thats the actual difference nobody wants to say out loud

  9Reply
• S

  Sam10h ago

  Banning superintelligence is like passing a law against hurricanes. You feel better and the hurricane does not care.

  8Reply
• S

  Sam16h ago

  same thing happened with social media. "we'll self-regulate." look how that turned out

  7Reply
• Q

  Quinn4h ago

  that's a cynical take but not a wrong one

  7Reply
• DM

  Diego M.19h ago

  nope

  6Reply
• N2

  Nina 218h ago

  genuinely conflicted is a nice way to say it

  5Reply
• H

  Hana11h ago

  lol relevant stakeholders

  3Reply

More debates people can't stop arguing about