An LLM is a large and complex machine, not a screwdriver. Large and complex [physical] machines are built with safeguards to prevent misuse, injury, etc by regulation.
LLM's are in principle text in / text out machines. If the user extends its capability to have agency over a production database or a machine, there's nothing that can safeguard the safety.
Imagine I ask an LLM to instruct left/right/speed up/slow down while driving. I can simply bypass any safeguard by stating i suddenly became blind while driving a car. While in fact i'm blindfolded and doing an experiment on a highway.
Funny enough Omaha, Nebraska does this more or less citywide. Aves are always West of their corresponding Street, East of the next highest Street, and some of the subdivisions out west get real whacky with things like 178th ave/ct/place/lane/drive/road/trail/plaza/terrace/...
Pretty common here in Chicago, the unrelenting grid makes it pretty easy to just tell someone what major cross streets to target, or directions from an L stop.
Chicago does pretty good in general, the intersection of North Western Ave and West North Ave is always the most obvious, it's easy to get Central Ave and Central Park Ave mixed up too, they run parallel to each other and they're both pretty far west. There's also Lawrence and St. Lawrence, but they're like 25 miles apart so nobody really gets them mixed up.
Two of them have somehow crashed into bus shelters, smashing the glass, most recently on Lawrence iirc. That's good enough reason for me. I use those bus shelters, especially on Lawrence!
reply