Jayden Bogle thought he had earned the Blades a point with a drilled equaliser after 85 minutes, but Spain midfielder Rodri blasted into the top left corner three minutes later to give City the win.
Erling Haaland earlier missed a first-half penalty before heading the opener from Jack Grealish's cross just after the hour mark.
The win was Pep Guardiola's 200th Premier League win as City manager in just his 269th game, the quickest any boss has reached 200 wins in English top-flight history.
Although he was absent from the touchline, recovering from back surgery, Guardiola has been pulling the strings from afar in recent days.
This result also marked the first time City have won the first three games of a campaign since the Spaniard's first season as manager in 2016-17.
City leave it late to snatch points
City, with assistant boss Juanma Lillo giving orders, deserved to take all three points after dominating possession and creating the better chances.
However, they very nearly paid for unusual profligacy in front of goal.
Haaland spurned a chance to put City ahead when his first-half penalty hit the inside of the right post and bounced to safety, though he made no mistake later when converting Grealish's inviting cross from the left.
Bramall Lane erupted when United capitalised on a mistake by former Blades defender Kyle Walker to equalise through Bogle, but City responded almost immediately.
Walker atoned for his error by breaking into space down the right and putting in a dangerous cross that hit substitute Phil Foden before landing in the path of Rodri, who fired in a venomous shot.
Foden had been dropped despite his fine performance in the win over Newcastle, but his late introduction immediately after Bogle's leveller gave City much-needed impetus.
More to follow.