Aston Villa conceded twice from the 88th minute and suffered a 2-1 defeat at home by fourth-tier Stevenage in probably the biggest shock of the third round on Sunday.
Ahead thanks to Morgan Sanson's 33rd-minute goal, Villa were reduced to 10 men when Leander Dendoncker was sent off in the 85th for a professional foul that gave Stevenage a penalty.
- Stream on ESPN+: LaLiga, Bundesliga, more (U.S.)
That was converted and the visitors, who are in second place in League Two, grabbed an unlikely winner in the 90th.
Earlier, Leeds United rallied from a two-goal deficit against second-tier Cardiff to draw 2-2 thanks to an equaliser in the third minute of stoppage time by substitute Sonny Perkins.
It was Jesse Marsch's first taste of FA Cup competition since he moved to Elland Road last season and it was a game to remember.
Rodrigo began Leeds' recovery by pulling a goal back in the 65th before the Spain striker had a penalty saved in the 81st after Cardiff defender Joel Bagan stopped a goal-bound shot from Junior Firpo with his outstretched hand and was shown the red card.
The 18-year-old Perkins came to Rodrigo's rescue with a close-range goal to send the teams to a replay.
The last time they met in the famous competition was in 2002, when Leeds -- the Premier League leader at the time -- were stunned by a Cardiff team that was in 10th place in the third tier.
Also on Sunday, Derby advanced to the fourth round by beating fellow third-tier team Barnsley 3-0, while there were wins for second-tier teams Stoke and Blackburn.
An all-second tier match between Bristol City and Swansea finished 1-1 and Walsall were denied victory against fellow fourth-tier team Stockport by conceding late in a 1-1 draw.