Adam Stansfield opened the scoring for Exeter early in the second half in front of almost 4,000 fans.
But Grayson deservedly pulled a goal back and as the game headed for a draw he volleyed home in the third minute of injury time.
It was no more than his side deserved after a fine second-half performance.
Exeter were denied by a string of fine saves from Danny Alcock and also lacked a cutting edge in front of goal.
The manner of the defeat was tough on the Grecians, but Rangers are now starting to show that they can perform on any stage in the Conference.
Exeter entered the game on a fine run of eight games unbeaten and remained undefeated at home so far this season.
The Grecians started brightly and set out their intent from the start. Jamie Mackie was tripped on the edge of the box and Billy Jones curled the ball into the arms of the goalkeeper from the resulting free kick.
At the other end Kevin Street struck a sweet volley at Martin Rice from 18 yards.
Stafford could easily have gone ahead when City failed to clear Craig Talbott's free-kick and Ged Murphy forced Rice to tip the ball on to the crossbar with a shot from six yards.
Exeter have gained a reputation for starting games slowly at St James Park this season but this clash was an open affair from the start and the chances continued to come at both ends.
Stansfield forced Alcock to save at full-stretch with a low shot from the left side of the penalty box and Mackie had a shot charged down as the Grecians pushed forward.
Exeter began to dominate as the half worn out but failed to break the deadlock.
Paul Buckle's long-range effort was saved by Alcock at full-stretch and Challinor flashed a shot past the post from a tight angle.
Stafford offered little as an attacking force, but Andy Taylor needed to make a fine blocking tackle as striker Guy Madjo used his strength to twist and turn inside the City box.
After the break Exeter continued to dominate and it only seemed a matter of time before they took the lead.
Taylor hammered a free-kick into the wall and drilled the rebound at the goalkeeper after Challinor was brought down by Richard Sutton just outside the box.
Exeter forced a string of corners of eventually broke the deadlock ten minutes after the restart.
Taylor's long corner struck defender Sutton and when the ball fell to Stansfield he stabbed it home from close range.
But instead of pushing on to try and put the result beyond doubt, Exeter allowed the visitors to seize the initiative.
Wayne Daniel's corner was met by Street and his header flew just wide.
Madjo then thumped a superb volley against the underside of the crossbar from 15 yard with a powerful drive that had Rice well beaten.
Murphy had a deflected shot well saved by Rice before Stafford were finally level in the 70th minute.
Substitute Danny Edwards picked out Grayson with a pinpoint cross and the striker made no mistake with a powerful header.
It was his 50th goal for the club and was well deserved for his side's fine second-half performance.
Stafford boss Phil Robinson was sent off for holding on to the ball when it flew into the dugout.
But deep into injury time Street was allowed to cross the ball from the left and Grayson reacted first to volley past the keeper from close range.