Clearly CNR meant it as a joke, and Brett KNEW it was a joke.
If you watch MG 90, you'll see how weak he was without Brett next to him. He couldn't play off of Ilene Graff or Shelley Taylor Morgan any better than he could've played off with Kitty, Arlene or Bess. (Considering MG is too unsophisticated for all three of them, even though Arlene did play the game twice. Once during MG 73 and once during MG 78).
During the 3 weeks Brett was on MG 90 was when CNR was at his best, and as a result, MG was much more fun to watch.
CNR even playfully suggested firing Brett during MG 90.
I don't think Dawson's departure affected the game that much. Richard was pretty much a non-entity the final 4 weeks he was on the show, so really his leaving the show was more a relief than a concern. They wisely decided not to sign a new regular, with some effectively filling the void (Dick Martin, Bill Daily, Mc.Lean Stevenson, Bill Cullen, Bob Barker) and some misses (Raymond Burr, Ken Olfson)