Password Plus' only two ratings' issues in it's entire run have nothing to do with Tom Kennedy or celebrities. Both involve time period moves, per the data.
In July 1980, the show was doing a 4.8/18 @ 78% clearance at 12:30PM (beating Card Sharks almost 2:1, and being almost as strong vs. Wheel). The show had been hovering in the mid 4's since Fall 1979, with P+ being THE beneficiary of the schedule shuffle that ultimately started Hollywood Squares' downfall and brought about Mindreaders.
NBC shuffled their daytime around in August 1980, a couple of months before Tom debuted, moving The Doctors to 12:30p in the same schedule change that saw Letterman's daytime show and Another World condensed to 1 hour to accommodate the debuting Texas. Password went to 11:30AM, and while the 90 minute game block slid up a half hour, Password and Card Sharks also traded positions, which wound up hurting P+ and made no difference for Card Sharks. Even though clearances went up from 78% to 90% of affiliates, the show's rating immediately dropped to a 3.7 (with the share holding flat at 18) for the first available data period (still Ludden).
Password then held on to a rating somewhere between 3.5 and 4.0 for the first year (including the Allen to Tom change) until they moved it to Noon in October '81, losing about a third of it's viewers for the first month...before going right back to the same mid-high three (even with about 20% fewer clearances) that it then held onto right up until the day it was cancelled. +
Both Password Plus and Super Password's numbers were stunningly consistent over their runs, only ever changing when viewers found the show (it took everyone six months to realize Super Password existed, and Password Plus absolutely siphoned eyeballs from the former occupant of a timeslot, Hollywood Squares) or they moved it to a new slot. In general, if a viewer liked Password as a game beyond the initial sampling, they made a point to find it and watch it no matter who hosted or what time it aired (as long as it aired in their market).