Since Toronto Mayor John Tory announced he will resign in the wake of an affair with one of his staff, a range of voices has urged him not to quit. But the advisers who've been by Tory's side the longest are remaining steadfast in their belief that he must step down, multiple sources tell CBC News.