Celebrity doctor feared he would lose everything after staff