"Not easy to give direct challenge to PM": Congress leader S