Singh won't break pact with Liberals despite concern PM isn'