He does not really have a reason to fight either of them (well except for the we want your bodies to experiment on them part, which Luffy does not really seem to care about much) And Caesar does not seem to be the type of guy who would go after the Straw Hats himself.