"If we say, 'From heaven,' He will say to us, 'Why then did you not believe him?'" (Matt. 21:25). Specifically, Jesus would have said - "Why didn't you believe what he said about Me? You admit that John's message was from heaven and he affirmed that I am the Messiah! What more authority do I need?" Obviously, the religious leaders do no believe this; thus, they will not confess John's baptism to be "from heaven." They will not acknowledge that John' ministry was authorized by God because of what John declared about Jesus. Tragically, their reasoning among themselves seems to be primarily focused upon giving the "best answer" (i.e., the most expedient one), not the truthful one.
"But if we say, 'From men,' we fear the multitude, for all count John as a prophet" (Matt. 21:26). Jesus has placed them in a dilemma. Either way they answer His question, they are going to lose. If they say, "From heaven," then Jesus will win the argument. However, if they say, "From men," they will lose to the multitude. This is the case because the Jews almost universally accepted John as a prophet. They want to build their popularity with the crowd, not destroy it completely. Plus, they also feared that the crowd would stone them if they denied that John the baptizer was a prophet (cf. Luke 20:6). So, although they do not believe John to be a prophet, they do not want to publicly say so at this time.
Thus, they feel forced to answer - "We do not know." What a pitiful answer they end up offering to Jesus! It is a lie for they certainly did have an answer to His question, but they were just too afraid to speak it! This answer, though pathetic, was the best they could do to avoid offending the multitude and still not admit absolute defeat to the One they hated. Of course, this answer was still a strong victory for Jesus. These teachers were supposed to be the guides of the people prepared to answer any religious question, yet now they humiliate themselves by saying they don't know the answer to a simple question! Certainly they were embarrassed on this occasion by Jesus and would have been made all the more bitter toward Him.
Jesus replied to their pathetic answer - "Neither will I tell you by what authority I do these things" (Matt. 21:27). Jesus exposed their hypocrisy by stating that, He--just like them--would not answer the question asked. He knew that they could answer His question, but they simply would not. They had intentionally turned His question aside unanswered. Thus, He felt no obligation to explicitly answer their question (though His reference to John the baptizer was an indirect answer since John testified that Jesus was the Messiah).
Certainly the multitude would see Jesus' words here as fair. They would think: "Jesus shouldn't have to answer the question of the religious leaders since they won't answer a simple question that everyone knows the answer to!"