"Levi" is another name for the apostle Matthew. He invited his old friends to this feast. Luke describes this occasion as a "great feast" though Matthew modestly omits that detail from his own account (cf. Matt. 9:10).
The religious elite posed the question - "Why do You eat and drink with tax collectors and sinners?" (cf. Luke 15:2). Matthew 9:11 indicates that the question was about Jesus, yet it was spoken to the disciples. This would have been a legitimate question to ask Jesus, but only a coward or a troublemaker would ask His disciples about His actions. It was the custom of Jews not to eat or associate with Gentiles (e.g., Acts 10:28) in order to avoid being defiled (i.e., by coming in contact with those who were ceremonially unclean). Tax collectors were often considered to be both traitors and heathens because they worked for the Roman government (i.e., Gentiles). Thus, the religious leaders want to know why Jesus associates with such "terrible" people--people who were known for neglecting the law and being unclean. One should not presume that the Pharisees were necessarily present at the feast itself, but they were surely aware of such a large gathering and had some opportunity to ask His disciples this question.
Jesus answered the question for His disciples by stating - "Those who are well have no need of a physician, but those who are sick" (Luke 5:31). The Pharisees considered His behavior to be reckless, but Jesus easily defended His actions and influence. He began by comparing Himself to a physical doctor. As a physical doctor must have contact with the physically sick in order to heal them, likewise Jesus must have contact with the spiritually sick (i.e., "sinners") in order to heal them (of their sins). There are many interesting parallels between a good physician and Jesus. Both are knowledgeable, compassionate, and easily accessible. Both keep good records and neither makes mistakes. One might even take this analogy a bit further by considering the characteristics of a good patient (both physically and spiritually). A good patient goes to the doctor when sick, follows the doctor's orders, returns to the doctor when necessary, and recommends the doctor to others.
Matthew's account of this event includes more information - "But go and learn what this means: 'I desire mercy and not sacrifice'" (Matt. 9:13; Hos. 6:6). This is an example of a figure of speech sometimes called a Hebraism. It should not be interpreted in a strictly literal fashion. Had these religious leaders understood Hosea, they would not have questioned the appropriateness of Jesus' actions toward "sinners." Jesus' meaning here is this: if a person's attitude is wrong towards others (i.e., he is without mercy), then God is not pleased with him regardless of what great sacrifices he may offer Him. Jesus did not teach (and Hosea did not teach) that God wants mercy only and no sacrifice (note the first portion of John 6:27 for another example of this type of figure of speech). The Lord wants both mercy and sacrifice but is here emphasizing that sacrifice, even to the extent of the Pharisees, is worthless without a proper attitude. The Pharisees considered their active avoidance of tax collectors and sinners to be a praiseworthy "sacrifice" or service rendered to God. However, God would have preferred that they show mercy and compassion to such lost souls by trying to restore them instead of avoiding them and criticizing Jesus' actions.
"For I did not come to call the righteous, but sinners, to repentance" (Matt. 9:13). Since Jesus came to accomplish this purpose, then it is surely the case that He must go to the sinners, as He was doing here. Clearly Jesus used the word "righteous" here in reference to the religious leaders. However, by using this word He was not implying that the Pharisees really were righteous. He merely referred to them as they considered themselves, which a simple consideration of the wording of their original question shows. Those who do not first see themselves as sinners are not ready to come to repentance just as those who don't consider themselves to be sick will not go to the doctor. Jesus didn't come to "heal" those who thought they were already well!
Let it be noted that none of the arguments Jesus makes here justify any Christian in keeping company with "sinners" for any other purpose than to do them good; that is, to help guide them out of sin. Jesus' purpose in being with these "sinners" was not to make friends or socialize. His association with them was for the purpose of leading them to repentance.