I hear that on days from people who are answering accusations of sin.
Two responses to the reply come to my mind. One is that when John saw Jesus revealed for the first time ever as the Judge of the living and the dead, he saw Jesus sit, and the terror invoked by the sight was so fierce that John said Earth and Heaven tried to flee but could not (Revelation 20:11). As David said in response to a judgment by Yahweh, it is better to fall into the hands of Yahweh than the hands of men (2 Samuel 24:14). Also, though, as David said in the same breath, he was very distressed to even fall into Yahweh’s hands in judgment. To be boasting in making the statement that only God can judge us is a bit out of place. Terror would be more appropriate for an arrogant sinner.
The second response to the statement is based on the fact that Jesus has begun to build His ekklesia in the earth. He is giving His servants representative authority as He calls them together as ekklesia. He has all authority in Heaven and in Earth. Paul pointed out that believers no longer need the courts of human governments to make judgments for them, because believers are called to judge earth and heaven (1 Corinthians 6:1-11).
The city clerk who broke up the riot in Ephesus (Acts 19) used the word ekklesia several times to describe the place where the riot was occurring (illegal ekklesia) and the place where the rioters should take Paul and his disciples to have them judged (legal ekklesia).
Jesus said in His teaching about recovering a brother who has sinned (Matthew 18:15-17) that the final judgment for the matter should be made by ekklesia.