The first line of Psalm 73 may have been a popular proverb at the time the Psalm was written. At the very least, it is a common theme in the Psalms. Those who are the true worshipers of God are pure in heart. In Psalm 24, for example, only those who have clean hands and a pure heart may ascend the holy hill of God (Ps 24:3-4). In Psalm 51:10 David famously asks God to create a clean heart and a right spirit within him.
But Psalm 24:5-6 goes on to say that the one who has clean hands and a pure heart will be blessed by God; they can expect that the blessings of the covenant will come their way. The converse of this would be that the one who is not pure in hear will not receive the blessings, but rather the curses of the covenant. A “pure heart” is therefore a way of describing a total commitment to God (Kidner, Psalms, 259).
This proverb reflects the covenant relationship which Israel has with God. In Deuteronomy God promised he would bless the nation when they kept the covenant and that he would punish them when they broke the covenant (curses and blessings). If a person did make a good-faith effort to keep the Law and followed the Law when they encountered impurity, then they ought to experience physical prosperity. God ought to give the good health and peace because they are “pure in heart.”
For me, I am less upset when an evil person succeeds than when a shallow, useless person succeeds. Like the Psalmist, I feel like shouting, “hey God, are you paying attention to these people? Read their twitter feeds and judge them with hellfire!”
If verse one is true on some sort of universal “proverbial” level, is it fair that a long time servant of God dies painfully with inoperable cancer when a mass murderer lives out his years in relative comfort? This is the issue the psalmist explores in Psalm 73.