Let me illustrate by using I Peter 3:3, which is one of two passages specifically addressing modesty in the New Testament.  If the pastor is preaching 1 Peter then the verses must be preached. He cannot avoid them or explain them away.  He must preach them.  But how?  Well he first exegetes the text explaining what it means. Then he must apply the sermon to both groups in his body.  Here is how that might work.

                The legalist assumes they know what modesty means. They know it means that you only wear skirts or that skirts must come below the knee or that any sleeveless shirt is a sin. The pastor must get the legalist to see that their judgmental attitude is contrary to the Scriptures. He must get them to see that physical modesty flows out of a love of Christ. If their modesty is a hammer to bash others or a source of pride instead of the adornment of a Christian woman, it is not pleasing to the Lord.  He must get them to see that modesty is not whatever they think it is. Some types of dress are obviously immodest.  Others are not so obviously immodest.  Sometimes to decide between the two requires wisdom. The pastor should remind them that love covers a multitude of sins.  

                The worldly minded Christian assumes that their love of Christ does not have to reflect itself in their dress. They do not want to be thought of as legalists.  Even if they dress modestly, they rarely, if ever, judge anyone else as immodest. They believe that by not addressing issues like modesty they will show the world the love of Christ. They do judge those judgmental Christians who think modesty is a matter of dress instead of a matter of the heart.  The worldly minded Christian must be taught that our love of Christ will be reflected in concrete ways, including how we dress.  Modesty begins with a gentle and quiet spirit, but is reflected in a gentle and quiet dress.  They need to be taught that just because there are no specific modesty standards in the Scriptures does not mean anything goes.   They need to be reminded that that there are good ways to love the world (John 3:16) and bad ways to love the world. (I John 2:15)  They need to be reminded that in a pagan culture like ours we should expect to look differently. Some types of dress are obviously modest.  Others are not so obvious. To determine between the two requires wisdom.  The pastor should remind them that love covers a multitude of sins, even the sin of legalism.

                In a sermon like this I would try to address both dangers. To fail to do this is pastorally naive and will keep the church from maturing in Christ.