Stay stitch those seams!
|I'll be honest: this solution is going to result in pants that don't fit|
The best way to avoid this is to stay stitch the waist and crotch seams first thing, before you start handling them. Twill weaves will stretch its why we like them for work clothes, but you want them to stretch the way you want them to.
When it's too late, you are going to want to get out your pattern pieces and recut that waistband (and probably the back crotch seam). And it may not work very well. Another option is pleating the fabric to take up that slack.
Since I eat my feelings, I need new pants. Nudity is frowned upon, and it's winter here. I am working on a set of pants about posts. No, wait, reverse that.
My brain is a little overtaxed.