There are slight pull lines across the hips, meaning there is too little fabric somewhere.
I took Lynda Maynard's Craftsy course advice to heart, and cut a muslin that was on proper grain, had 1" seam allowances and, most importantly, followed the grain marks with contrast thread. I even made extra sure these grain lines were on actual grain by removing one woven thread from the muslin before stitching the line:
Then, I enlisted Mr T's help to figure out the problem. The grain markings (both crosswise and lengthwise grains) need to be parallel / perpendicular to the floor. Any distortion indicates an issue with fit. I didn't have any issue with the crosswise grain (parallel to floor), but the lengthwise one was off both in the front and in the back. He was very helpful in telling me exactly what was wrong!
I actually messed up a lot doing this. I thought the problem was not enough fabric between the waist and the hip, but multiple attempts to fix it yesterday did not do anything for fit. The lengthwise grain was still waaay off. This morning I stitched everything back together (zig zag stitch was great for patching everything up!) and started again.
Having re-watched the "Fitting the lower body" part of the class during the morning, I realized I might need to try to get more room at the hips: that proved to be the issue!
I only altered the front (Mr Ts help only goes so far), but ended up with 9/10" extra at the hip level on the left, and 1 6/10" on the right. This adds up to roughly 2.5" together, which I thought was too much because I could pinch the right side seams a bit with the muslin on. Unfortunately I did not actually pin & pinch to see what the difference was...
Because 2 - 2.5" seemed too much, I then pulled out "The perfect fit" to see what they said about broad hips. I never thought I had broad hips, so I couldn't really explain why I needed so much extra. However, the text in the perfect fit said one needs to measure hips, then add 2" to 3" and use that measurement for measuring the pattern. Said and done, my pattern pieces were 38" in total, and with my computed extra on the muslin would match exactly what the book said.
While I was doing that I realized that my pattern pieces said "Size 4 waist, 0 hips". I took my original pattern out and measured - it had in total 2" at the hips (1/2" at each seam).
So I took the book's guidance and made a broad hip adjustment on my pattern:
I toyed with the idea of recopying the pattern piece, but I had made length adjustments and waistband adjustments so I didn't really feel like doing that. Besides, the text in The perfect fit actually says to add at the side seams only if the adjustment is 3/8" each, and mine needed to be bigger. So I went with the "major adjustment" listed in the book, which required me to actually move the pattern in the hip area outwards.
Since my muslin needed more in front than in the back, I added 6/8" to the front and 2/8" to the back. This should result in 2" in total. I also added 3/8" to the stitch line for the waist on each side, because by measuring against the pattern that seemed to be necessary.