How much laundry soap are you using?
Using too much causes it to not rinse out properly and makes clothing look yellow and dingy (the soap discolors in the dryer).
Also stuffing the washer too full can also cause soap to not rinse out properly.
If you want to see if you are using too much,
try washing a load and then set it on rinse again.
Look in the washer after it starts to agitate again.
If you see any soap suds,
cut in half how much you use.
Unless you work in very grimy conditions you don`t need that much soap,
just enough to refresh your clothes and get out perspiration.
I use less than a 1/3 of what is recommended unless it`s something I have been wearing to dig in the garden.
Also check the washing instructions on the slacks.
Does the manufacturer recommend using bleach or bleach substitute?