How’s your drainage? If your lawn gets saturated too frequently / easily, the worms will tunnel up to escape drowning and leave piles everywhere. I had that problem, and some good soil amendment + drainage fixes eliminated it entirely.
This is probably blasphemous in the lawn care sub, but in addition to mowing higher, I suggest you plant some shrubs or a tree or two on your property. The fence plus the lack of beneficial vegetation may be limiting the amount of prey animals that eat the worms. Having some shrubs or a tree could invite worm-eating birds to take up residence and help reduce the population, and in turn, volume of castings. I can sit and watch robins and killdeer eat dozens of worms every morning around my yard.
Feel your pain - has been the same here. Keep reading advice to brush or rake them, but that isn’t feasible they’re always wet and sludgy.
Daniel Hibbert sells a product called Baize which claims to reduce worm casts by adding nitrogen and sulphur. Haven’t tried it, but I did put down some iron sulphate and a little surfactant (baby shampoo) and the worms seemed to back off a little. Best thing I’ve done is not mow; blades just smear the worm casts across the grass.
I’m expecting to overseed in the spring to fill the gaps; I don’t think it’s possible to eliminate them entirely.