I think that's one of the things that sometimes gets a novice GM upset with the players. However, one of the secrets is that if you're clever enough everything can be plot related.

Abandoned mine? Perfect bad guy hideout, former or otherwise. Link it back into the main plot and presto!