Interesting evolution of your garden's location! I wonder have you thought about also keeping this garden since you have already made some efforts into the soil and fencing? Since it's a bit further than you'd like you could plant some veggies, trees, shrubs and other perennials that wouldn't need as much constant attention as your primary garden. Just a thought! Your post also left me wondering if you are doing much animal manure amendments? You might also check out EM effective microorganisms , one great way of boosting the soil life in a new garden. Keep up the great work and remember every year the soil will get better and you'll likely have less problems with pests!