First off, I'd like to point out that what I'm about to say is merely my personal opinion and you'll have to make up your own mind whether to groom or not.
My horse, Frosty, also lives out side and I hardly groom her at all. When I do, it is usually just a quick brush off just to get rid of the dirt round the saddle area where I will sit (bareback). But that is more out of laziness than anything else.
Yes, brushing, do remove natural oils from the horses coat, but in my opinion I think it doesn't remove that much of the oils that you need to be worried about it. If you like a cleaning freak, grooming 3 times a day for an hour at a time, then I'd say you should start thinking about what effect that might have on your horse's coat. On the other hand, washing a horse that stay outside, would be more of a concern to me, since washing with shampoo and water will remove most, if not all of the oil in the horse's coat. Again, giving your horse a wash, once in a couple of months for a special occasion won't do that much harm, since the skin will regenerate and replace the oil over time, but washing a horse that is staying out side daily or even weekly, I don't think is such a good idea.
If I could give you some good advice regarding this and any thing else that you are unsure of in the horse world. Read as much as you can and ask as many people as you can, but don't just ask what you should or shouldn't do, also be sure to ask why you should or shouldn't do it? Even in books, look for reasons. Remember the authors of books are also just people like you and me, they not always right, or more precise what is right for that particular author, might not be right for you and your horse. So, if you get a good explanation that make sense
to you and you think it will work for you and your horse, then go with it, but if it doesn't make sense to you and you feel uncomfortable with whatever suggestion someone made, go with your gut feeling and leave it. At the end of the day it is your horse and you are responsible for his well being. And last, but not least, listen to your horse. If your horse don't seem to like something, like the bit in his mouth, a particular saddle or what ever, chances are that it is hurting him, so believe him when he say so.

Hope you can make sense from what I'm saying. Been some time since someone posted on the forum so I'm just rambling about, since it is seldom that there is anyone around here to chat to.
Hope to hear from you again.