I usually do about 20 aways each season so not quite every one and believe me it's bloody tough!
As you say, it's a real problem getting time off work and uses up quite a lot of holidays, especially for midweek games which quite often require a whole day off and that's just getting there, you then often don't get back until well after midnight which means that if you do go into work the next day you're usually pretty knackered.
If you have a limited amount of holiday time available, and a family life to fit in as well, it can be quite difficult to justify taking midweek holidays to travel away.
Also, it certainly can be expensive! Even if you went to every game on the Supporters coaches, I reckon travel and ticket alone is an average of about £50 per game. Let's say you buy a programme (average £3), a couple of pints (average £8) and take your own food (average £5), your looking at almost £70 per game, assuming you do not qualify for a concession on travel or tickets.
Multiply that by 23 away games (£1610) and you're looking at a sizeable financial commitment

without even taking into account your season ticket and costs relating to the home games!
For those reasons as well as others, it seems that quite a lot of away fans are from groups who benefit from concessions (over 60's, students, etc) but there are a lot of fans who pay full price.
I would think that there are actually very few fans who go to every game of the season, particularly those who are paying full price! Having said that, some of the best times can be had on away trips.
The downside is also that you tend to see the team win less often when they play away, but when they do win away it's a great feeling and even the longest trip home seems to pass more quickly.