When you begin to examine what prevents you from actually accomplishing Yoga, you will see that the wider environment cannot be kept wholly out. There is pain, there is weakness, there is death. And these are here, in your place of rest, too. Like a cow is prepared for slaughter with a safe pen and grain, we have chosen to not ignore the consequences of our Karma: there is an urgent necessity to practice Karma Yoga, and taking control of this destiny requires self-sacrifice, the sacrifice of "self", the sacrifice of identity. This requires skill in the four Jnanas: self-control begins with the body, the mind's thoughts and feelings, the consciousness and ultimately with the identity. Perfecting this skill (Siddhi) toward your contentment requires understanding sufficiency (Svaha), and this requires a mastery of Hatha Yoga. Ashrams are comfortable, and easy - and are frequently necessary. But an island in a flooding river is no place to stay very long. We have chosen to not be afraid of the water - we want to cross entirely over, and see what lies beyond. It is necessary to journey on from each Ashrama, from Brahmacharya to Grihastha, to Vanaprastha, to Sannyasa.
A house provides shelter and safety. But a Yogi cannot stay at home all day.