<p>That would certainly be possible. Many students that go to grad school for astronomy haven’t really taken many astronomy courses in undergrad. Most people going to grad school for astronomy/astrophysics will major in physics during undergrad, and may never even take any astronomy courses. The physics is the most important thing. Get as much of that under your belt as you can. </p>
<p>Have you considered double majoring in physics and math?</p>