Amherst is a liberal arts college and as such does not have a graduate program in physics. However, the physics program it offers is comparable to most other schools. If you are interested in pursuing a graduate program in physics, you can certainly get a good preparation at Amherst as long as you make sure to take as many extra physics courses as possible beyond the minimum required. Specifically, you should try to take a full year of Quantum Mechanics if possible. In addition, take a lot of extra mathematics.
At my university, Illinois Institute of Technology
, we often get graduate students from liberal arts colleges. Those who have had a full year of Quantum Mechanics are able to start with our graduate Quantum sequence right away.
The quality of education that you get in physics is largely dependent on YOU. If you put the effort in, you will be fine no matter where you get your Bachelor's.