OP, you wouldn’t be compromising anything in terms of academic quality – quality of teaching, class sizes, access to professors, etc.
The only potential sacrifice would be this: Brown will offer more majors and courses than Amherst will.
But that is essentially a “university vs. LAC” issue, and it is nearly a constant: universities tend to offer more courses and majors than LACs do.
But class for class, the quality will likely be quite similar.
You mentioned working in a foreign service or policy-making body. If you are truly interested in that, I suggest that you take a look at Georgetown University in addition to Amherst and Brown. Georgetown is known for its immensely strong Foreign Service program.