The latter is commonly described as “libertarian” these days.
“Conservative” could mean on social issues (e.g. those based on religion, race, and/or gender) or fiscal/economic issues or both. People are not always “conservative” (or not) on every issue, even within the broad groupings of issues (e.g. some religious conservatives may have conservative gender views, but find racism abhorrent, while some racists are not religious and do not have conservative gender views). However, it is common enough to be “conservative” on most or all of these issues that describing someone as “conservative” can connote that impression, even it is not accurate in the individual case. Obviously, this is annoying to “conservative” people who are not racist, sexist, or otherwise bigoted.