No, you’re not crazy, but you should not be making any knee-jerk decisions. For instance, what year are you? Are the academics meeting your needs, or is this more of a social life (or fit) problem? You never mention the actual classes, so I am guessing that it is the environment you don’t like. Don’t get me wrong. Sometimes a bad fit is good enough reason to transfer. On the other hand, lots of students have first semester or even first year adjustment problems. It took me half a year to find my group in college, and then I had a blast.
That said, yes, people leave Ivies. Yes, people sometimes arrive at an Ivy (choosing it solely because it was an Ivy) and find out that it is a terrible fit. Still others are stunned when they arrive and find out that what they had pictured (an institution dedicated to educating young minds) does not at all match reality (ginormous classes, inaccessible profs, intense focus on research and graduate students).
Research by Dale and Krueger found that students who are accepted to Ivies but choose to go elsewhere have no difference in outcomes (the exception to this is URMs who attend elite schools), so you’re not in any real risk if you do leave. That said, again, think carefully about why you want to leave. If you are finishing up your very first semester, maybe give it some time, give it another chance, give it the ol’ college try!