It also depends upon which AP Physics. AP Physics 1 is designed not to need a prior physics course, while one would be find it difficult, but not impossible, ton take AP Physics C with no prior knowledge.
Anyway back to the original question, physics is recommended (but not required) for most top colleges (excluding some high powered STEM unis which do require it). For colleges that are not in the stratosphere, no physics is fine.