Kayaking in Delaware Bay, Cape May Harbor, Salt Creek which runs behind the city itself, and the back wetlands between Wildwood and Cape May. Since we’re down there a lot, we also explore behind West Wildwood, North Wildwood, Stone Harbor, and Avalon, but someone on a single vacation wouldn’t probably go that far afield. Last summer, on the Bay off Sunset beach, we were surrounded by dolphins at one point. T hat was pretty special.
The biking is mostly roads, but once you get out of the city itself (which is tiny), the roads are fairly clear, have wide shoulders, and in general are very bike friendly. On the island, there are pretty routes to Sunset Beach and back farmlands, and if you cross one of the bridges, you can bike all the way up the Delaware Bay if you want.
As nj2011mom says, there are kayak rental places on the back creek and on both sides of the Harbor.