You may get lucky, but bring warm clothes to enjoy the outdoors and be prepared if the weather doesn’t cooperate! My family includes many year-round Cape Codders, and they all remark -every year - that “we don’t get much spring here, and it’s pretty late if it happens.”
Having said that, it’s a joy to walk and bike without crowds. I’m not super familiar with Nantucket, but I’d also check out what’s happening at community year-round venues like libraries, book stores, etc. My town has great programming in the off season.