i see people out there in boats barely bigger than a row boat and they seem to do fine. pay attention to weather (wind direction, storms, etc) and know the capabilities of your equipment and you will be fine. sometimes fishing out of my dads 16 foot aluminum is more fun than the 28 footer in the spring.
when you fish in NJ do you troll? if you troll into a 2-3 footer with a boat that doesn't have a big bow you might be taking some water over the front so be careful.