A plan for people to build their own homes near a medieval village has been rejected after objectors lined up to voice their worries about it.