Gaylordsville is located in the northwest corner of New Milford. It is part of the Valley known to the Indians as the Wheniuck or Red Plumb Plain. On the East the boundary is Quanuctnic or Long Mountain, but it has never been decided whether it should be at the foot of the mountain or somewhere up on top. The Southern boundary is also vague, usually considered to be an imaginary line leaving the Housatonic River somewhere South of the Tory Cave and extending across Squash Hollow. The Sherman town line forms the Western boundary, although several homes in Sherman are usually considered to be part of the Gaylordsville community.
The Housatonic River runs through the center of the village and is joined by the Wimisink, Womunshenuck, Naromiyocknowhusunkatankshunk, and Squash Hollow brooks. The South end of the valley is divided into two narrow valleys by Straits Mountain or Pauguiack. The North end of this overlooks the village and is called "the Pinnacle". The area usually considered to be Gaylordsville is about four miles long and one mile wide.