Settled by descendants of Southold's founders, this little farming hamlet was first known as West Southold Plantation. Later it was became known as Hermitage because of an elderly recluse who lived on Peconic Lane.
Adjacent to the Peconic Post Office of today is what used to be known as the Jefferson store. It was built in 1855 by Captain Horace Prince, owner of a lumber yard in Peconic. When President Grover Cleveland was in office he appointed Jesse Lewis Case Postmaster of Peconic. In the same building was a hat shop run by Frank D. Smith who was known as the "Man Milliner of Peconic". The second floor of the store was a hall where plays and dances were held and it served as a kind of community center for the hamlet.
North on Mill Lane was the Old Mill, built in 1841, and wrecked by a terrible November storm in 1898. To the south, on the Peconic Bay side, the land between Broadwaters and Hutchinson's creeks is known as Indian Neck. There are many beautiful restored 18th and 19th century houses in Peconic, all private homes.