This was a really strange house when I first saw it," says Robert Duffy, smiling as he shows off the octagonal living room of his country house in New York's Hudson Valley. The space is exquisitely decorated, with pale walls the color of a duck's egg and, at its center, a vast mahogany table dominated by an enormous Chinese bowl of periwinkle hydrangeas.

[image id="f70e81f8-6b27-40f2-b461-1bfbdf1ab250" mediaId="4800fbd3-d3ba-4a89-bb29-628d5b585309" caption="" loc="C" share="true" expand="true" size="M"][/image]

The copper lanterns flanking the entrance are by Bevolo.

Clearly, there had to be a mysterious backstory. "Thirty years ago, I had a weekend home across the river," Duffy, founder and deputy chairman of Marc Jacobs International, explains. At the time, an acquaintance showed him an uninhabited folly on the grounds of Ferncliff, the old Astor estate near Rhinebeck. He learned that when the Astors needed a break from the constrained formality of their main residence, they retreated—via miniature railroad—to this neoclassical brick pavilion, which they called their "teahouse." Duffy and his friends would go there to "hang out and have fun."

[image id="a84ca8ad-65b1-4064-a2d8-c8aa9f6905f1" mediaId="6c42d14b-0eaa-4d11-b74c-4708c225c0b1" caption="" loc="C" share="true" expand="true" size="M"][/image]

The kitchen's wine refrigerator and vent hood are by GE Monogram; the Regency-style mahogany dining chairs are from the 1940s, and the collection of copper cookware includes antique and vintage pieces.

The estate later sold off the fanciful structure to an older couple, who expanded it into a residence but never ended up living there. The home sat empty for seven years until Duffy came along to rescue it. There was considerable work to do: The classical columns framing the front porch had been covered in a Styrofoam-like substance ("My fingers actually sank into it when I touched one," Duffy says), and a jungle had grown up around the building, entirely blocking sublime views of the Hudson River.

[image id="9adf00ec-6e1e-44cf-86e2-99ea5acaebd5" mediaId="4674b214-f922-4feb-8c31-ed0929d7fc58" caption="" loc="C" share="true" expand="true" size="M"][/image]

A hallway showing the original front wall and doorways of the Astor teahouse; the 19th-century brass lantern is from Morocco, and the silk wallcovering is by Fromental.

He turned to his longtime decorator, Richard McGeehan, for assistance. The first order of business was to reface the home's stucco-clad additions in antique brick to match the teahouse's original exterior. The spongy columns were replaced with four antique cast-iron pillars that McGeehan found in London.

[image id="7e693ada-6b84-49d1-a6a8-544a7d2616e7" mediaId="0a189492-3d4b-4e2b-ad16-48a2c610f144" caption="" loc="C" share="true" expand="true" size="M"][/image]

A cabinet is topped with ginger jars, platters, and bowls from Duffy's extensive collection of 18th- to 20th-century blue-and-white ceramics; the curtains are of a Scalamandré damask.

The men then shifted their laser focus to transforming the interior. To say that this homeowner has definite views on decorating would be a serious understatement: After renovating and decorating more than 10 houses, Duffy knows precisely what he likes.

[image id="4df26de5-cb8d-4bec-8245-814d4dabe38e" mediaId="5fce3000-bab4-412c-8e6e-95a7eb642ee3" caption="" loc="C" share="true" expand="true" size="M"][/image]

Bluestone surrounds the swimming pool, while the adjacent pavilion is faced in limewashed brick and has a copper roof; the limestone table and benches are from Elizabeth Street Gallery.

A small sampling of his directives: "I must have blue, my favorite color, in every house. I always have a room with chinoiserie wallpaper and a dark space with a window seat where I can read." Not to mention that he has amassed 28 Moroccan Berber rugs, 50 Persian carpets, and enough scented candles to illuminate a small cathedral.

[image id="4d325a61-2b0a-493c-bc8b-3b2c2a91b870" mediaId="477269f1-2a90-4bda-8c4d-81ba61ec7d59" caption="" loc="C" share="true" expand="true" size="M"][/image]

McGeehan designed the walnut bookshelves in the library; the brass library ladder is from the 19th century.

Having collaborated eight times over the years, Duffy and McGeehan have developed a subtle, symbiotic way of working together. "Robert's interests, objects, and tastes are diverse," the designer says, "and my job is to bind them together into a pleasing ensemble." For example, in the living room, the wood floor is covered in a jewellike patchwork of nine rare Persian rugs. An 18th-century English tall-case clock stands near a small Moroccan table inlaid with ebony and bone, while a vitrine is filled with unusually fine scrimshaw from Provincetown, Massachusetts, where Robert used to own a home (Elle Decor, September 2009).

[image id="8a8cfc8a-e4a4-4dac-a25f-f41a0c84674d" mediaId="7505425a-2d0e-4453-926e-a84b741284dc" caption="" loc="C" share="true" expand="true" size="M"][/image]

In the master bedroom, the lacquered canopy bed is from Fernworks Antiques, the Chinese Art Deco bench is covered in a Clarence House velvet, and the hand-painted wallpaper is by de Gournay; the curtains and pelmet are of Lelievre damasks, and the 1920s painting is signed Constant; the 20th-century Persian rug was shaved to make it look older.

Despite its grandeur and a profusion of priceless objets, the house is decidedly not just for show. Duffy shares his home with his two young children, five-year-old Victoria and baby boy Caldwell. There are also two dogs in residence, a Kerry blue terrier named Mike and a giant schnauzer, Hudson. Look behind the 18th-century sofa, upholstered in blue (of course!), and one discovers a delightfully uncurated mix of things, including a skateboard, a Hula-Hoop, and a glitter-encrusted soccer ball. If somebody—child, dog, or guest—should happen to break something, "That's never a big deal," Duffy promises.

[image id="dbe9e6fa-f422-4a63-b598-3052fa66f277" mediaId="3c87cf43-bb89-4503-8fcd-7d63bf3f4e92" caption="" loc="C" share="true" expand="true" size="M"][/image]

In the living room of Robert Duffy's home in New York's Hudson Valley, which was decorated by Richard McGeehan, the 18th-century English Chippendale sofa and George II–style antique library chairs are covered in a Brunschwig & Fils damask; the mahogany library table is from the late 19th century, the lacquered chinoiserie secretary is 18th-century English, and the bone-inlaid armoire is from Morocco; the antique Persian rugs are from Rug & Kilim, the circa-1910 portraits are by Charles Webster Hawthorne, and the wall color was mixed on-site.

McGeehan made sure that every one of his client's directives was met. The dining room is covered in a hand-painted wallpaper with a pattern of flowers and birds (there is also a 19th-century Chinese sedan chair, which is Victoria's favorite place to hide). Adjacent to the master bedroom, a compact library lined with floor-to-ceiling bookshelves and a window seat for reading satisfies Duffy's desire for a small, dark retreat.

[image id="6a7a846b-5978-49bd-869e-68c1137af55e" mediaId="3864e9a9-344b-45db-aaba-598015a819cd" caption="" loc="C" share="true" expand="true" size="M"][/image]

The landscape design by Michael Trapp features a path flanked by boxwood balls and an iron arbor original to the Astor estate, now covered in New Dawn roses.

Outside, the glorious Hudson is now fully visible thanks to some strategic chainsaw thinning and a herd of leaf and weed–munching goats. What's more, in a shady corner of the garden, what was once Vincent Astor's grave (his widow, Brooke, moved him many years ago to Sleepy Hollow Cemetery) today has a new occupant: the current homeowner's late, deeply lamented dog.

This story originally appeared in the October 2016 issue of ELLE DECOR.