What’s Inside Vietnam’s Bowl of Secrets?

Hoi An’s iconic noodle dish was tangled in mystery for decades. Then writer David Farley came to town.

What’s Inside Vietnam’s Bowl of Secrets?

Photos by Andrew Rowat

“Ba Le…?” I said to a shirtless man, who halted me from going any farther down the narrow lane. He simply pointed, his long fingernail directing me the opposite way. I was on the back streets of Hoi An, a prettied-up UNESCO-protected town on the central coast of Vietnam. I had ventured into this neighborhood, clearly not a place many tourists wander, in the hope of finding a well. The Ba Le well was actually listed in my guidebook, which I’d accidentally donated to the seat pocket of the airplane earlier that morning.

As I reversed course and proceeded deeper into the neighborhood, the buzz of scooters grew fainter with each step, replaced by the incongruent sounds that poured out of open doors and tangled in the open air—American B-grade movie dialogue and Vietnamese pop music. A few turns later, there I was, staring at the square-shaped, concrete Ba Le well, one of its sides abutting the wall of an old building. Two men were using a bucket, tied to a rope, to fish out water and then pour it into huge plastic containers on a bicycle fashioned into a cart. They ignored my presence as I looked down the well: Splotches of Day-Glo-green moss clung to the bricks, and a billowy image of myself looked back at me from 10 feet below.

Until relatively recently, Hoi An’s 80 or so wells, centuries old, were the town’s main sources of drinking water. The town now has a modern system for running water, but many residents still make the pilgrimage to Ba Le. The well is thought to have been built in the 10th century by the Cham people, a native population whose empire flourished in central Vietnam until the 15th century. As the story goes, there’s something special about the water from the Ba Le well. An entire mythology has accrued around it. Some say the water is medicinal; others claim the well has some mystical connection to fairies.

But finding the well was only the first step of my quest in Hoi An. My main motivation for traveling 10,000 miles from New York was to eat one particular dish identified with this town, a dish supposedly made exclusively with Ba Le water.

These days, most major cosmopolitan cities offer nearly every cuisine, including Vietnamese. One Vietnamese specialty you’ll almost never find, though—not in Little Saigon in Southern California nor even in the real Saigon in southern Vietnam—is cao lau (pronounced “cow laow”). According to tradition, this pork-a-licious, herb-scented noodle dish has an umbilical attachment to Hoi An. The uniquely textured noodles must be made with local ingredients—specifically, water from the Ba Le well and ash from a certain tree that grows on the Cham Islands, some 13 miles off the coast of Hoi An. But not many people know how to make cao lau noodles. One family has, for generations, had a monopoly on making the six-inch-long rectangular rice noodles that look like a thicker, shorter version of fettuccine. And its members have never told anyone the recipe. For these reasons, cao lau is one of the few dishes in the world that has largely escaped the giant sponge we call globalization. You can’t take cao lau out of Hoi An, nor can you take the Hoi An out of cao lau. It’s the ultimate example of culinary terroir.

I had tried cao lau for the first time a year earlier, when I visited Hoi An after eating my way around the rest of Vietnam. It was a revelation, or a series of them: the snap of crisp aromatic sprouts, basil, and coriander; the sublime unctuous quality of thinly sliced salty pork; the crunch of flat, square croutons (made from the same ingredients as the noodles); the silky, smoky broth, spiked with Chinese five-spice seasoning; and at the heart of the experience, the rice noodles, thick and chewy with a coarse texture on the outside and a slightly starchy taste. I’d never had a dish like cao lau in Vietnam. Its flavors and composition were completely unlike pho, the noodle dish most people associate with the country. I thought, this can’t be Vietnamese.

After my first visit to Hoi An, I did a little research. It seems no one really knows where cao lau came from. The prevailing origin tale is that Japanese traders brought it with them when they set up shop in Hoi An in the 16th and 17th centuries. The most famous Japanese remnant in town is the stout wooden bridge that crosses a small canal and connects the two major streets of the old quarter. Other versions of the dish’s genesis story claim cao lau originated with the Cham. There are also tales of Chinese adventurers who turned up with the noodles in the 19th century and taught someone how to make them. The name does seem to come from the Chinese characters for “high steamer”; unlike most noodles, which are boiled, cao lau noodles are steamed.

Now, a year later, I had returned, hoping to uncover the mysteries behind a dish that seemed like an enigma wrapped in rice noodles and shrouded in slices of pork. Where did cao lau really come from? What made the noodles so magical that the dish couldn’t be reproduced anywhere else? On my first afternoon back in Hoi An, the ordinary-looking Ba Le well wasn’t revealing anything. I walked away even more determined to find the answers.


Photo by Andrew Rowat

That evening, I was sitting in a restaurant talking to Thao, a 29-year-old Hoi An local and friend of a friend, who agreed to help me with my mission. We were at Trung Bac, a restaurant in the center of town that is famous for its cao lau, mostly because it has been serving it for so long. An engraved wooden sign states (in Vietnamese), “This restaurant has been open more than 100 years.”

Thao said he took me there not only because the cao lau is good but also because the owner of the restaurant is related to the cao lau noodle-making family.

“Is there anything you want to ask him?”

I figured I’d shoot for the Holy Grail right away. “Yes. Ask him if I can watch the family make noodles,” I said.

Let me get one thing straight: I was not looking to steal the recipe for cao lau noodles and sell it on the culinary black market where foodies and chefs linger down dark alleyways, waiting for a guy to open up his trench coat to reveal contraband recipes for sale. I’m just a food-loving traveler obsessed with unraveling weird mysteries.

Moments later, Tran Tan Man, the owner of Trung Bac, stood at our table. Thao introduced me and explained that I was in town to learn about cao lau. Then he popped the question about meeting Tran’s extended family to find out how the noodles were made. “Absolutely not,” Tran said, shaking his head from side to side. “No one watches!” And then he ran off to greet a gaggle of tourists that had just streamed in.

“He’s right,” Thao said. “No one watches. I don’t think they have ever let anyone from outside the family watch them.”

A couple of decades ago no tourist groups traipsed into restaurants in Hoi An. In fact, the few travelers who knew about the town saw a very different Hoi An. It was a rundown backwater. There was one place to stay, the Hoi An Hotel, which was more like a boardinghouse. There were few restaurants, save for the usual makeshift eateries one finds in the alleyways of every Vietnamese town. All that began to change when Lonely Planet included Hoi An in its guidebook Vietnam, Laos & Cambodia in 1991 and South-East Asia on a Shoestring the following year. The attention quickly cemented the town’s place on the backpacker’s grand tour of Southeast Asia. Cheap and midrange hotels popped up in the late 1990s. And after UNESCO designated Hoi An’s historical central district (Ancient Town) as a World Heritage site in 1999, posh resorts, most of them along the coast just outside town, laid out their red carpets. Western-oriented Vietnamese restaurants followed soon after.

Thanks to all those tourists (and support from UNESCO), Hoi An is no longer the ram-shackle town it once was; its buildings glow with that recently refurbished shine. But the local economy has benefited in other ways as well. Thao works at the upscale Nam Hai resort, and as I went around town with him, we kept running into friends of his who were employed by Nam Hai or other resorts. “Before, young people couldn’t wait to get out of here,” he told me one day over coffee. “Now we all have jobs and no reason to leave.”

All this development has put an indelible mark on the town. Walking through the streets, flanked by one- and two-story buildings and ornate Chinese-style temples, you are accosted by the usual “Hello” and “Buy something!” from vendors and “Where you going?” from motorbike taxi drivers. But Hoi An also has a kinder, gentler way of assaulting the visitor’s senses: Classical music wafts out of loudspeakers affixed to telephone poles throughout the car-free center of town (Eine Kleine Nachtmusik is a recurring standard), while the sweet smell of incense seeps into your olfactory glands. If you factor in the upscale restaurants aimed at wealthy Westerners, the absurd number of tailor shops, and the pro-Communist government slogans on banners stretching over the street, you’ve got one of the oddest towns in Asia. It’s a place trying so hard to seem “authentic” that it becomes wildly inauthentic, like some kind of culturally dissonant theme park, collectively dreamed up by Ho Chi Minh, Karl Marx, Milton Friedman, and Wolfgang Amadeus Mozart.


Photo by Andrew Rowat

One person who caught the tourist wave early is Hoi An native Trinh Diem Vy, who opened Mermaid, her first restaurant (which has since closed), in 1991. The place served a menu of Vietnamese staples and Hoi An specialties, including cao lau, and over time Ms. Vy, as she is known locally, built up a mini empire that now counts a handful of restaurants, a cooking school, and a new book called Taste Vietnam. She has become the culinary face of Hoi An.

While Thao was at work one day, I met with Ms. Vy at Morning Glory, her cooking school and restaurant, which specializes in dishes inspired by street food and old family recipes. We sat at a small table next to the kitchen. Lines of Western tourists snaked through the high-ceilinged front dining room on their way to the upstairs classroom-kitchen. Ms. Vy had a bowl of cao lau brought over to me. Because she prefers an extra dose of crispy texture, it had rice crackers among the toppings in addition to the fried noodle-dough croutons. The key to good cao lau, Ms. Vy said, is the pork, barbecued in the Chinese style called char siu. “This is what I call ‘tourist pork,’ ” she said, jabbing a chopstick at a thin slice resting atop the mound of noodles and herbs. “Fatty pork is better—it just melts in your mouth—but I fear my customers won’t eat it.”

As for how the noodles are made, Ms. Vy said she didn’t know the secret. But she told me that two branches of the same family make them, and she gave me a lead: I could find Madame Trai, the matriarch of the family, selling cao lau noodles at a stand in the central food market.

After finishing my bowl of extra-crunchy cao lau with Ms. Vy and watching more tourists flow through her restaurant, I walked to the central market. More than a few food stalls had hand-lettered signs advertising cao lau and pho. Pangs of nervousness poked at my stomach. This, I thought, could be a defining moment in my hunt for the secrets of cao lau. At the end of the lane, I came to a table with piles of brownish cao lau noodles stretched across it. A teenage girl behind the table made eye contact with me. “Madame Trai?” I asked. She nodded toward a plump sexagenarian sitting at the other end of the table. The woman looked like a rotund hen sitting on her eggs. The scowl on her face made her a forbidding presence.

“Xin chao,” I said, hello in Vietnamese. She hardly moved, and uttered nothing. “Do you speak English?” I asked. Madame Trai remained mute. “Cao lau,” I said, pointing to the noodles on the table in front of her. “I want to know about cao lau. Can you tell me how it’s made?” Just then the younger woman said something—translating, I figured—for Madame Trai.

But Trai just stared back at me. As her grimace deepened, I felt myself shrinking. Her eyes narrowed with suspicion, as if I really were a culinary secret agent who had come to Hoi An to steal the family cao lau recipe.

I walked away, ignoring the girls tugging at my arm to take me to their families’ tailor shops. I felt defeated; my quest might be over.

Later that day I met up with Thao, and he took me to Thanh, a favorite place for cao lau, about a 15-minute walk from the center of town. I told Thao about my encounter with Trai at the market and how discouraged I was about my chances to get into the noodle-making family’s home factory.

“I can talk to them,” he said, as Ms. Thanh, the owner, put two bowls of cao lau in front of us.

As I dug in, a horrified look spread over Thao’s face. “No!” he said, glaring at the spoon in my hand. “Never use a spoon! That’s for pho.” Cao lau is, in a way, the anti-pho. Broth is the star in pho, Vietnam’s national dish; rare brisket, tendons, and soft noodles play supporting roles. Conversely, cao lau is short on broth; a dark shallow pool comes as a hidden surprise when you get to the bottom of the bowl. Ms. Thanh’s broth was especially good. She deviates slightly from the traditional recipe, she said.


Photo by Andrew Rowat

“How so?” I asked.

“It’s a secret,” she said, smiling wryly.

A secret. What is it, I wondered, with this town?

When Thao picked me up on his scooter the next day, he told me that maybe the greatest food mystery in this town of culinary secrets was about to be unraveled for me.

“I went to see the family that makes cao lau,” he said, referring to Madame Trai’s branch, “and I told them about you.” He paused, waiting for a reaction, and then added, “They said yes. They are going to welcome you to their house where they make the noodles.”

I put my hands on my head in disbelief. Even Thao seemed surprised.

“We have to go there early,” he said. “They make it very early so it’s ready for breakfast.”

At five the next morning I was on the back of Thao’s scooter, riding down a dirt road on the outskirts of town, past a small outdoor market, a few bars with open facades, a colorful Buddhist temple, and a wandering cow. I still had my doubts that this was going to happen. It seemed too easy.

The front of the one-story house was almost obscured by two 10-foot-high piles of wood. We hopped off the scooter and walked around the house to the back. There, in two adjacent, low-ceilinged rooms lit only by fire, where spiderwebs hung to eye level, four family members worked silently, each at a different step in the cao lau noodle-making process. Other than a fan on a floor stand, I saw nothing that used electricity. Fires crackled and steam rose from pots. The few machines in use—to knead dough and cut noodles—were manually operated. It felt as though I’d just stepped into a medieval workshop. I’d made it to the secret epicenter of cao lau noodle production.Ta Ngoc Em, 54, the family patriarch, came out to greet us, brushing powder from his hands. Thao handed him a cigarette and lit it for him, as if that were part of the deal, and they exchanged a few words. Em, who had an Uncle Ho–style beard and blackened teeth, then went back to stirring a giant cauldron of bubbling dough. The cigarette dangled from his lips as he pounded a wooden stick into the dough with both hands.

“You can ask them anything you want,” Thao said, fanning his right hand toward the silent workers.

I was suspicious. Why did they allow me to come here? I suppressed my skepticism. I should take advantage of my good fortune, I thought, and I began peppering Em with questions about the process.

Here’s what I learned: Em had already mixed ash into the water (he burns wood to create ash only once every few months, he told me) and then sifted out the larger bits. The ash helps give the noodle its chewiness. Em pours rice into that ashy water and boils it and pounds it until he gets a huge cauldron of thick dough that he stirs and works for 45 minutes. Then he puts the hunk of dough on a broad flat basket in a huge fire-heated metal steamer for 75 minutes. The ash, the local water, and the steam all contribute to the unique character of cao lau noodles.

At the end of the steaming, Em and his brother carry the dough into the next room, where several women put it through a giant crank-operated mixer, like a preindustrial restaurant-size KitchenAid. After the dough is kneaded, rolled flat, and brushed with peanut oil, Em’s sister-in-law puts the flat sheets through a manual pasta cutter and slices them into six-inch-long threads. But that’s not the end. The noodles go back into the steamer again for another 75 minutes. Finally, they’re covered in banana leaves to cool off. Then they’re done.

Em said his brother—that’s the second branch of the family that Ms. Vy referred to—also makes cao lau noodles but in much smaller quantities. Between the two sides of the family, he said, they supply all the restaurants in Hoi An with the precious noodles. He said there’s a guy just outside of Hoi An who figured out how to make them but he doesn’t produce much.

When I asked Em, the fourth generation of his family to make the noodles, where he thought cao lau originally came from, he shrugged and said, “China?” That’s how his family became the sole producers of the noodles, he explained. Someone from China taught his great-grandfather the recipe, and the family has kept it a closely guarded secret ever since.

How, I asked, do they transport the water from the Ba Le well in the center of town out to this house? Em stood up, took the cigarette from his mouth, and leaned against a pole. “We stopped doing that during my father’s time,” he said. “We dug our own well here”—he nodded to his right, toward the side of the house, where chickens were scratching in the dirt—“and it had properties similar to the Ba Le well.”

The main component, he said, was alum, a chemical compound that has been known to have medicinal and preservative properties (and is sometimes used in baking powders).


Photo by Andrew Rowat

“What about the wood,” I asked, “the wood from the Cham Islands?”

Em laughed, shook his head, and said, “Cham is now protected, so we get all this wood from the surrounding area. It’s the same type of tree, though.”

Em added, “Sure, the water and the wood are essential to make cao lau, but our family has been making cao lau noodles for so long that we have a reputation now.” Translation: It’s not so much about the water or the wood; it’s the technique. Em has been doing this routine 364 days a year, taking a day off for the beginning of Tet, the lunar new year, since he was 12.Just then his son turned up on a motorbike, returning from his first round of delivering noodles to various restaurants around town. Em nodded toward him and said, “He’s next. At least I hope so.”

“So you’re not sure he’ll take over your job?” I asked.

“We don’t want to push him. He has a good career as a tailor in the center of town,” Em said. “But if he doesn’t do it, then it could mean the end of our family making cao lau noodles.”A silence fell over us for a moment. Then Em told us a story:

One day, about a year or so ago, a government official showed up and said the family had to be more open about the secret of the noodles—that if something happened to the family, there would be no one to make them. Now that Hoi An is thriving with tourism, city officials said it would be a disaster if cao lau, the city’s chief culinary attraction, disappeared.

“This is why,” Em said, “we decided it would be OK if you watched us make it.” Everything was starting to make sense. “Since we’re not sure what the future holds for cao lau noodles, I would teach someone how to make them,” Em said, pausing before adding: “For a price.” When I prompted him, he said he’d do it for 100 million dong (about $4,800). I was tempted to take him up on his offer but it felt like too much pressure to be the sole upholder of the cao lau legacy. So I passed.

Given the greater opportunities young Vietnamese have in Hoi An these days, and the laborious life of a cao lau maker, it’s hard to imagine someone would want to assume the cao lau–making mantle. The irony is that Hoi An’s rise as a prosperous tourist attraction is also threatening the existence of the town’s most iconic dish. Cao lau’s survival depended on the people I was standing with: Em, his son, and the various family members who help make cao lau noodles daily (as well as Madame Trai, who sells them at the market).

And with that, Em’s son, who makes his deliveries every day before going to his tailor job, wrapped a bungee cord around the packages of freshly made noodles on the back of his scooter and rode off. I stood there with Em and Thao, watching as Em’s son sped down the gravel road until we lost sight of him, and the buzz of his scooter faded away.

>>Next: The Dish Worth Traveling to Vietnam For (and It Isn’t Pho)

David is a contributing Writer at AFAR, author of “An Irreverent Curiosity: In Search of the Church’s Strangest Relic in Italy’s Oddest Town” (Penguin/Gotham Books, 2009), and the editor of “Travelers’ Tales Prague and the Czech Republic: True Stories” (Travelers’ Tales, 2006).
From Our Partners
Sign up for our newsletter
Join more than a million of the world’s best travelers. Subscribe to the Daily Wander newsletter.
More From AFAR