Customers have expressed their dismay at the closure of a much-loved cafe, which has shut after 20 years.