For 35 years, beginning in 1823, the lighting of the approaches to Simons Bay was a theoretical idea bandied between the Admiralty, the secretary of State, the colonial authorities, and the Commander in Chief. The factors in constant interplay were: a) the usefulness of a site for a lighthouse b)the expense of building a lighthouse c)the expense of upkeep.

Four sites vied with each other. Noah's Ark, a mainland rock called the "Pebble" , south of Noah's Ark , Roman Rock and Millers Point.With economics playing an important role and ease of access, the plans were drawn up in May 1844 for a Pharos, or lighthouse to be built on the farm Millers Point. The prevailing opinion amongst Naval Officers at the station was that a light at Millers Point would be of little use. The essence of the case is that Roman Rock, awash at low tide and covered at high water springs, is the greatest hazard nearest to Simons Bay.So Millers Point was not chosen as the site for the lighthouse as we know it today.In years to come Millers Point would be synonymous with boating of a different nature.

In 1814 Tomas Drury was first granted use of this land site for cattle. It was later sold to Edmund Miller and inherited by his son James William Miller in 1838.Whaling played an important part on this site during this period, their valuable oil being used to provide light for the newly constructed lighthouses at Cape Point and Roman rock.The Millers Point property belonged to the Molteno family for many years with the homested and dairy farm we now know as the Black Marlin restaurant.