The original headquarters of HBC operations on the Pacific Coast of North America at the time of Victoria's founding was Fort Vancouver (now Vancouver, Washington) on the lower Columbia River, but it had struggled for years to turn a profit; its location was difficult to defend, inaccessible to ships and too far from the lucrative furs in New Caledonia farther north. With American settlers pouring into the region, in 1843 the company sent James Douglas to build a fort some distance north on Vancouver Island and made him its Chief Factor. The signing of the Oregon Treaty in 1846 settled the matter of Fort Vancouver’s further suitability as the United Kingdom gave up its claims in the area to the United States and in return kept Vancouver Island while the Americans dropped their claims north of the 49th parallel.

Erected in 1843 on a site originally called Camosun (a variant of the Lekwungen word "Camossung", the name of a girl turned into stone by the spiritual being Hayls, the Transformer to watch over the resources in what is now known as the Gorge waterway.[2]) The fort was known briefly as "Fort Albert", though the name Fort Camosun continued in use until 1846, when it was renamed in honour of the young Queen.[3][4] The Fort was built using labour from local First Nations people, who were paid one Hudson’s Bay blanket for every 40 pickets they cut.[5] The Songhees people soon established a village across the harbour from the fort. The Songhees' village was later moved to the north shore of Esquimalt Harbour. In 1844, a dispute led to a gathering of warriors led by Tzouhalem, a Quamichan chief, who peppered the fort with threats and musket balls for two days until the fort's Chief Factor Roderick Finlayson demonstrated the power of the fort's nine-pounder cannon.[6]

In 1849, the crown Colony of Vancouver Island was established and the HBC was granted exclusive proprietary rights over Vancouver Island. The condition imposed by the Colonial Office was that the company would establish a settlement within five years or see their grant revoked. It was also to spend ninety percent of what it made on land sales on infrastructure such as roads and schools. A town was laid out on the site and made the capital of the colony. London sent Richard Blanshard to be its governor.

Even as the settlement began to grow, the nature of the company’s business was changing. Animal populations were beginning to dwindle from overtrapping, slowing the fur trade, but the California gold rush created a huge demand for resources with few places to buy them on the unsettled west coast. By 1850 there were several sawmills operating at Victoria to feed the hungry California market. The company was soon trading salted salmon with Hawaii and outfitting Royal Navy ships with supplies for the Crimean war.

James Douglas, who remained Chief Factor of the fort, was made the second governor of the Vancouver Island colony in 1851, and would be the leading figure in the early development of the city until his retirement in 1864, and of the Mainland Colony from 1858-1862 when he was replaced as governor of that colony by Frederick Seymour.