In 1849 he moved to Edinburgh, where he met Mary Foley (1837–1920). They were married on 31 July 1855. She was the daughter of William Foley (1804–1841), of Lismore, County Waterford, and Catherine Pack, daughter of William Percy Pack (first cousin of Major-General Sir Denis Pack) and his wife Catherine Scott, whose great uncle was The 1st Earl of Clonmell. Mrs Doyle's mother, Catherine Pack, had been disinherited by her family on converting to Catholicism for her marriage to Mr Foley. When Mr Foley died at Clonmel, Mrs Doyle's mother had returned to her native Kilkenny, but not relinquishing her new faith.

She started her own girls' school there, which was taught principally in French. Either in consequence of the Irish Potato Famine or that her Protestant family had made life intolerable for her there, Mrs Doyle's mother gave up her school and sold her property in Kilkenny, moving to Edinburgh. There, she set up an institution supplying British and Foreign governesses to families and schools. Charles and Mary met in Edinburgh, and were the parents of several children including Sir Arthur Conan Doyle, creator of Sherlock Holmes, John Francis Innes Hay Doyle (known as Innes or Duff), and Jane Adelaide Rose Foley née Doyle (known as Ida).

Doyle was not as successful an artist as he wished, and suffered depression and alcoholism. His paintings, which were generally of fairies, such as In the shade or A Dance Around The Moon, or similar fantasy scenes, reflected this, becoming more macabre over time.