1871 census: James (20, born High Wycombe, Bucks) was a soldier in the District Sheffield Barracks, Nether Hallam, Yorkshire.

James enlisted for the 14th Regiment fo Foot on 13 September 1870 and served in the army for 21 years 132 days, leaving as a Sergeant on 26 January 1892. Between 6 October 1878 and 10 January 1892 he served in East India, where he married and had four ** children.
James's service record is online at findmypast.co.uk.
** the service record lists as children:
James Andrew 21.11.77
Margaret 29.6.80
William Henry 4.6.88
Gladys 12.7.90
but the birth dates of the first two are before the date given for James's marriage to Elizabeth Dick and, in the 1911 census, it was stated that Elizabeth had had only 3 children. So for the time being I'm assuming that James Andrew and Margaret were NOT James's children. It is of course possible that James married someone else before Elizabeth but if he did, the name doesn't appear on the service record.

A record at ancestry.com shows that James was admitted as a Chelsea Pensioner on 26 January 1892.

1901 census: at 41 Gordon Road, Aldershot, Hampshire (all WORLEY), James (head, 49, army pensioner, clerk OSC, born High Wycombe, Bucks), Elizabeth (wife, 44, born Essex), William H (son, 12, born India), Gladys M (daughter, 10, born India), Doris L (daughter, 8, born High Wycombe, Bucks). Also visitors Dudley CLINTON (33, actor, born Brighton, Sussex) and Edith L CLINTON (27, actress, born Blackheath, Kent), both married, probably to each other.
[Dudley Clinton was quite well-known: he died on 1 October 1908 aged 39, while on tour in Australia. His real name was J W E Gillam. There is less on the web about Edith Clinton but she appears to have outlived Dudley by many years, appearing in the 1939 film 'Men without Honour'.]