Gilmore was born in Ballygar, County Galway. He started his music career at age fifteen, and spent time in Canada with an English band. He settled in Boston, Massachusetts in 1848, becoming leader of the Suffolk, Boston Brigade, and Salem bands in swift succession. He also worked in the Boston music store of John P. Ordway and founded Ordway's Aeolians, a group of blackface minstrels. With the Salem Band, Gilmore performed at the 1857 inauguration of President James Buchanan.