Grifasi was born in Buffalo, New York, the son of Patricia (née Gaglione) and Joseph J. Grifasi, a skilled laborer. Grifasi graduated from Bishop Fallon High School, a now-defunct Roman Catholic high school in Buffalo, when he made the decision that he wanted to be, not just any old actor, but a character actor. He played football and acted in many of the school's plays. Grifasi briefly attended Canisius College in Buffalo before joining the United States Army. He went on to study at the Yale School of Drama. While at the Yale School of Drama, he met his Future wife, jazz soprano-saxophonist Jane Ira Bloom; they were married in 1974.