The Vatican is built on the site where St Peter was crucified and buried, said to be the most holy site in the world. A small shrine was built, then elaborated, then a church, and then before you know it, Vatican City had been built on the site of the burial.
The body of St. Peter still lies in a crypt below Vatican City.
Also, it was the Roman emperor, Constantine, that legalised Christianity in the Roman empire, so Rome would be a good place to base the religion that took over as the major religion from paganism over the first few centuries AD.