From your list, "Language" and then "other", specifically 'normal' food and the weather. Also the time to travel. I only go where they speak English and have 'English' food. This pretty much limits my holidays to the USA or the UK ... the USA usually winning out on the weather aspect (Australia being a good prospect too but a long way to travel so you need to take at least three weeks to make it worth while).
Cost is almost never an issue as I have enough free hotel nights and airmiles for one or tow holidays each year.