The Hay Festival had accused the governor of failing to protect freedom of expression and of inaction over violence against journalists.
It said 300 writers around the world had called for the cancellation.
Veracruz is one of the states in Mexico most affected by organised crime and violence against journalists.
In January, Moises Sanchez became the 15th journalist to be killed or to disappear in the state since 2010.
The decapitated body of the activist and journalist from Medellin de Bravo was found three weeks after he was kidnapped from his home by armed men.
A prosecutor said a former police officer confessed to taking part in Mr Sanchez's murder on orders from a local mayor.
The Hay Festival called for Mr Duarte to tackle the violence against journalists.
"Since Mr Duarte started in government in 2010 in Veracruz eleven journalists have been murdered and four more have disappeared," it said in a statement.
"A festival of culture and freedom like the Hay Festival cannot take place in an environment of violence against culture and freedom."
But Veracruz Governor Duarte de Ochoa defended Xalapa, saying it was less dangerous than some other sites where the Hay Festival takes place. He cited Beirut, claiming that "violence there was not comparable with rates in Veracruz".
He also said he would ensure a festival did take place in Xalapa so that the local economy did not suffer, though he did not give details.
The Hay Festival says it will hold its event online in October.