In the fights for the Hungarian and Croatian throne Duke John Hunyadi attacked the Counts of Celje in 1446 with the army of 15 000 Hungarians.
He burnt and pillaged all over the Podravina region, after which he occupied Varaždin without a fight and began the siege of the Counts of Celje fort in Varaždin. The fort was well protected and ready for the siege so it resisted Hunyadi's attacks. Enraged, Hunyadi burnt down Varaždin and began the conquest of other estates owned by the Counts of Celje in Styria and Međimurje.
The awful misfortune that hit Varaždin, along with the growing danger from the Turks, convinced the locals to start building the walls to protect the town from new attacks. It was a big venture, so they decided to protect at least the centre of their town.
Enormous earth dikes were built, and extra protection was ensured by ditches filled with water from the river Drava.
The only way to come to the centre behind the walls was from the north and the south, across long wooden bridges. There were square towers built for the protection of the bridges.
One of these towers, called Lisak's tower in Ban Jelačić's Square, was kept until today. Other fortifications that stretched along the Šenoina Street towards the city promenade and today's Zagrebačka Street in the south, where the Southern tower was built, along the Draškovićeva Street and Uršulinska Street to the Old Town in the west, and Bakačeva Street to the Banus market, were mostly torn down in the 19th century.