Steff,
Just checking that the default Qt5 package in Debian is not adequate.
I guess one could look at the qmake.conf file found at mini2440vietnam and try changing the more obvious lines, ie the tool chain directory and the march and mtune values to your processor. I am not sure about whether or not -msoft-float is needed.
This is likely to be a painful process, hope someone has already done it.
Good luck.