There is no support for ov5640 on kernel 4.4, you should use kernel 3.4 instead of 4.4.
Kernel 3.4 has support for CAM500B.
There was a preliminary support for ov5640 on kernel 4.14 but seems to be halted or postponed. I have tried to enable ov5640 support on 4.4 without much success.