After a bit of digging and quite a bit of suggestions and advice from people (thanks guys it was all helpful stuff), I found the problem.
Basically the chip I use (an Atmel AVR) has a number of interrupts, and they have a ‘priority’, what was happening is that the timer interrupt would sometimes start and then an SPI receive would complete, now with an AVR when an interrupt is being serviced, all the others are suspended. This resulted in the SPI receive interrupt (which has a lower priority than the timer) from being ‘missed’, or even late.
This highlights the need for me to make my timer function a LOT shorter. The picture below shows the Timer 1 firing between the 2nd and 3rd byte being received, and finishing after. It also shows my “fix” (the 3rd byte is received this time.