Note drifts with TI using VC

[L.F.]

When using the Ti in Cubase sx it sometimes feels like the notes slightly drift. This has nothing to do with the DSP load, and is mostly felt on the bassline and other fast attack sounds.
I've also noticed that lowering the latency lowers the drift, but, as you know, it's hard to keep a project at 2ms latency for long.. a few too many effects and clicks start occurring.
Is anyone else having similar problems? Any solutions found?

Setup:
E-MU 1212m
SX3
virus Ti v3
XinXP SP3
c2d@3.5GHz
2GB DDR2