Have you tried to optimize the frames?

[gogojjh]

Have you tried to optimize the past frames through the connection of mappoints? I think that Bundle Adjustment might be a good choice to the scale drift problem. The error will accumulate, and if the environment changes quickly, this situation will be worse.
By the way, why did you say that your algorithm can not calcullate the relative pose? R and t are calculated by the comparison from prevframe to currentframe.
Thank you!