One big problem you might have is the files you have saved. If you always save your file with backwards compatibility then you are ok.
I've had this problem with Office 2010. I did a database for the Mrs work but she couldn't use it because they are on Office 2007 and it wont load in Access 2010 files, and there is no way of saving the files in an older version.
With Word and Excel they both have options to save the files as older versions, I would check all your files first and save them as "office 2003" before downgrading.
But I agree with Biffo that Office itself shouldnt slow your PC down.
Did you buy it or is it a downloaded version and if you downloaded it do you know if it's 100% clean.