Turn off your phone, then hold down Volume Up, Center Home and Power button for 5 seconds.
If you don’t see ClockworkMod Recovery but rooted, you can easily re-install using ROM Manager available on Play Store.
VN:F [1.9.17_1161]Sometimes, it may take a bit for your new custom ROM to settle down as when you first install a custom ROM and you re-load all your apps, they may try to simultaneously sync.
But heating up is pretty common with Galaxy S3 smartphones in general if you are using it heavily, especially with the U.S or Canadian variants.
If it keeps happening after a day, try another custom ROM or you can always unroot your phone to test it with stock ROM.
But it’s not going to fry your phone anything, so don’t worry