Alright, it seems that the reason for the difference in temperature with and without the SD card might be due to the content on the card. I have a lot of PDFs, some very large and some with poor names. Android's indexing service is seen to be using 35% of the battery which would translate to the CPU being utilized heavily by this app. The CPU was between 50-60%. It even uses up more power than the screen.
I removed the large files and then cleaned up all the file names. I moved the files to the onboard storage and formatted the SD card. The indexing service is still running, but I believe this would take some time until it finishes. I will try to move the files, one at a time, back to the external SD card once this gets under control. The CPU is now around 42-45% and the temperature at 38C. Hopefully this should sort itself out soon.