Car making very strange noise, what could it be?

For the past few months my 1997 Subaru Legacy Brighton AWD has been having some trouble starting, when I turn the key to Start I hear a loud click from under the hood, and it doesnt start. Usually after about 5 or 6 tries it does start. Today though, I went to start it and It started fine in the morning and a few hours later I started it again and I started hearing the clicking but this time the engine was on and running and it sounded like the car was turning over, when it was already running! Then I immediately turned the key to OFF but car would not shut off. It just sputtered and sounded like it was going to stall but it would not shut off no matter what I did. I turned the key back to the ON position and the engine started right back up as if I had put the key in the START position! Could this be the work of a faulty starter? I touched the starter after I was able to shut the car off (after a few minutes) and it was very very hot. It was a very strange and nerve wracking experience and I hope someone has an answer for me. Thanks