I've thought about experimenting with RPG mechanics in the Dagon engine, but I got the feeling I'd be better off using a different engine alltogether if I ever make an RPG. It's truly an adventure game engine. Besides, battling and psychological horror don't mix that well in my opinion. When you kill something scary, the fear dies with it. This is why adventure games like Scratches are infinitely more horrifying then some Clive Barker games for example. I'm just not scared of something if I know I can chop its head(s) off.
Condemned had some brilliantly intense melee battling for sure even if you mostly battled hobos, but they where seperated from the truly scary parts and for good reason. If I remember correctly I felt the most unease when I had to use the non-lethal detective-equipment. Same with the game F.E.A.R. where the horror and the shooting never took place at the same time.
That said, some creepy encounters where the evil only moves when you're not looking at it could work really really well in the Dagon engine. You wouldn't expect something else moving or chasing you in a pre-rendered environment.