I think there is a necessary discontinuity between a self-modifying solver algorithm that acts within an internally-consistent formal framework; and an intelligence that is able to modify the formal system itself. On some level this might be considered analogous to self-awareness. Past that discontinuity though, my guess is that all bets are off. There are many historical examples of the depths of madness that may be plumbed in unfulfilled pursuit of a formal goal.

To your second point, absolutely agreed: http://infoproc.blogspot.com/2012/03/only-he-was-fully-awake...