Maybe try to give it marked pairs of rhyming words to learn (close together) in addition to the poems. Or even make it learn IPA notation of those words as a side-task.
I'm no expert but I used char-nn as a fantasy name generator "similar to those 300 names I have". 300 names is not enough to learn from, so I augmented it with other sources: standard German and English names, cities and dictionary words. My input looked like this:
The RNN easily learned to generate matching tags (most of the time), and to distinguish between the different statistics inside, even for the rare tags I was after. I think this trick worked pretty well. Here is my data: http://log2.ch/misc/namegen/
