Here is a link to the original paper.
This was a formal project I did along with Sudeep Katakol in my last active semester of college, under supervision of Tirtharaj Dash. We were trying to improve on PerformanceRNN. Wrestling around with the architecture, intermediate layers and/or removing and adding linkages to the memory bank, none of it seemed to work that well. We presented our findings and code so far and left the project to be continued by the upcoming batch of students.
I do not have the permission to share our code, since this work is currently being continued by others.