During my spare time recently, I've been testing several MMOs so I can create better suggestions for Runescape.
In every game that I've played in the past month, all had several of the same elements. They had hotbars, abilities, classes, and talent trees.
Runescape already had classes, and it obtained hotbars and abilities through EoC. As it stands, nothing really defines your class other than your weapon. In Runescape, you can DPS or Tank with any class, but what defines your role is what you are wielding and wearing. Jagex tried to define these roles through the recent armour update, but it's still not there yet.
In case you don't know what a Talent Tree is, it looks something like this:
The way a Talent Tree works is that you choose a Tree to start off in (Survival, Marksman, or Beast Master in the image). Once you choose a Tree, you cannot put talents in another Tree until you reach the bottom. In WoW, a Tree defines what role you will play. This cannot be the same for Runescape since we have 3 classes and 2 roles for each, and 6 Trees would over complicate things for most players.
So, the way a Talent Tree would be implemented in Runescape is by creating 3 Trees: Ranged, Melee, and Magic.
Unlike WoW, you are not locked to a single Tree. You will gain Talent Points for each Tree when you level up that skill. I would say that you begin getting Talent Points at level 6 of a skill, and you'll get another every 3 levels, maxing out at 31 Points for each Tree at level 99.
A majority of these Talents would be passive skills that would only work if you have a weapon or shield of the same style equipped. These could revolve around bonus damage, extra Crit, and damage reduction against another style.
To move down to the next set of Talents, you need to have at least 3 Talent Points in a lower tier for each row of Talents unlocked. Let's say you want to unlock the third row, you could have 4 Talents in the 1st row, and 2 Talents in the 2nd row, and still unlock the third row. Because there would be 31 Talents and each row requires 3, the final row, Level 99, would consist of two Talents, but you can choose only one. One would be for tanks, and the other for DPS.
As with any Talent Tree system, there will always be a "cookie-cutter" setup that will be superior in a DPS/Tank scenario. In a PvP scenario, you are free to mix and match any talents you want, so you could go bulky Ranged DPS, or offensive Mage Tank.
Jagex could do a lot with a Talent Tree system. They wouldn't have to wait till level 99 Weapons and Abilities to make a level 99 Talent. They could include a few skills that are unlocked through Talents. They could give each class a defining role, such as making Melee hit higher, making Ranged deal more bleeds/DoTs, or making Magic deal more crits.
Hybridding might also be revived since someone could switch to another style to try and make use of a certain talent.
Right now, this idea is just in it's conceptual stages. I have not thought of all the talents, but I have thought of a few. What are your thoughts on the matter?