System
This is it, the moment you have been waiting for, the math. You know the crunchy bits that make your game run. Mostly this is dice-based and has a few charts. You can decide how crunchy you want it to be. For ease of use we are going to split the system into 3 phases. Phase 1: dice and system. Phase 2: test and combat. Phase 3: character creation and advancement.
Your System: Phase One
When attempting to figure out your system, start with your goals in mind. Are you going to build an extensive system with long-term gameplay in mind or a simple system with short-term (one-shot type) play.
Dice/Math
While not necessary for an RPG they are the most common way of deciding the outcome of an action in a Table Top Role Playing Game. So we will assume that you are using some as the center of your system. You probably already know that dice are made in many different varieties and shapes. The different shape produce a random number in a different range. So a pyramid shaped 4 sided die produces a result between 1 and 4 with each possible result having a %25 chance of happening. Traditionally the 6 sided and 20 sided dice are the most common in use. If you are working on a game with a long character progression cycle then I suggest a larger die so that you will have smaller increases for each digit on a die.
Working Dice/Math
In Space Pirates the Musical, I chose to go with a 2d6 spread for the main part of the system with some smaller variations of d6 for damage and character creation. I like the steep bell of 2d6 with the center of the curve having the highest probability. For the light and simple system/game, I decided that the small amount of movement it gave was acceptable.
Resources
For this one, I highly recommend looking at what others have done before you. Only this time, look at the math that is involved in it. If you have problems with probability then check out some math text and read up.
Tips
If it ain’t broke, don’t fix it. Check other games and see what they did, then follow in their footsteps. You don’t need to steal from others, but most games have used the same dice and math for a reason. Make your own decision and use what you think works best.