That's a good question. I'm guessing children born to pure blood families are more likely to have magic than children born to muggles or one magic and one not. If magic is a trait then it would be like any other trait. It's probably recessive because there aren't as many magical people as non-magical. So the children born to muggles with magic had a very low percentage of having magic and the pure bloods have a higher percentage but it is still possible not to get magic, just rare. If a muggle marries a pureblood the child would probably have a 50 or 75% chance. I don't know if they have like magical gene carriers or something. That's just what I think. I don't know that much about genetics so I might be way off.
I would say magic is the dominant trait, because otherwise muggleborns would be much, much rarer than they are, and two half bloods would have a fifty percent chance of having a squib, but squibs are rare. I would say that a squib is anyone with at least one magical parent who isn't magical themselves, as there is nothing in the book suggesting that Filch is a pureblood, and though she was wrong, the Weasley's Auntie Muriel thought that Ariana was a squib and Kendra a muggleborn, suggesting that it is entirely possible for a squib to be born of a muggleborn.
Now, I must confess my biology knowlegde is limited, but from what Ron 'explains' to Harry, when he tells him what a squib is, it seems that magic is usually the dominant trait (as evidenced by the many many halfbloods and the very little squibs) and if you're a squib that's like a 'defect'. It's some a disabled gene, if you know what I mean. And (though I could be wrong about that) it appears as though squibs can still pass the magic on, so they are not like muggles. The latter is just an educated guess though.
Magical abilities cannot be traced as either dominant or recessive. If it was dominant, there would be no muggleborns because the parents would have to display the trait if they had it to pass on to their children. If it's recessive, squibs would be way more common, and not as many halfbloods would have magical abilities. so it must be determined some other way than Mendellian genetics.
My biology teacher used Hermione as an example of genetics, assuming magic to be recessive. Her parents could both carry the magical gene, but they wouldn't display the trait, but as mentioned above that opens other questions.
He was a big Harry Potter fan; he also purchased the 7th movie, part 1 the day it came out before school, and we started watching it in class.