Created a metatable where __add and __sub take a table and an number.
__add and __sub take two operands. One of them will necessarily be a table or userdata which has your metatable, the other can be anything at all. Lua doesn't care about the type of the other operand.
If either operand has a metatable with a handler for the particular operator (+ -> __add, - => __sub), that handler will be called.
In your example, Lua not only doesn't care if n is positive or negative, it doesn't care if it's a number. The - in -n has nothing to do with the __sub metamethod -- that's the unary minus operator, whereas __sub handles the binary minus operator.
How does Lua determine which to use?
The + operator is an "__add" event. When you sayd op1 + op2, Lua checks op1 for a metatable with an __add handler. If it finds one, it calls it. Otherwise it checks op2.