Run mexext in your command window. If it does not return mexmac (or mexw32 which is also present in the folder) then you you won't be able to run the BoxQP function. These are simply old files - I see that they're from 2006. If you're using any form of OS X these days you'll get mexmaci64, which is not compatible. However, it looks like they have helpfully included all of the source code so you'll be able to hopefully compile the binaries for your system. For details on compiling mex code see this.
Then, as @alrikai suggests, your BoxQP function needs to be on your path. Either make sure that you call your code from the same folder or run path from the command window and check if that directory has been properly added (if you did so). See also addpath.