I do not really know what is the problem, but you should be able to do
var no_compile = _Roles.MyCustomFunction();
If _Roles defines a new MyCustomRoleProvider since the function is public
Example
MyCustomRoleProvider _Roles = new MyCustomRoleProvider();
var no_compile = _Roles.MyCustomFunction();
Notice that: You can not call MyCustomRoleProvider.MyCustomFunction() directly because it is not a public static string.
Notice that: System.Web.Security.Roles is a class which contains the same function name of your class IsUserInRole(string username, string roleName) but the function is not the same in both classes. So, you can not access Roles.MyCustomFunction() because the class System.Web.Security.Roles does not contain a definition for MyCustomFunction() and that's because you did not define a new function in the class
MyCustomRoleProvider and System.Web.Security.Roles are TWO different classes which have different functions

Notice that: You can not modify or add functions to System.Web.Security.Roles as it is write-protected

Thanks,
I hope you find this helpful :)