I’ve just tested and the custom validation rule works. I think the problem might be the way you enqueue your script (that contains custom rule). Somehow, it’s enqueued before jQuery or jQuery validation script is output.
In my test, I use
wp_footer hook with priority 9999 to enqueue the JS, so it’s always after jQuery and jQuery validation. Please try and let me know how it goes.