*** The new variable rmail-file-coding-system specifies the code
conversion to use for RMAIL files. The default value is nil.

When you read mail with Rmail, each message is decoded automatically into Emacs' internal format. This has nothing to do with rmail-file-coding-system. That variable controls reading and writing Rmail files themselves.