The three checkboxes next to the regex fields (/i, /g, /x) are called “modifiers” as they modify the behaviour of the match pattern in various ways.
Normally a match pattern is case-sensitive. When /i is used literal letters in the regex (that is, not escaped operator characters) will match either case. Operators, such as \w & \W, still have different meanings.
If a pattern can match text within filename more than once, it will normally only replace the first match it finds. If \g is checked the regex will replace as many times as possible. If you use a match pattern that can only match once (for example, if you anchor the match to the beginning or end using ^ or $) then this modifier will have no effect.
In most implementations of regular expressions, /x enables various special extensions. In RegexRenamer the only relevant changes are to ignore unescaped whitespace and allow # comments at the end of the line. Whitespace (which normally means spaces, tabs, newlines, etc) in this case is limited to spaces. Normally the space and # characters in a match pattern will match themselves. When /x is checked, spaces are ignored unless they are escaped (“\ ”, that’s backslash-space) or inside a character class (“[ab cd]”). Likewise, the # character will cause itself and any following characters to be ignored unless it is also escaped (“\#”). This enables you to space out a complex regex to make it easier to read, and add comments to the end.