Bot Policy Configuration

Bot detection rules and action configuration structs

Overview

Bot policy configuration defines rules for identifying and handling different types of traffic. Each rule specifies matching criteria and an action to take when traffic matches.

BotConfig

Defines a bot detection rule with matching criteria and action.

Type Definition

type BotConfig struct {
    UserAgentRegex *string           `json:"user_agent_regex,omitempty" yaml:"user_agent_regex,omitempty"`
    PathRegex      *string           `json:"path_regex,omitempty" yaml:"path_regex,omitempty"`
    HeadersRegex   map[string]string `json:"headers_regex,omitempty" yaml:"headers_regex,omitempty"`
    Expression     *ExpressionOrList `json:"expression,omitempty" yaml:"expression,omitempty"`
    Challenge      *ChallengeRules   `json:"challenge,omitempty" yaml:"challenge,omitempty"`
    Weight         *Weight           `json:"weight,omitempty" yaml:"weight,omitempty"`
    GeoIP          *GeoIP            `json:"geoip,omitempty"`
    ASNs           *ASNs             `json:"asns,omitempty"`
    Name           string            `json:"name" yaml:"name"`
    Action         Rule              `json:"action" yaml:"action"`
    RemoteAddr     []string          `json:"remote_addresses,omitempty" yaml:"remote_addresses,omitempty"`
}

Fields

Name · string · required

Unique identifier for this bot rule. Used in logs and metrics.

Action · Rule · required

Action to take when traffic matches this rule. See Rule enum.

UserAgentRegex · *string

Regular expression to match against the User-Agent header. Cannot be used with PathRegex.

PathRegex · *string

Regular expression to match against the request path. Cannot be used with UserAgentRegex.

HeadersRegex · map[string]string

Map of header names to regular expressions. All specified headers must match for the rule to apply.

Expression · *ExpressionOrList

CEL (Common Expression Language) expression for advanced matching logic. See ExpressionOrList.

RemoteAddr · []string

List of CIDR ranges to match against the client IP address (e.g., "192.168.1.0/24").

Challenge · *ChallengeRules

Challenge configuration when Action is CHALLENGE. See challenge-rules.

Weight · *Weight

Weight adjustment when Action is WEIGH. If not set, defaults to {Adjust: 5}.

GeoIP · *GeoIP

Geographic IP matching configuration. See GeoIP.

ASNs · *ASNs

Autonomous System Number matching configuration. See ASNs.

Validation Rules

• Must set Name
• Must set at least one matching criterion: UserAgentRegex, PathRegex, HeadersRegex, RemoteAddr, Expression, GeoIP, or ASNs
• Cannot set both UserAgentRegex and PathRegex
• Regular expressions must not end with newline (use >- in YAML instead of >)
• RemoteAddr entries must be valid CIDR notation
• If Action is CHALLENGE, must set Challenge
• If Action is WEIGH and Weight is nil, defaults to {Adjust: 5}

Example

bots:
  - name: Block malicious crawlers
    user_agent_regex: >(MJ12bot|AhrefsBot|SemrushBot)
    action: DENY

  - name: Challenge suspicious bots
    expression: weight > 10
    action: CHALLENGE
    challenge:
      algorithm: fast
      difficulty: 20

  - name: Weigh Russian IPs
    geoip:
      countries: ["ru"]
    action: WEIGH
    weight:
      adjust: 15

Rule Enum

Defines actions to take when traffic matches a bot rule.

Type Definition

type Rule string

const (
    RuleUnknown   Rule = ""
    RuleAllow     Rule = "ALLOW"
    RuleDeny      Rule = "DENY"
    RuleChallenge Rule = "CHALLENGE"
    RuleWeigh     Rule = "WEIGH"
    RuleBenchmark Rule = "DEBUG_BENCHMARK"
)

Values

ALLOW · string

Immediately allow the request without further checks.

DENY · string

Block the request with a configured status code.

CHALLENGE · string

Present a proof-of-work challenge to the client. Requires Challenge configuration.

WEIGH · string

Add weight to the client’s score. When thresholds are exceeded, triggers challenges or blocks.

DEBUG_BENCHMARK · string

Debug action for performance testing. Logs timing without affecting traffic.

ExpressionOrList

Flexible expression matching supporting single expressions or logical combinations.

Type Definition

type ExpressionOrList struct {
    Expression string   `json:"-" yaml:"-"`
    All        []string `json:"all,omitempty" yaml:"all,omitempty"`
    Any        []string `json:"any,omitempty" yaml:"any,omitempty"`
}

Fields

Expression · string

Single CEL expression. Can be set directly as a string in YAML/JSON.

All · []string

List of CEL expressions that must all evaluate to true (logical AND).

Any · []string

List of CEL expressions where at least one must evaluate to true (logical OR).

Validation Rules

• Must set either Expression, All, or Any
• Cannot set both All and Any

Example

# Single expression
expression: "weight > 10"

# All expressions must match (AND)
expression:
  all:
    - "weight > 5"
    - "request.headers['User-Agent'].contains('bot')"

# Any expression can match (OR)
expression:
  any:
    - "geoip.country == 'CN'"
    - "geoip.country == 'RU'"

GeoIP

Geographic IP address matching based on country codes.

Type Definition

type GeoIP struct {
    Countries []string `json:"countries"`
}

Fields

Countries · []string · required

List of two-letter ISO 3166-1 alpha-2 country codes (case-insensitive).

Validation Rules

• Country codes must be exactly 2 letters matching [a-zA-Z]{2}
• Country codes are normalized to lowercase

Example

geoip:
  countries: ["cn", "ru", "kp"]

ASNs

Autonomous System Number (ASN) matching for network-based rules.

Type Definition

type ASNs struct {
    Match []uint32 `json:"match"`
}

Fields

Match · []uint32 · required

List of ASNs to match against the client’s network.

Validation Rules

• ASNs must not be in private use ranges (RFC 6996):
  • 64512-65534
  • 4200000000-4294967294

Example

asns:
  match: [15169, 16509]  # Google and Amazon

Weight

Score adjustment for traffic matching a WEIGH rule.

Type Definition

type Weight struct {
    Adjust int `json:"adjust" yaml:"adjust"`
}

Fields

Adjust · int · required

Integer value to add to the client’s weight score. Can be negative to reduce weight.

Example

weight:
  adjust: 10

ImportStatement

Import bot rules from external files.

Type Definition

type ImportStatement struct {
    Import string      `json:"import"`
    Bots   []BotConfig
}

Fields

Import · string · required

Path to YAML file containing bot rules. Use (data)/ prefix to load from embedded data.

Example

bots:
  - import: "(data)/botPolicies.yaml"
  - name: Custom rule
    user_agent_regex: "CustomBot"
    action: DENY

Errors

var (
    ErrNoBotRulesDefined                 = errors.New("config: must define at least one (1) bot rule")
    ErrBotMustHaveName                   = errors.New("config.Bot: must set name")
    ErrBotMustHaveUserAgentOrPath        = errors.New("config.Bot: must set either user_agent_regex, path_regex, headers_regex, or remote_addresses")
    ErrBotMustHaveUserAgentOrPathNotBoth = errors.New("config.Bot: must set either user_agent_regex, path_regex, and not both")
    ErrUnknownAction                     = errors.New("config.Bot: unknown action")
    ErrInvalidUserAgentRegex             = errors.New("config.Bot: invalid user agent regex")
    ErrInvalidPathRegex                  = errors.New("config.Bot: invalid path regex")
    ErrInvalidHeadersRegex               = errors.New("config.Bot: invalid headers regex")
    ErrInvalidCIDR                       = errors.New("config.Bot: invalid CIDR")
    ErrRegexEndsWithNewline              = errors.New("config.Bot: regular expression ends with newline (try >- instead of > in yaml)")
    ErrNotCountryCode                    = errors.New("config.Bot: invalid country code")
    ErrPrivateASN                        = errors.New("bot.ASNs: you have specified a private use ASN")
)