Troubleshooting

Common issues and their solutions when running Anubis.

Configuration Issues

Policy File Not Loading

Symptom: Anubis uses default policy instead of custom policy file. Check:

# Verify policy file exists
ls -l /etc/anubis/policy.yaml

# Check Anubis logs
anubis --policy-fname /etc/anubis/policy.yaml 2>&1 | grep "loading policy"

Common causes:

File path wrong:

# Wrong
--policy-fname policy.yaml  # Relative path

# Correct
--policy-fname /etc/anubis/policy.yaml  # Absolute path

File permissions:

# Fix permissions
chmod 644 /etc/anubis/policy.yaml
chown anubis:anubis /etc/anubis/policy.yaml

YAML syntax errors:

# Validate YAML
python3 -c 'import yaml; yaml.safe_load(open("/etc/anubis/policy.yaml"))'

Regular Expression Errors

Symptom: Error on startup:

config.Bot: invalid user agent regex: error parsing regexp: ...

Common issues:

Trailing newline (ErrRegexEndsWithNewline):

# Wrong - includes newline
user_agent_regex: >
  Mozilla.*

# Correct - use >-
user_agent_regex: >-
  Mozilla.*

Unescaped special characters:

# Wrong - . matches any character
user_agent_regex: "example.com"

# Correct - escape the dot
user_agent_regex: "example\\.com"

Invalid regex syntax:

# Wrong - unclosed group
user_agent_regex: "(bot"

# Correct
user_agent_regex: "(bot)"

Test regex:

Use regex101.com (select Golang flavor)
Test with --debug-benchmark-js flag

Configuration Validation Errors

Symptom: Startup fails with validation error.

No Bot Rules Defined

config: must define at least one (1) bot rule

Fix: Add at least one bot rule:

bots:
  - name: default
    user_agent_regex: ".*"
    action: ALLOW

Bot Must Have Name

config.Bot: must set name

Fix: Every rule needs a name:

bots:
  - name: my-rule  # Required
    user_agent_regex: "bot"
    action: DENY

Bot Must Have Matcher

config.Bot: must set either user_agent_regex, path_regex, headers_regex, or remote_addresses

Fix: Rules need at least one matcher:

# Wrong - no matcher
- name: empty-rule
  action: DENY

# Correct - has matcher
- name: bots
  user_agent_regex: "bot"
  action: DENY

Invalid CIDR

config.Bot: invalid CIDR: ...

Fix: Use proper CIDR notation:

# Wrong
remote_addresses:
  - 192.168.1.1  # Missing /mask

# Correct
remote_addresses:
  - 192.168.1.0/24
  - 10.0.0.0/8

Storage Issues

BoltDB Permission Denied

Symptom:

can't open bbolt database /data/anubis.bdb: permission denied

Fix: Check directory permissions:

# Verify directory exists and is writable
ls -ld /data
sudo chown anubis:anubis /data
sudo chmod 750 /data

# Test write access
sudo -u anubis touch /data/test && rm /data/test

BoltDB File Lock

Symptom:

can't open bbolt database /data/anubis.bdb: timeout

Cause: Another Anubis instance has the database locked. Fix:

# Find process holding lock
lsof /data/anubis.bdb

# Stop conflicting instance
sudo systemctl stop anubis

# Or kill process
kill <PID>

Valkey Connection Failed

Symptom:

valkey.Factory: ping failed: dial tcp: connection refused

Check:

# Test Valkey connectivity
redis-cli -h valkey.example.com -p 6379 ping
# Expected: PONG

# Check network
telnet valkey.example.com 6379

Common causes:

Wrong URL:

# Wrong protocol
url: http://valkey:6379

# Correct
url: redis://valkey:6379/0

Network policy blocking:
- Check firewall rules
- Verify Kubernetes NetworkPolicy
- Check security groups (cloud)

Authentication required:

# Add credentials to URL
url: redis://username:password@valkey:6379/0

S3 Access Denied

Symptom:

can't put s3 object: AccessDenied: Access Denied

Check:

# Verify environment variables
echo $AWS_ACCESS_KEY_ID
echo $AWS_SECRET_ACCESS_KEY
echo $AWS_REGION

# Test AWS credentials
aws s3 ls s3://anubis-data/

Fix: Update IAM policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:HeadObject"
      ],
      "Resource": "arn:aws:s3:::anubis-data/*"
    }
  ]
}

Runtime Issues

Challenges Not Working

Symptom: Users see challenge page but it doesn’t solve. Check browser console for JavaScript errors. Common causes:

JavaScript disabled: Use metarefresh algorithm
CORS issues: Check proxy configuration
Ad blocker: Whitelist Anubis
CSP headers: Ensure challenge scripts can run

Users Keep Getting Challenged

Symptom: Users solve challenge but are challenged again on next request. Causes:

Cookies disabled: Check browser settings

Cookie domain mismatch:

# Wrong - too broad
--cookie-domain .com

# Correct
--cookie-domain example.com

Signing key changed: Users need to re-solve after key rotation
JWT expired: Check --cookie-expiration-time
IP restriction enabled: Mobile users change IPs

Debug:

# Check cookie in browser DevTools
# Application > Cookies > anubis-auth

# Verify JWT
curl -v http://localhost:8923/
# Look for Set-Cookie header

High Memory Usage

Symptom: Anubis process using excessive memory. Check:

# View memory usage
ps aux | grep anubis

# Get Go memory stats
curl http://localhost:9090/metrics | grep go_memstats

Causes:

Memory storage backend: No size limits
- Fix: Use bbolt or valkey

DNS cache growth: Large cache with high DNS TTL

Fix: Lower DNS TTL:

dns_ttl:
  forward: 60   # Lower from 300
  reverse: 60

Goroutine leak: Check goroutine count

curl http://localhost:9090/metrics | grep go_goroutines

High CPU Usage

Symptom: Anubis using 100% CPU. Causes:

Challenge difficulty too high:

# Lower difficulty
challenge:
  difficulty: 2  # Instead of 8+

Too many requests: Scale horizontally
Expensive CEL expressions: Optimize rules

Profile CPU:

# Enable profiling (development only)
go tool pprof http://localhost:9090/debug/pprof/profile

Network Issues

Client IP Always Shows Internal IP

Symptom: X-Real-IP shows proxy IP, not client IP. Causes:

Missing X-Forwarded-For header: Proxy not configured
Using --use-remote-address: Only for bare metal

Fix: Configure proxy to set headers:

# Nginx
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

# Traefik (labels)
traefik.http.middlewares.anubis-headers.headers.customrequestheaders.X-Real-IP=<clientip>

DNS Lookups Failing

Symptom: verifyFCrDNS() always fails. Check:

# Test DNS from container
dig +short google.com @8.8.8.8

# Check DNS cache
curl http://localhost:9090/metrics | grep dns

Causes:

DNS server unreachable: Check /etc/resolv.conf
Firewall blocking port 53: Allow UDP/TCP 53
DNS cache poisoned: Restart Anubis

Performance Issues

Slow Challenge Response

Symptom: Challenge page takes seconds to load. Causes:

Storage backend latency: Check backend health
DNS lookups in expressions: Cache results
Complex CEL expressions: Simplify rules

Benchmark:

# Test challenge performance
time curl -A "Mozilla/5.0" http://localhost:8923/

High Latency

Symptom: All requests slow through Anubis. Causes:

Backend unreachable: Check target server
TLS handshake slow: Use --target-insecure-skip-verify (dev only)
Storage backend slow: Monitor storage metrics

Profile:

# Check target connectivity
time curl http://backend:3000/

# Test without Anubis
time curl --resolve example.com:80:backend-ip http://example.com/

Key and JWT Issues

Generating Random Key Warning

Symptom:

WARN generating random key, Anubis will have strange behavior when multiple instances are behind the same load balancer target

Impact: Challenges invalidated on restart. Fix: Configure signing key:

# Generate key
openssl rand -hex 32 > /etc/anubis/ed25519.key

# Use it
export ED25519_PRIVATE_KEY_HEX_FILE=/etc/anubis/ed25519.key

Key Validation Failed

Symptom:

failed to parse and validate ED25519_PRIVATE_KEY_HEX: supplied key is not 32 bytes long, got X bytes

Fix: Key must be exactly 64 hex characters:

# Wrong length
ED25519_PRIVATE_KEY_HEX=abc123  # Only 6 hex chars = 3 bytes

# Correct length
ED25519_PRIVATE_KEY_HEX=$(openssl rand -hex 32)  # 64 hex chars = 32 bytes

JWT Signature Invalid

Symptom: Users challenged every request despite having cookie. Causes:

Key changed: Signing key rotated
Multi-instance with different keys: Each instance using random key
Clock skew: JWT nbf (not before) in future

Fix:

Use same key across all instances
Sync system clocks (NTP)

Container/Kubernetes Issues

Container Exits Immediately

Symptom: Container starts then exits. Check logs:

# Docker
docker logs <container-id>

# Kubernetes
kubectl logs <pod-name>

Common causes:

Missing target: --target not set or invalid
Configuration error: Invalid policy file
Port already bound: Another process using port 8923

Health Check Failing

Symptom: Kubernetes pod stuck in CrashLoopBackOff. Check:

kubectl describe pod <pod-name>
# Look for: Liveness probe failed

# Test health check manually
kubectl exec <pod-name> -- /usr/local/bin/anubis --healthcheck

Fix:

Increase initialDelaySeconds (allow more startup time)
Increase timeoutSeconds (slow storage backend)
Check metrics port binding

Secret Not Mounted

Symptom:

failed to read ED25519_PRIVATE_KEY_HEX_FILE /secrets/ed25519.key: no such file or directory

Check:

# Verify secret exists
kubectl get secret anubis-key

# Check if mounted
kubectl exec <pod-name> -- ls -l /secrets/

Fix: Verify volume mount:

volumeMounts:
  - name: signing-key
    mountPath: /secrets
    readOnly: true
volumes:
  - name: signing-key
    secret:
      secretName: anubis-key  # Must match

Debugging Tips

Enable Debug Logging

# Via flag
anubis --slog-level DEBUG

# Via policy
logging:
  sink: stdio
  level: DEBUG

Test Mode

# Force all requests to show challenge (no blocking)
anubis --debug-benchmark-js

Validate Configuration

Anubis validates config on startup. Watch for errors:

# Run in foreground to see startup logs
anubis --policy-fname /etc/anubis/policy.yaml 2>&1 | less

Check Prometheus Metrics

# View all metrics
curl http://localhost:9090/metrics

# Check specific metric
curl -s http://localhost:9090/metrics | grep anubis_policy_results

# Monitor in real-time
watch -n 1 'curl -s http://localhost:9090/metrics | grep anubis_policy_results'

Trace Request Flow

# Enable verbose logging
export SLOG_LEVEL=DEBUG

# Make request with identifiable user agent
curl -A "TEST-REQUEST-$(date +%s)" http://localhost:8923/

# Find in logs
grep "TEST-REQUEST" /var/log/anubis/anubis.log

Getting Help

If you’re still stuck:

Check logs: --slog-level DEBUG
Review config: Validate YAML syntax
Test components: Storage, DNS, target server
Simplify: Remove complex rules, test with minimal config
Report issue: https://github.com/TecharoHQ/anubis/issues

Include:

Anubis version (anubis --version)
Configuration (redact secrets)
Error messages and logs
Steps to reproduce

Next Steps

Monitoring - Set up metrics and alerts
Security - Key generation and rotation
Policy Configuration - Validate and optimize policy

Documentation Index

​Configuration Issues

​Policy File Not Loading

​Regular Expression Errors

​Configuration Validation Errors

​No Bot Rules Defined

​Bot Must Have Name

​Bot Must Have Matcher

​Invalid CIDR

​Storage Issues

​BoltDB Permission Denied

​BoltDB File Lock

​Valkey Connection Failed

​S3 Access Denied

​Runtime Issues

​Challenges Not Working

​Users Keep Getting Challenged

​High Memory Usage

​High CPU Usage

​Network Issues

​Client IP Always Shows Internal IP

​DNS Lookups Failing

​Performance Issues

​Slow Challenge Response

​High Latency

​Key and JWT Issues

​Generating Random Key Warning

​Key Validation Failed

​JWT Signature Invalid

​Container/Kubernetes Issues

​Container Exits Immediately

​Health Check Failing

​Secret Not Mounted

​Debugging Tips

​Enable Debug Logging

​Test Mode

​Validate Configuration

​Check Prometheus Metrics

​Trace Request Flow

​Getting Help

​Next Steps

Configuration Issues

Policy File Not Loading

Regular Expression Errors

Configuration Validation Errors

No Bot Rules Defined

Bot Must Have Name

Bot Must Have Matcher

Invalid CIDR

Storage Issues

BoltDB Permission Denied

BoltDB File Lock

Valkey Connection Failed

S3 Access Denied

Runtime Issues

Challenges Not Working

Users Keep Getting Challenged

High Memory Usage

High CPU Usage

Network Issues

Client IP Always Shows Internal IP

DNS Lookups Failing

Performance Issues

Slow Challenge Response

High Latency

Key and JWT Issues

Generating Random Key Warning

Key Validation Failed

JWT Signature Invalid

Container/Kubernetes Issues

Container Exits Immediately

Health Check Failing

Secret Not Mounted

Debugging Tips

Enable Debug Logging

Test Mode

Validate Configuration

Check Prometheus Metrics

Trace Request Flow

Getting Help

Next Steps