Troubleshooting

Common issues and their solutions when running Anubis.

Configuration Issues

Policy File Not Loading

Symptom: Anubis uses default policy instead of custom policy file.

Check:

# Verify policy file exists
ls -l /etc/anubis/policy.yaml

# Check Anubis logs
anubis --policy-fname /etc/anubis/policy.yaml 2>&1 | grep "loading policy"

Common causes:

1. File path wrong:

   # Wrong
   --policy-fname policy.yaml  # Relative path
   
   # Correct
   --policy-fname /etc/anubis/policy.yaml  # Absolute path
   

2. File permissions:

   # Fix permissions
   chmod 644 /etc/anubis/policy.yaml
   chown anubis:anubis /etc/anubis/policy.yaml
   

3. YAML syntax errors:

   # Validate YAML
   python3 -c 'import yaml; yaml.safe_load(open("/etc/anubis/policy.yaml"))'
   

Regular Expression Errors

Symptom: Error on startup:

config.Bot: invalid user agent regex: error parsing regexp: ...

Common issues:

1. Trailing newline (ErrRegexEndsWithNewline):

   # Wrong - includes newline
   user_agent_regex: >
     Mozilla.*
   
   # Correct - use >-
   user_agent_regex: >-
     Mozilla.*
   

2. Unescaped special characters:

   # Wrong - . matches any character
   user_agent_regex: "example.com"
   
   # Correct - escape the dot
   user_agent_regex: "example\\.com"
   

3. Invalid regex syntax:

   # Wrong - unclosed group
   user_agent_regex: "(bot"
   
   # Correct
   user_agent_regex: "(bot)"
   

Test regex:

• Use regex101.com (select Golang flavor)
• Test with --debug-benchmark-js flag

Configuration Validation Errors

Symptom: Startup fails with validation error.

No Bot Rules Defined

config: must define at least one (1) bot rule

Fix: Add at least one bot rule:

bots:
  - name: default
    user_agent_regex: ".*"
    action: ALLOW

Bot Must Have Name

config.Bot: must set name

Fix: Every rule needs a name:

bots:
  - name: my-rule  # Required
    user_agent_regex: "bot"
    action: DENY

Bot Must Have Matcher

config.Bot: must set either user_agent_regex, path_regex, headers_regex, or remote_addresses

Fix: Rules need at least one matcher:

# Wrong - no matcher
- name: empty-rule
  action: DENY

# Correct - has matcher
- name: bots
  user_agent_regex: "bot"
  action: DENY

Invalid CIDR

config.Bot: invalid CIDR: ...

Fix: Use proper CIDR notation:

# Wrong
remote_addresses:
  - 192.168.1.1  # Missing /mask

# Correct
remote_addresses:
  - 192.168.1.0/24
  - 10.0.0.0/8

Storage Issues

BoltDB Permission Denied

Symptom:

can't open bbolt database /data/anubis.bdb: permission denied

Fix: Check directory permissions:

# Verify directory exists and is writable
ls -ld /data
sudo chown anubis:anubis /data
sudo chmod 750 /data

# Test write access
sudo -u anubis touch /data/test && rm /data/test

BoltDB File Lock

Symptom:

can't open bbolt database /data/anubis.bdb: timeout

Cause: Another Anubis instance has the database locked.

Fix:

# Find process holding lock
lsof /data/anubis.bdb

# Stop conflicting instance
sudo systemctl stop anubis

# Or kill process
kill <PID>

Valkey Connection Failed

Symptom:

valkey.Factory: ping failed: dial tcp: connection refused

Check:

# Test Valkey connectivity
redis-cli -h valkey.example.com -p 6379 ping
# Expected: PONG

# Check network
telnet valkey.example.com 6379

Common causes:

1. Wrong URL:

   # Wrong protocol
   url: http://valkey:6379
   
   # Correct
   url: redis://valkey:6379/0
   

2. Network policy blocking:

   • Check firewall rules
   • Verify Kubernetes NetworkPolicy
   • Check security groups (cloud)

3. Authentication required:

   # Add credentials to URL
   url: redis://username:password@valkey:6379/0
   

S3 Access Denied

Symptom:

can't put s3 object: AccessDenied: Access Denied

Check:

# Verify environment variables
echo $AWS_ACCESS_KEY_ID
echo $AWS_SECRET_ACCESS_KEY
echo $AWS_REGION

# Test AWS credentials
aws s3 ls s3://anubis-data/

Fix: Update IAM policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:HeadObject"
      ],
      "Resource": "arn:aws:s3:::anubis-data/*"
    }
  ]
}

Runtime Issues

Challenges Not Working

Symptom: Users see challenge page but it doesn’t solve.

Check browser console for JavaScript errors.

Common causes:

1. JavaScript disabled: Use metarefresh algorithm
2. CORS issues: Check proxy configuration
3. Ad blocker: Whitelist Anubis
4. CSP headers: Ensure challenge scripts can run

Users Keep Getting Challenged

Symptom: Users solve challenge but are challenged again on next request.

Causes:

1. Cookies disabled: Check browser settings

2. Cookie domain mismatch:

   # Wrong - too broad
   --cookie-domain .com
   
   # Correct
   --cookie-domain example.com
   

3. Signing key changed: Users need to re-solve after key rotation

4. JWT expired: Check --cookie-expiration-time

5. IP restriction enabled: Mobile users change IPs

Debug:

# Check cookie in browser DevTools
# Application > Cookies > anubis-auth

# Verify JWT
curl -v http://localhost:8923/
# Look for Set-Cookie header

High Memory Usage

Symptom: Anubis process using excessive memory.

Check:

# View memory usage
ps aux | grep anubis

# Get Go memory stats
curl http://localhost:9090/metrics | grep go_memstats

Causes:

1. Memory storage backend: No size limits

   • Fix: Use bbolt or valkey

2. DNS cache growth: Large cache with high DNS TTL

   • Fix: Lower DNS TTL:
     dns_ttl:
       forward: 60   # Lower from 300
       reverse: 60
     

3. Goroutine leak: Check goroutine count

   curl http://localhost:9090/metrics | grep go_goroutines
   

High CPU Usage

Symptom: Anubis using 100% CPU.

Causes:

1. Challenge difficulty too high:

   # Lower difficulty
   challenge:
     difficulty: 2  # Instead of 8+
   

2. Too many requests: Scale horizontally

3. Expensive CEL expressions: Optimize rules

Profile CPU:

# Enable profiling (development only)
go tool pprof http://localhost:9090/debug/pprof/profile

Network Issues

Client IP Always Shows Internal IP

Symptom: X-Real-IP shows proxy IP, not client IP.

Causes:

1. Missing X-Forwarded-For header: Proxy not configured
2. Using --use-remote-address: Only for bare metal

Fix: Configure proxy to set headers:

# Nginx
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

# Traefik (labels)
traefik.http.middlewares.anubis-headers.headers.customrequestheaders.X-Real-IP=<clientip>

DNS Lookups Failing

Symptom: verifyFCrDNS() always fails.

Check:

# Test DNS from container
dig +short google.com @8.8.8.8

# Check DNS cache
curl http://localhost:9090/metrics | grep dns

Causes:

1. DNS server unreachable: Check /etc/resolv.conf
2. Firewall blocking port 53: Allow UDP/TCP 53
3. DNS cache poisoned: Restart Anubis

Performance Issues

Slow Challenge Response

Symptom: Challenge page takes seconds to load.

Causes:

1. Storage backend latency: Check backend health
2. DNS lookups in expressions: Cache results
3. Complex CEL expressions: Simplify rules

Benchmark:

# Test challenge performance
time curl -A "Mozilla/5.0" http://localhost:8923/

High Latency

Symptom: All requests slow through Anubis.

Causes:

1. Backend unreachable: Check target server
2. TLS handshake slow: Use --target-insecure-skip-verify (dev only)
3. Storage backend slow: Monitor storage metrics

Profile:

# Check target connectivity
time curl http://backend:3000/

# Test without Anubis
time curl --resolve example.com:80:backend-ip http://example.com/

Key and JWT Issues

Generating Random Key Warning

Symptom:

WARN generating random key, Anubis will have strange behavior when multiple instances are behind the same load balancer target

Impact: Challenges invalidated on restart.

Fix: Configure signing key:

# Generate key
openssl rand -hex 32 > /etc/anubis/ed25519.key

# Use it
export ED25519_PRIVATE_KEY_HEX_FILE=/etc/anubis/ed25519.key

Key Validation Failed

Symptom:

failed to parse and validate ED25519_PRIVATE_KEY_HEX: supplied key is not 32 bytes long, got X bytes

Fix: Key must be exactly 64 hex characters:

# Wrong length
ED25519_PRIVATE_KEY_HEX=abc123  # Only 6 hex chars = 3 bytes

# Correct length
ED25519_PRIVATE_KEY_HEX=$(openssl rand -hex 32)  # 64 hex chars = 32 bytes

JWT Signature Invalid

Symptom: Users challenged every request despite having cookie.

Causes:

1. Key changed: Signing key rotated
2. Multi-instance with different keys: Each instance using random key
3. Clock skew: JWT nbf (not before) in future

Fix:

1. Use same key across all instances
2. Sync system clocks (NTP)

Container/Kubernetes Issues

Container Exits Immediately

Symptom: Container starts then exits.

Check logs:

# Docker
docker logs <container-id>

# Kubernetes
kubectl logs <pod-name>

Common causes:

1. Missing target: --target not set or invalid
2. Configuration error: Invalid policy file
3. Port already bound: Another process using port 8923

Health Check Failing

Symptom: Kubernetes pod stuck in CrashLoopBackOff.

Check:

kubectl describe pod <pod-name>
# Look for: Liveness probe failed

# Test health check manually
kubectl exec <pod-name> -- /usr/local/bin/anubis --healthcheck

Fix:

1. Increase initialDelaySeconds (allow more startup time)
2. Increase timeoutSeconds (slow storage backend)
3. Check metrics port binding

Secret Not Mounted

Symptom:

failed to read ED25519_PRIVATE_KEY_HEX_FILE /secrets/ed25519.key: no such file or directory

Check:

# Verify secret exists
kubectl get secret anubis-key

# Check if mounted
kubectl exec <pod-name> -- ls -l /secrets/

Fix: Verify volume mount:

volumeMounts:
  - name: signing-key
    mountPath: /secrets
    readOnly: true
volumes:
  - name: signing-key
    secret:
      secretName: anubis-key  # Must match

Debugging Tips

Enable Debug Logging

# Via flag
anubis --slog-level DEBUG

# Via policy
logging:
  sink: stdio
  level: DEBUG

Test Mode

# Force all requests to show challenge (no blocking)
anubis --debug-benchmark-js

Validate Configuration

Anubis validates config on startup. Watch for errors:

# Run in foreground to see startup logs
anubis --policy-fname /etc/anubis/policy.yaml 2>&1 | less

Check Prometheus Metrics

# View all metrics
curl http://localhost:9090/metrics

# Check specific metric
curl -s http://localhost:9090/metrics | grep anubis_policy_results

# Monitor in real-time
watch -n 1 'curl -s http://localhost:9090/metrics | grep anubis_policy_results'

Trace Request Flow

# Enable verbose logging
export SLOG_LEVEL=DEBUG

# Make request with identifiable user agent
curl -A "TEST-REQUEST-$(date +%s)" http://localhost:8923/

# Find in logs
grep "TEST-REQUEST" /var/log/anubis/anubis.log

Getting Help

If you’re still stuck:

1. Check logs: --slog-level DEBUG
2. Review config: Validate YAML syntax
3. Test components: Storage, DNS, target server
4. Simplify: Remove complex rules, test with minimal config
5. Report issue: https://github.com/TecharoHQ/anubis/issues

Include:

• Anubis version (anubis --version)
• Configuration (redact secrets)
• Error messages and logs
• Steps to reproduce

Next Steps

• Monitoring - Set up metrics and alerts
• Security - Key generation and rotation
• Policy Configuration - Validate and optimize policy