> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/TecharoHq/Anubis/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

Common issues and their solutions when running Anubis.

## Configuration Issues

### Policy File Not Loading

**Symptom**: Anubis uses default policy instead of custom policy file.

**Check**:

```bash theme={null}
# Verify policy file exists
ls -l /etc/anubis/policy.yaml

# Check Anubis logs
anubis --policy-fname /etc/anubis/policy.yaml 2>&1 | grep "loading policy"
```

**Common causes**:

1. **File path wrong**:
   ```bash theme={null}
   # Wrong
   --policy-fname policy.yaml  # Relative path

   # Correct
   --policy-fname /etc/anubis/policy.yaml  # Absolute path
   ```

2. **File permissions**:
   ```bash theme={null}
   # Fix permissions
   chmod 644 /etc/anubis/policy.yaml
   chown anubis:anubis /etc/anubis/policy.yaml
   ```

3. **YAML syntax errors**:
   ```bash theme={null}
   # Validate YAML
   python3 -c 'import yaml; yaml.safe_load(open("/etc/anubis/policy.yaml"))'
   ```

### Regular Expression Errors

**Symptom**: Error on startup:

```
config.Bot: invalid user agent regex: error parsing regexp: ...
```

**Common issues**:

1. **Trailing newline** (`ErrRegexEndsWithNewline`):
   ```yaml theme={null}
   # Wrong - includes newline
   user_agent_regex: >
     Mozilla.*

   # Correct - use >-
   user_agent_regex: >-
     Mozilla.*
   ```

2. **Unescaped special characters**:
   ```yaml theme={null}
   # Wrong - . matches any character
   user_agent_regex: "example.com"

   # Correct - escape the dot
   user_agent_regex: "example\\.com"
   ```

3. **Invalid regex syntax**:
   ```yaml theme={null}
   # Wrong - unclosed group
   user_agent_regex: "(bot"

   # Correct
   user_agent_regex: "(bot)"
   ```

**Test regex**:

* Use [regex101.com](https://regex101.com) (select Golang flavor)
* Test with `--debug-benchmark-js` flag

### Configuration Validation Errors

**Symptom**: Startup fails with validation error.

#### No Bot Rules Defined

```
config: must define at least one (1) bot rule
```

**Fix**: Add at least one bot rule:

```yaml theme={null}
bots:
  - name: default
    user_agent_regex: ".*"
    action: ALLOW
```

#### Bot Must Have Name

```
config.Bot: must set name
```

**Fix**: Every rule needs a name:

```yaml theme={null}
bots:
  - name: my-rule  # Required
    user_agent_regex: "bot"
    action: DENY
```

#### Bot Must Have Matcher

```
config.Bot: must set either user_agent_regex, path_regex, headers_regex, or remote_addresses
```

**Fix**: Rules need at least one matcher:

```yaml theme={null}
# Wrong - no matcher
- name: empty-rule
  action: DENY

# Correct - has matcher
- name: bots
  user_agent_regex: "bot"
  action: DENY
```

#### Invalid CIDR

```
config.Bot: invalid CIDR: ...
```

**Fix**: Use proper CIDR notation:

```yaml theme={null}
# Wrong
remote_addresses:
  - 192.168.1.1  # Missing /mask

# Correct
remote_addresses:
  - 192.168.1.0/24
  - 10.0.0.0/8
```

## Storage Issues

### BoltDB Permission Denied

**Symptom**:

```
can't open bbolt database /data/anubis.bdb: permission denied
```

**Fix**: Check directory permissions:

```bash theme={null}
# Verify directory exists and is writable
ls -ld /data
sudo chown anubis:anubis /data
sudo chmod 750 /data

# Test write access
sudo -u anubis touch /data/test && rm /data/test
```

### BoltDB File Lock

**Symptom**:

```
can't open bbolt database /data/anubis.bdb: timeout
```

**Cause**: Another Anubis instance has the database locked.

**Fix**:

```bash theme={null}
# Find process holding lock
lsof /data/anubis.bdb

# Stop conflicting instance
sudo systemctl stop anubis

# Or kill process
kill <PID>
```

### Valkey Connection Failed

**Symptom**:

```
valkey.Factory: ping failed: dial tcp: connection refused
```

**Check**:

```bash theme={null}
# Test Valkey connectivity
redis-cli -h valkey.example.com -p 6379 ping
# Expected: PONG

# Check network
telnet valkey.example.com 6379
```

**Common causes**:

1. **Wrong URL**:
   ```yaml theme={null}
   # Wrong protocol
   url: http://valkey:6379

   # Correct
   url: redis://valkey:6379/0
   ```

2. **Network policy blocking**:
   * Check firewall rules
   * Verify Kubernetes NetworkPolicy
   * Check security groups (cloud)

3. **Authentication required**:
   ```yaml theme={null}
   # Add credentials to URL
   url: redis://username:password@valkey:6379/0
   ```

### S3 Access Denied

**Symptom**:

```
can't put s3 object: AccessDenied: Access Denied
```

**Check**:

```bash theme={null}
# Verify environment variables
echo $AWS_ACCESS_KEY_ID
echo $AWS_SECRET_ACCESS_KEY
echo $AWS_REGION

# Test AWS credentials
aws s3 ls s3://anubis-data/
```

**Fix**: Update IAM policy:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:HeadObject"
      ],
      "Resource": "arn:aws:s3:::anubis-data/*"
    }
  ]
}
```

## Runtime Issues

### Challenges Not Working

**Symptom**: Users see challenge page but it doesn't solve.

**Check browser console** for JavaScript errors.

**Common causes**:

1. **JavaScript disabled**: Use `metarefresh` algorithm
2. **CORS issues**: Check proxy configuration
3. **Ad blocker**: Whitelist Anubis
4. **CSP headers**: Ensure challenge scripts can run

### Users Keep Getting Challenged

**Symptom**: Users solve challenge but are challenged again on next request.

**Causes**:

1. **Cookies disabled**: Check browser settings

2. **Cookie domain mismatch**:
   ```bash theme={null}
   # Wrong - too broad
   --cookie-domain .com

   # Correct
   --cookie-domain example.com
   ```

3. **Signing key changed**: Users need to re-solve after key rotation

4. **JWT expired**: Check `--cookie-expiration-time`

5. **IP restriction enabled**: Mobile users change IPs

**Debug**:

```bash theme={null}
# Check cookie in browser DevTools
# Application > Cookies > anubis-auth

# Verify JWT
curl -v http://localhost:8923/
# Look for Set-Cookie header
```

### High Memory Usage

**Symptom**: Anubis process using excessive memory.

**Check**:

```bash theme={null}
# View memory usage
ps aux | grep anubis

# Get Go memory stats
curl http://localhost:9090/metrics | grep go_memstats
```

**Causes**:

1. **Memory storage backend**: No size limits
   * **Fix**: Use bbolt or valkey

2. **DNS cache growth**: Large cache with high DNS TTL
   * **Fix**: Lower DNS TTL:
     ```yaml theme={null}
     dns_ttl:
       forward: 60   # Lower from 300
       reverse: 60
     ```

3. **Goroutine leak**: Check goroutine count
   ```bash theme={null}
   curl http://localhost:9090/metrics | grep go_goroutines
   ```

### High CPU Usage

**Symptom**: Anubis using 100% CPU.

**Causes**:

1. **Challenge difficulty too high**:
   ```yaml theme={null}
   # Lower difficulty
   challenge:
     difficulty: 2  # Instead of 8+
   ```

2. **Too many requests**: Scale horizontally

3. **Expensive CEL expressions**: Optimize rules

**Profile CPU**:

```bash theme={null}
# Enable profiling (development only)
go tool pprof http://localhost:9090/debug/pprof/profile
```

## Network Issues

### Client IP Always Shows Internal IP

**Symptom**: `X-Real-IP` shows proxy IP, not client IP.

**Causes**:

1. **Missing `X-Forwarded-For` header**: Proxy not configured
2. **Using `--use-remote-address`**: Only for bare metal

**Fix**: Configure proxy to set headers:

```nginx theme={null}
# Nginx
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
```

```yaml theme={null}
# Traefik (labels)
traefik.http.middlewares.anubis-headers.headers.customrequestheaders.X-Real-IP=<clientip>
```

### DNS Lookups Failing

**Symptom**: `verifyFCrDNS()` always fails.

**Check**:

```bash theme={null}
# Test DNS from container
dig +short google.com @8.8.8.8

# Check DNS cache
curl http://localhost:9090/metrics | grep dns
```

**Causes**:

1. **DNS server unreachable**: Check `/etc/resolv.conf`
2. **Firewall blocking port 53**: Allow UDP/TCP 53
3. **DNS cache poisoned**: Restart Anubis

## Performance Issues

### Slow Challenge Response

**Symptom**: Challenge page takes seconds to load.

**Causes**:

1. **Storage backend latency**: Check backend health
2. **DNS lookups in expressions**: Cache results
3. **Complex CEL expressions**: Simplify rules

**Benchmark**:

```bash theme={null}
# Test challenge performance
time curl -A "Mozilla/5.0" http://localhost:8923/
```

### High Latency

**Symptom**: All requests slow through Anubis.

**Causes**:

1. **Backend unreachable**: Check target server
2. **TLS handshake slow**: Use `--target-insecure-skip-verify` (dev only)
3. **Storage backend slow**: Monitor storage metrics

**Profile**:

```bash theme={null}
# Check target connectivity
time curl http://backend:3000/

# Test without Anubis
time curl --resolve example.com:80:backend-ip http://example.com/
```

## Key and JWT Issues

### Generating Random Key Warning

**Symptom**:

```
WARN generating random key, Anubis will have strange behavior when multiple instances are behind the same load balancer target
```

**Impact**: Challenges invalidated on restart.

**Fix**: Configure signing key:

```bash theme={null}
# Generate key
openssl rand -hex 32 > /etc/anubis/ed25519.key

# Use it
export ED25519_PRIVATE_KEY_HEX_FILE=/etc/anubis/ed25519.key
```

### Key Validation Failed

**Symptom**:

```
failed to parse and validate ED25519_PRIVATE_KEY_HEX: supplied key is not 32 bytes long, got X bytes
```

**Fix**: Key must be exactly 64 hex characters:

```bash theme={null}
# Wrong length
ED25519_PRIVATE_KEY_HEX=abc123  # Only 6 hex chars = 3 bytes

# Correct length
ED25519_PRIVATE_KEY_HEX=$(openssl rand -hex 32)  # 64 hex chars = 32 bytes
```

### JWT Signature Invalid

**Symptom**: Users challenged every request despite having cookie.

**Causes**:

1. **Key changed**: Signing key rotated
2. **Multi-instance with different keys**: Each instance using random key
3. **Clock skew**: JWT `nbf` (not before) in future

**Fix**:

1. Use same key across all instances
2. Sync system clocks (NTP)

## Container/Kubernetes Issues

### Container Exits Immediately

**Symptom**: Container starts then exits.

**Check logs**:

```bash theme={null}
# Docker
docker logs <container-id>

# Kubernetes
kubectl logs <pod-name>
```

**Common causes**:

1. **Missing target**: `--target` not set or invalid
2. **Configuration error**: Invalid policy file
3. **Port already bound**: Another process using port 8923

### Health Check Failing

**Symptom**: Kubernetes pod stuck in `CrashLoopBackOff`.

**Check**:

```bash theme={null}
kubectl describe pod <pod-name>
# Look for: Liveness probe failed

# Test health check manually
kubectl exec <pod-name> -- /usr/local/bin/anubis --healthcheck
```

**Fix**:

1. Increase `initialDelaySeconds` (allow more startup time)
2. Increase `timeoutSeconds` (slow storage backend)
3. Check metrics port binding

### Secret Not Mounted

**Symptom**:

```
failed to read ED25519_PRIVATE_KEY_HEX_FILE /secrets/ed25519.key: no such file or directory
```

**Check**:

```bash theme={null}
# Verify secret exists
kubectl get secret anubis-key

# Check if mounted
kubectl exec <pod-name> -- ls -l /secrets/
```

**Fix**: Verify volume mount:

```yaml theme={null}
volumeMounts:
  - name: signing-key
    mountPath: /secrets
    readOnly: true
volumes:
  - name: signing-key
    secret:
      secretName: anubis-key  # Must match
```

## Debugging Tips

### Enable Debug Logging

```bash theme={null}
# Via flag
anubis --slog-level DEBUG

# Via policy
logging:
  sink: stdio
  level: DEBUG
```

### Test Mode

```bash theme={null}
# Force all requests to show challenge (no blocking)
anubis --debug-benchmark-js
```

### Validate Configuration

Anubis validates config on startup. Watch for errors:

```bash theme={null}
# Run in foreground to see startup logs
anubis --policy-fname /etc/anubis/policy.yaml 2>&1 | less
```

### Check Prometheus Metrics

```bash theme={null}
# View all metrics
curl http://localhost:9090/metrics

# Check specific metric
curl -s http://localhost:9090/metrics | grep anubis_policy_results

# Monitor in real-time
watch -n 1 'curl -s http://localhost:9090/metrics | grep anubis_policy_results'
```

### Trace Request Flow

```bash theme={null}
# Enable verbose logging
export SLOG_LEVEL=DEBUG

# Make request with identifiable user agent
curl -A "TEST-REQUEST-$(date +%s)" http://localhost:8923/

# Find in logs
grep "TEST-REQUEST" /var/log/anubis/anubis.log
```

## Getting Help

If you're still stuck:

1. **Check logs**: `--slog-level DEBUG`
2. **Review config**: Validate YAML syntax
3. **Test components**: Storage, DNS, target server
4. **Simplify**: Remove complex rules, test with minimal config
5. **Report issue**: [https://github.com/TecharoHQ/anubis/issues](https://github.com/TecharoHQ/anubis/issues)

Include:

* Anubis version (`anubis --version`)
* Configuration (redact secrets)
* Error messages and logs
* Steps to reproduce

## Next Steps

* [Monitoring](/admin/monitoring) - Set up metrics and alerts
* [Security](/admin/security) - Key generation and rotation
* [Policy Configuration](/admin/policy-configuration) - Validate and optimize policy