Public Search Engine

Overview

This project is a publicly accessible SearXNG instance running at search.tyfsadik.org. SearXNG is a privacy-respecting meta-search engine that aggregates results from Google, Bing, DuckDuckGo, Wikipedia, GitHub, and other backends in parallel, then deduplicates and ranks them — without exposing any user identity or search history to those engines. No account is required, no cookies are set, and no queries are logged.

The instance is configured with a custom dark theme and deployed behind Nginx with rate limiting to prevent abuse from automated scrapers or bots. Redis provides result caching to reduce redundant outbound queries and improve response times for popular searches. The deployment uses the official searxng-docker repository as a base, with custom settings.yml modifications to enable preferred engines, disable tracking, and tune the result ranking.

Architecture

Tech Stack

• SearXNG — open-source meta-search engine with pluggable backends
• Docker & Docker Compose — containerized deployment via searxng-docker
• Nginx — reverse proxy with limit_req_zone rate limiting and SSL
• Redis — result caching and rate limit state for SearXNG
• Let's Encrypt / Certbot — TLS certificate for search.tyfsadik.org
• Linux (Debian) — host operating system

Build Process

git clone https://github.com/searxng/searxng-docker.git
cd searxng-docker

# Generate a unique secret key
SECRET_KEY=$(openssl rand -hex 32)

# Edit searxng/settings.yml
# Set: general.instance_name, server.secret_key
sed -i "s/ultrasecretkey/$SECRET_KEY/g" searxng/settings.yml

cat searxng/settings.yml | grep secret_key

# searxng/settings.yml key settings:
general:
  instance_name: "search.tyfsadik.org"
  debug: false

server:
  secret_key: "..."
  limiter: false          # Nginx handles rate limiting
  image_proxy: false

search:
  safe_search: 0
  autocomplete: "google"

# Enable specific engines (excerpt):
engines:
  - name: google
    engine: google
    timeout: 3.0
    weight: 2
  - name: bing
    engine: bing
    timeout: 3.0
  - name: duckduckgo
    engine: duckduckgo
    timeout: 3.0
  - name: wikipedia
    engine: wikipedia
    timeout: 2.0
  - name: github
    engine: github
    timeout: 3.0

docker compose up -d

# Check all containers are up
docker compose ps

# Test SearXNG is responding locally
curl "http://localhost:8080/search?q=test&format=json" | jq '.results | length'

# Check Redis is connected
docker compose logs searxng | grep -i redis

# /etc/nginx/nginx.conf (http block)
limit_req_zone $binary_remote_addr zone=searxng_search:10m rate=10r/m;
limit_req_zone $binary_remote_addr zone=searxng_static:10m rate=60r/m;

# /etc/nginx/sites-available/search.tyfsadik.org
server {
    listen 443 ssl;
    server_name search.tyfsadik.org;

    ssl_certificate /etc/letsencrypt/live/search.tyfsadik.org/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/search.tyfsadik.org/privkey.pem;

    location /search {
        limit_req zone=searxng_search burst=5 nodelay;
        limit_req_status 429;
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    location / {
        limit_req zone=searxng_static burst=20;
        proxy_pass http://localhost:8080;
    }
}

# searxng/static/themes/simple/css/searxng.min.css (custom overrides)
# Create custom.css and add to bind mount in compose file

# In searxng/settings.yml:
ui:
  default_theme: simple
  theme_args:
    simple_style: dark

# Or override via a custom CSS file:
# services.searxng.volumes in docker-compose.yaml:
# - ./searxng/custom.css:/usr/local/searxng/searx/static/themes/simple/css/custom.css

# Verify theme is applied
curl https://search.tyfsadik.org | grep "dark"

apt install certbot python3-certbot-nginx -y

certbot --nginx -d search.tyfsadik.org \
  --agree-tos --email [email protected]

# Verify HTTPS is working
curl -I https://search.tyfsadik.org
# Expected: HTTP/2 200

# Check certificate expiry
echo | openssl s_client -connect search.tyfsadik.org:443 2>/dev/null \
  | openssl x509 -noout -dates

Request Flow

Challenges & Solutions

• Some engines rate-limiting the SearXNG IP: Google and Bing occasionally returned CAPTCHA challenges or empty results after frequent requests from the same server IP. Resolved by setting conservative engine timeouts (3 seconds), marking engines as disabled when they return persistent errors, and relying on DuckDuckGo and Brave Search as fallback engines that are less aggressive with bot detection.
• High resource usage from concurrent searches: Each search spawns multiple outbound HTTP requests in parallel, and with several users searching simultaneously the Python worker count became a bottleneck. Tuned by adjusting the server.workers setting in settings.yml from the default of 4 to 2 on the constrained server, reducing memory usage while maintaining acceptable response times.
• Bot traffic overwhelming the instance: Shortly after the instance was indexed by search engines, automated scrapers began sending hundreds of requests per minute. The Nginx limit_req_zone rate limiter reduced this to manageable levels, and returning HTTP 429 to violators causes well-behaved bots to back off.
• Redis connection refused on first start: SearXNG started before Redis was ready, causing connection errors in the logs. Fixed by adding depends_on: redis with a condition: service_healthy health check in the compose file.

What I Learned

• Nginx limit_req_zone rate limiting: zones, burst allowance, and nodelay semantics
• SearXNG engine configuration: timeout tuning, weight scoring, and engine disabling
• Redis result caching strategies and TTL selection for search result freshness
• Meta-search architecture: parallel fan-out requests, result deduplication by URL hash
• Bot mitigation strategies: rate limiting, HTTP 429 responses, and Nginx access logging analysis
• Docker Compose service health checks and dependency ordering