CORE MODULE — ANALYTICS & SEO

WordPress Data Pipeline Plugin With ClickHouse, RabbitMQ, and Redis

Route every event from your WordPress site through a three-tier data pipeline that writes to ClickHouse for analytics, queues through RabbitMQ for async processing, and falls back to the WordPress database when nothing else is available. Data Core is the storage engine behind every LuperIQ module.

WordPress Was Never Designed to Handle Event Data at Scale

Every time a visitor views a page, a WooCommerce order is placed, or a cron job finishes, your site generates event data. WordPress stores it in the options table, in post meta, in custom tables, or not at all. There is no unified pipeline, no schema enforcement, and no way to query historical events without building custom SQL for each plugin. When traffic grows, MySQL becomes the bottleneck for both your site's front end and your analytics backend.

Data Core solves this by introducing a structured event pipeline directly inside WordPress. Every LuperIQ module publishes events through a single Pipeline::publish() call. The pipeline routes each event through a three-tier transport chain — ClickHouse for columnar analytics storage, RabbitMQ for async message queuing, and the WordPress database as an always-available fallback. If ClickHouse is down, events flow to RabbitMQ. If RabbitMQ is down, they land in WPDB. When services recover, you replay the buffer with one WP-CLI command.

Every Feature in the Data Core Module

Who This Module Is Built For

Any WordPress site that needs structured event storage, analytics-grade data infrastructure, or a centralized pipeline connecting multiple plugins to a single data backend.

WooCommerce and E-commerce Operators

• Order events, revenue exports, and shipping logistics flow through typed serializers into ClickHouse for real-time analytics.
• The commerce serializer captures checkout events, payment processing, and order item details in columnar format for fast aggregation.
• Retention policies let you keep two years of order data in ClickHouse while expiring debug logs after 30 days.

Agencies and DevOps Teams

• Provisioning installs Redis, ClickHouse, and RabbitMQ with one click and manages them as namespaced services that do not conflict with existing infrastructure.
• AutoBootstrap means zero manual configuration on first deploy — services are detected, settings are adopted, tables are created, and caches are warmed automatically.
• WP-CLI commands for consuming, replaying, and bootstrapping integrate cleanly into deployment scripts and CI/CD pipelines.

High-Traffic Publishers and SaaS Sites

• ClickHouse handles millions of events per day without adding query load to your MySQL database.
• The three-tier transport chain with circuit breakers ensures zero data loss even during infrastructure outages.
• The Data Dictionary with Redis caching provides sub-millisecond key-value lookups for site-wide configuration and feature flags.

How the Three-Tier Transport Chain Works

When any LuperIQ module calls Pipeline::publish(), the event enters a deterministic routing chain. The pipeline reads the configured transport from the admin settings. If the transport is set to ClickHouse and the connection is live, the event is written directly to a typed ClickHouse table using the appropriate serializer. If ClickHouse is unreachable (detected by the circuit breaker), the event falls through to RabbitMQ.

RabbitMQ publishes the event as a persistent JSON message to a durable topic exchange (luperiq.pipeline by default). The routing key is the event alias, so consumers can subscribe to specific event types. If RabbitMQ is also down or not configured, the event lands in the WordPress database fallback table (wp_luperiq_pipeline_events) with a timestamp, alias, and JSON payload.

Both ClickHouse and RabbitMQ transports use circuit breakers with a 10-minute cooldown. When a transport fails, the breaker trips and all subsequent events skip that tier for 10 minutes before retrying. This prevents slow timeouts from blocking page loads. Events that accumulate in the WPDB fallback can be replayed into ClickHouse later using the wp luperiq:replay CLI command with alias, date range, and batch size filters.

The Data Dictionary: Redis-Backed Key-Value Storage

The Data Dictionary is a namespaced key-value store that other LuperIQ modules use for fast configuration lookups, cached computations, and cross-request state. It uses a composite store architecture: Redis serves as the primary cache layer with configurable key prefix (default liqd:) and TTL (default 86400 seconds). MySQL provides durable persistence via a dedicated dictionary table.

When a key is requested, the Dictionary checks Redis first. On a cache miss, it reads from MySQL and auto-populates Redis for the next request. Writes go to both stores simultaneously when shadow persistence is enabled (the default). The warm-on-boot feature repopulates Redis from MySQL after a server restart, so the first visitor after a reboot does not pay the cold-cache penalty. You can warm specific namespaces or the entire dictionary from the admin UI or WP-CLI.

Smart Service Detection and Hosting Tiers

Data Core includes a ServiceDetector that probes localhost for Redis (port 6379), ClickHouse (port 8123), and RabbitMQ (port 5672) using non-blocking TCP connections. Detection results are cached for 24 hours to avoid repeated probes on shared hosting. Based on what is available, your site is classified into one of three hosting tiers.

Basic tier means no external services are detected — the pipeline uses the WordPress database for everything. Standard tier activates when Redis or ClickHouse is available, enabling the Dictionary cache layer or direct analytics writes. Enterprise tier requires all three services (Redis, ClickHouse, and RabbitMQ) and unlocks the full async pipeline with message queuing, typed serialization, and materialized views. The module adapts automatically to whatever infrastructure is present, so you get the best performance your hosting can deliver without manual tuning.

Frequently Asked Questions

Does Data Core work without ClickHouse or RabbitMQ?

Yes. On basic hosting with only MySQL available, the pipeline writes all events to the WordPress database fallback table. You get full event logging and retention cleanup without any external services. When you add ClickHouse or Redis later, the module detects them automatically and upgrades your hosting tier.

What happens to events when ClickHouse goes down?

The circuit breaker trips immediately and routes events to the next available tier. If RabbitMQ is configured, events queue there for async processing. Otherwise, they land in the WordPress database. The circuit breaker resets after 10 minutes and retries ClickHouse. Events stored in WPDB can be replayed into ClickHouse using the wp luperiq:replay command.

How does AutoBootstrap work?

On the first admin page visit after activation, AutoBootstrap runs a six-step process: detect local services, register 20+ default alias-to-table mappings, scan for WordPress log files, create canonical ClickHouse tables, build materialized views for the master log, and warm the mapping cache. It runs once per version and can be re-triggered from the admin UI or WP-CLI.

Is Data Core the same as a WordPress analytics plugin?

No. Data Core is the storage and transport layer that analytics plugins build on. It does not render charts or dashboards on the front end. Other LuperIQ modules (like the analytics or journey tracking modules) publish events through Data Core's pipeline and then query ClickHouse to display results. Data Core handles the plumbing.

What serializers are included?

Twelve typed serializers ship with the module: AnalyticsEvents, CommerceEvents, JourneyEvent, EcommerceOrders, EcommerceOrderItems, RevenueExport, LogsWp, LogsHttp, LogsSecurity, PerformanceMetrics, AiProcessing, and LogisticsEvents. Each serializer defines a ClickHouse table schema, a row mapping function, and an engine clause with ORDER BY and optional TTL.

Can I add custom event aliases and tables?

Yes. The Queue Registry stores alias-to-table mappings in the WordPress options table. You can register new aliases programmatically with QueueRegistry::register() or add them through the admin UI. Each mapping specifies the ClickHouse table name, serializer type, whether to auto-create the table, and the TTL in days.

How does the Data Dictionary differ from WordPress transients?

Transients use the options table (or object cache if available) with no namespacing, no composite caching, and no built-in warm/flush operations. The Data Dictionary provides namespaced keys, a composite Redis + MySQL store with automatic cache population on miss, configurable TTL, write-through persistence, and bulk warm operations. It is designed for high-frequency reads across the entire LuperIQ module ecosystem.

What WP-CLI commands are available?

Five commands are available: luperiq:pipeline-consume starts a RabbitMQ consumer for background event processing, luperiq:replay replays WPDB fallback events into ClickHouse by alias and date range, luperiq:logs provides log management utilities, luperiq:autobootstrap lets you re-run or reset the bootstrap process from the command line, and luperiq:dictionary manages Data Dictionary operations including warm and flush.

Add Enterprise-Grade Data Infrastructure to WordPress

Data Core installs in seconds and auto-configures to your hosting environment. Start with the WordPress database fallback and scale up to ClickHouse and RabbitMQ when you are ready.

See Pricing