TE Product Crawler is a focused, WooCommerce-dependent product import plugin built by TERMIMAL that lets store owners pull product data directly from major Iranian e-commerce sites — Digikala, Technolife, Snappshop, and Torob — into their own WooCommerce catalog with a full preview-and-edit workflow before anything is saved. It requires WordPress 5.8+, WooCommerce 5.0+, and PHP 7.4+, and it declares full compatibility with HPOS (High-Performance Order Storage), Cart/Checkout Blocks, and the Product Block Editor, so it works cleanly on modern WooCommerce installations without triggering incompatibility warnings. The plugin lives under the TERMIMAL parent menu (which it creates automatically if missing) and exposes a single submenu — TE Product Crawler — with three internal tabs: Single Crawl, Bulk Import, and Settings.
The Single Crawl tab is the heart of the plugin: the user pastes a product URL, optionally selects the source site (or lets auto-detection handle it), and clicks Start Crawl. The crawler registry routes the request to the correct source handler, which fetches the product data and returns a standardized payload containing title, description, short description, regular price, sale price, brand, SKU, stock status, technical attributes, images, and suggested keywords. The result is rendered into a full preview and edit form where the user can modify the title, short description, rich description (via the WordPress editor), prices, SKU, stock status, brand (selecting an existing one or creating a new one), categories (selecting existing ones or creating new ones on the fly), keywords/tags (with clickable suggestions from both the crawler and existing WooCommerce tags), technical specifications (dynamic attribute rows), and images (removing crawled images, or selecting/uploading from the Media Library, with the first image automatically becoming the featured image). Only after the user clicks Save Product to WooCommerce is a real product created.
The Bulk Import tab supports two input methods: pasting multiple URLs (one per line, with optional per-line crawler_id) or uploading an Excel-compatible CSV file (with a downloadable sample that includes a UTF-8 BOM so Persian text opens correctly in Excel). All bulk imports are always saved as Draft, never published automatically, and they run through a background queue processed by WP-Cron every minute. The user can leave the page safely, and a live Queue Status panel shows pending items, the current resource mode, and a Recent Bulk Activity log with timestamps, URLs, statuses, and direct edit links to created products. A Resource Mode setting (Low / Medium / High, defaulting to Low) controls how aggressively the queue is processed — from 1 product per minute with a 2-second pause, up to 5 products per minute with no pause — making the plugin safe for shared hosting by default.
The crawler architecture is genuinely modular: every source site extends TE_Abstract_Crawler, which defines four required methods (get_id, get_name, supports_url, crawl) plus shared helpers for HTTP requests, text cleaning, and price normalization. Four crawlers ship out of the box. The Digikala crawler uses Digikala’s unofficial api.digikala.com/v2/product/{id}/ JSON endpoint, parses the product payload, and handles the Rial-to-Toman conversion (dividing by 10) for both rrp_price and selling_price, plus extracting specifications, brand, images, and stock status. The Technolife crawler parses the __NEXT_DATA__ JSON blob with an HTML fallback, and the Snappshop and Torob crawlers both parse JSON-LD structured data with HTML fallbacks. Every crawler auto-generates suggested keywords from the product title, brand, and top attributes, and prefixes SKUs with a source-specific code (DK-, TL-, SS-, TR-) to avoid collisions.
The TE_Product_Importer class handles the actual WooCommerce product creation with careful attention to data integrity. It creates a WC_Product_Simple, sets name, description, short description, regular price, and sale price (only if sale is strictly lower than regular), validates and deduplicates the SKU against existing products, sets stock status from a whitelist, sets catalog visibility and post status from a whitelist, assigns categories (existing IDs plus any newly created terms), applies tags, resolves brand either through the product_brand taxonomy or as a custom “Brand” attribute when that taxonomy is missing, adds technical specs as WooCommerce product attributes, and finally handles images. Image handling is notably robust: crawled image URLs are sideloaded via media_handle_sideload, deduplicated by a _te_pc_source_image meta lookup to prevent re-downloading the same image, and Media Library IDs are merged in. The first image becomes the featured image, the rest become gallery images, and the source URL and import timestamp are stored as meta for traceability.
From a scheduling and background processing perspective, the plugin registers two cron hooks: a legacy te_pc_scheduled_crawl (hourly, for the old single-URL queue with optional auto-import and auto-publish) and the modern te_pc_process_bulk (every minute, for the bulk queue which always saves as Draft). It also registers custom cron intervals (te_pc_every_minute, te_pc_every_six_hours, te_pc_twicedaily) and automatically reschedules the bulk processor when settings are saved. The queue uses a deduplication mechanism that skips URLs already present in the queue, and the bulk processor self-heals by re-scheduling itself whenever items remain. Logs are capped at 200 entries and stored in a WordPress option, keeping the database clean without requiring custom tables.
The admin UI is polished and consistent with TERMIMAL’s other plugins: a tabbed interface, card-based layout, live status indicators (loading / error / success), a dashicons-based header, a bulk grid that collapses to a single column on smaller screens, and a logs table with color-coded statuses (green for success, red for error) and direct edit links. A Media Library modal is integrated via wp_enqueue_media() for image selection, and the entire admin JavaScript is namespaced under tePc with localized i18n strings. The Settings tab covers resource mode, auto-import behavior for the legacy scheduled queue, and cron interval, while a Queue Info card displays the current pending count and documents how to add new source sites by extending TE_Abstract_Crawler.
Security and data handling are treated carefully throughout: every AJAX endpoint verifies the te_pc_nonce and checks manage_woocommerce capability, all input is sanitized with sanitize_text_field, esc_url_raw, sanitize_key, absint, or wp_kses_post as appropriate, all output is escaped with esc_html, esc_attr, esc_url, or wp_kses_post, and the sample CSV download is nonce-protected. The plugin never executes raw SQL, never mass-assigns request data, and scopes every product operation to explicit IDs. HPOS and Blocks compatibility are declared through WooCommerce’s official FeaturesUtil API on before_woocommerce_init.
In short, TE Product Crawler is a clean, modular, safety-first product import tool for WooCommerce — one that respects server resources by default, gives users a genuine preview-and-edit workflow before committing anything to the catalog, keeps bulk imports safely as drafts, and is architected so that adding a new source site is as simple as writing one class that extends the abstract crawler base.
Ask a question or leave a comment