WP Content Crawler – Get content from almost any website, mechanically!
Get content from almost any website to your WordPress weblog, mechanically!
Docs
| Demo
| Website
| Join our Discord server! (New)
FOR WHAT IT CAN BE USED
- Create a private website which collects information, posts, and many others. from your favourite websites to see them in a single place
- Use it with WooCommerce to gather merchandise from procuring websites
- Collect merchandise from affiliate packages to earn money
- Collect posts to create a take a look at atmosphere to your plugin/theme
- Collect plugins, themes, apps, photographs from different websites to create a group of them
- Keep monitor of rivals
- You can think about something. The web is stuffed with contents
Before you purchase, be sure you do the next:
-
Watch the quick start video and use the plugin within the
demo. You also can watch the opposite
video tutorials to discover ways to use
the plugin. There are additionally many guides
explaining easy methods to do sure issues with the plugin. -
Make certain the plugin can retrieve the information from the positioning you wish to crawl by following the directions in
Can I get content from X site? FAQ. -
If you’re nonetheless unsure if the plugin can retrieve content from a particular website, ask us within the feedback
part. -
You can verify the FAQs
when you’ve got any questions. If the reply to your query just isn’t there, you may all the time ask us within the feedback
part.
30-SECOND* SITE SETUP WITH CONFIG HELPER
*Config Helper is displayed when creating a brand new website. Its function is to hurry up the preliminary setup for a website.
Although Config Helper works for a lot of websites, not each website will be configured this simply, which means that sure
websites could require a handbook and extra elaborate setup. Even for these websites, you may attempt to create the fundamental
configuration with Config Helper after which modify the positioning settings later. Additionally, it’s doable to exit
Config Helper any time or disable Config Helper totally.
QUICK START
HOW IT WORKS
It’s all about CSS selectors and you’ll discover ways to use them in minutes by watching the introduction tutorial. The plugin’s Visual Inspector software additionally helps you discover CSS selectors simply by clicking onto the weather within the goal websites. Here is the gist of it:
WHAT WP CONTENT CRAWLER CAN DO
Here is the listing of some options of WP Content Crawler. To study the entire options, please see the options desk under.
HTML code of social media posts present within the goal put up web page is mechanically transformed to quick codes. By this
method, they’re displayed on the entrance finish of your web site appropriately. Additionally, all of the iframe components are
transformed to a brief code. An iframe quick code is displayed provided that its supply is trusted. If the supply of an
iframe just isn’t trusted by default, you may manually add a trusted area in order that the iframe is displayed. With this
technique, you may show media from third-celebration websites securely. Websites whose media are embedded mechanically
embrace Instagram, Imgur, YouTube, Vimeo, TikTook, Kickstarter, Twitter, Pinterest, and so forth.
SEE IT IN ACTION, LEARN IN MINUTES
|
| WP Content Crawler introduction video (English) |
|
| WP Content Crawler introduction video (Turkish) |
VIDEO TUTORIALS
|
| Quick Start Guide |
|
| Using CSS Selectors in WP Content Crawler |
|
| HTML & CSS Selectors |
|
| Using quick codes to put any information anyplace within the put up |
|
| Save photographs as WooCommerce product gallery |
|
| Save arcade video games |
MAIN FEATURES
| Save each put up element Title, excerpt, content, tags, classes, slug, date, customized meta, taxonomies, meta key phrases, meta description, featured picture, put up photographs, standing… Just all the pieces. | Visual Inspector Just click on to a component to seek out its CSS selector. You also can get different CSS selectors that you just could be occupied with. There is not any want to depart your admin panel anymore. | |
| Crawl (scrape, seize, save) posts After the settings are configured, the plugin finds URLs of the posts and crawls them mechanically within the background. | Recrawl (replace) posts | |
| Delete posts You wish to delete previous crawled posts? The plugin can delete them mechanically. | Control scheduling | |
| Save classes The goal class doesn’t exist in your website? No downside. The plugin can create the goal classes for you. Just outline the CSS selectors that discover class names. They may even be created as subcategories. | Save slugs (permalink) You can outline the permalink of the posts. You can get the permalink from the goal website, enter customized textual content, and even create templates for the slugs through the use of quick codes. | |
| Save taxonomies Save taxonomy values by retrieving them from the goal website or coming into manually. Saving particulars of customized put up sorts is simpler than ever. | Save posts into customized classes | |
| Custom put up meta Save something as customized put up meta. You can use a CSS selector or simply kind the worth. | Content templates | |
| Alternative selectors You can write different selectors to get the information even when the goal website has put up pages designed otherwise from one another. | Find and change something | |
| Paginated posts Target put up has a couple of web page? No worries. You can save paginated posts as nicely. | List kind posts Some websites create posts with a listing inside. You can extract the listing from the put up, create a template that needs to be utilized to every listing merchandise and even reverse the listing. | |
| Remove pointless components Sometimes it’s worthwhile to do away with some components, reminiscent of ads, feedback, you title it. Just write its CSS selector and it’s eliminated. | Automatically insert class URLs Target website has a whole bunch of classes? Piece of cake. Just write the CSS selector and the plugin will insert them for you. | |
| Post sorts Set put up kind. It is usually a put up, a web page, a product, or any different put up kind accessible in your WordPress set up. | Remove hyperlinks You can take away hyperlinks from the put up. Just verify the checkbox and the hyperlinks are gone. That simple. | |
| Password safety You can set a password for the posts to point out them solely to the customers who’ve the password. | Notes You can add notes for your self to remind you issues concerning the website. CSS selectors, TODO listing, something. | |
Test all the pieces on the fly | Test all of the settings of a website directly Using the tester, you may take a look at all choices you configured within the website settings to verify all the pieces works as you need earlier than enabling automated crawling. | |
| Tools Using the instruments, it can save you posts manually with their URL, recrawl posts with their ID or delete already-saved URLs. | Custom common settings for every website You can present customized common settings for every put up to override them and make them appropriate for a website. | |
| Post standing You can straight publish the saved posts or maintain them as draft to verify them earlier than publishing. | Save all photographs in put up content Saving all photographs within the content of the put up is as simple as checking a single checkbox. | |
Save photographs as gallery | Any information as quick code Get something from goal web page as a brief code and use the quick codes within the plugin’s templates to put any information anyplace you need. | |
| Proxy Use a proxy or proxies to get content from the websites to which your IP doesn’t have entry. | Cookies Attach cookies, reminiscent of session cookies, to every request. By this manner, for instance, you may crawl the goal website as in case you are logged in. | |
| Crawl as many posts as you need You can set what number of occasions put up crawling or URL assortment CRON occasions ought to run. By this manner, you may, e.g., save 100 posts each minute. Just watch out and think about your server’s capability. | Email notifications Set CSS selectors whose values shouldn’t be empty for class and put up pages. When an empty worth is discovered utilizing these selectors, you will get an e-mail notification. | |
| Get information from JSON When you allow JSON parsing for a CSS selector, you will get the values from the JSON simply. | Advanced HTML manipulations | |
Automatic translation | Automatic spinning Use spinning to mechanically rewrite crawled posts’ contents to enhance SEO. The plugin at the moment implements Spin Rewriter API and Turkce Spin API, that are paid companies. You can go to their web site to study the pricing particulars. | |
Duplicate put up verify | Scheduled posts You can add/take away minutes to/from the put up date. By this manner, you may schedule put up publishing. | |
| Save WooCommerce merchandise Save worth, stock, delivery, attributes, and superior choices. You can save the product as a easy or an exterior product. You also can set downloadable file choices and outline the product as digital. The choices can be found for WooCommerce variations larger than or equal to three.3. | Options field You have the management! Define many choices for the values discovered by a CSS selector. The choices embrace discover-change, calculation, template, and JSON parsing settings. You can simply import/export the choices outlined within the choices containers as nicely. | |
| Handle recordsdata like a professional Rename, copy, and transfer saved recordsdata simply. You also can outline title, description, caption, and alt texts for the saved media recordsdata utilizing templates by which you should utilize any quick code. It can also be doable to offer random names to the saved recordsdata. | Handle iframes and scripts like a professional WordPress doesn’t enable displaying iframes and scripts since they pose a safety threat. You can flip iframe and script HTML components into quick codes by simply checking a checkbox. The quick code will present iframes and scripts from the allowed supply domains outlined by you. | |
| Quick save With fast save button, it can save you the settings far more shortly. No want to attend for web page to reload. | Regular expressions Define common expressions in discover-change choices to seek out-change something. You also can use delimiters and modifiers to match extra exactly. | |
| Save “srcset” attributes When different sizes of the saved photographs can be found, the plugin assigns them into srcset attribute of img components in order that your pages will load sooner in numerous display screen sizes. | Save “alt” and “title” attributes | |
| Warnings Learn when there’s a downside. The plugin will present you the small print of the error so as to repair it immediately. | Handle character encoding issues | |
| Navigate between settings simply Fix navigation to the highest! The plugin shops the place you have been earlier than switching to a brand new tab and restores your earlier location while you activate that tab once more. No extra getting misplaced among the many settings. | Manual crawling software With handbook crawling software, save a number of posts by coming into their URLs. You also can enter class URLs in order that the software can get put up URLs from there. Moreover, you may set it to crawl a number of posts on the identical time. | |
Add URLs to the database | Enable/disable automated crawling for a particular website You can allow or disable automated crawling for every website individually. | |
| Import/export You can import and export website settings simply. Just copy and paste the code created by the plugin. | Unlimited Add limitless websites to the plugin and activate what number of of them you need. | |
Detailed dashboard | Get updates from your admin panel You can replace the plugin with only one click on each time an replace is prepared. Just go to your updates web page in your admin panel. | |
| Use essentially the most safe PHP The plugin helps the newest variations of PHP. | Use essentially the most fashionable browsers The plugin helps Chrome, Firefox, Safari, Opera, and Edge. | |
| Interactive guides Interactive guides present you easy methods to configure settings to realize sure issues, step-by-step, like a dwelling documentation. You can begin these guides everytime you need. You may even begin them from a particular step. | Online documentation You can verify the web documentation everytime you really feel a necessity. | |
| Quick guides proper subsequent to the settings Each setting within the plugin has a fast information that can provide help to perceive what every setting does. | Video tutorials Watch video tutorials to simply discover ways to use the plugin. | |
| Ready to translate You can translate the plugin into your personal language utilizing Poedit. | Filters | |
| Use OpenAI GPT (ChatGPT) You can use OpenAI GPT fashions to alter the title, content, tags, file names, and extra. You can use GPT-3.5 and GPT-4. With the superior quick code builder, you should utilize the chat, full, edit, and insert modes. To study extra, watch this video! | Convert JSON to HTML | |
| Embed social media posts mechanically Posts from 70+ web sites together with Instagram, Facebook, Amazon, YouTube, Twitter, Scribd, Vimeo, Pinterest, Spotify, Meetup, and lots of others are transformed to embed quick codes mechanically. By this manner, they’re displayed securely on the entrance finish of your web site! | Make customized requests You could make customized requests to different APIs or web sites and embrace the response into the present web page. Watch this video to see how highly effective this characteristic is! This additionally makes it doable to retrieve content added to the web page through AJAX. |
| Requirements | PHP >= 8.1, json, mbstring, curl, dom, fileinfo, WP-Cron. These are already accessible in most hosts. Even if the extensions will not be already lively, most internet hosting websites allow you to allow these from their management panel. See the documentation for extra info. |
| Tested with WP variations | 6.6, 6.5, 6.4, 6.3, 6.2, 6.1, 6.0, 5.9 |
| Tested with WooCommerce variations | 9.2, 9.1, 9.0, 8.7, 8.2, 7.9, 7.7, 7.5, 7.3, 6.9 |
| Languages | English, Türkçe |
| Shortcomings | The plugin can not retrieve content that’s created through the use of JavaScript. For extra info, please see Can I get content from X site?. |
HAPPY CUSTOMERS
WHY WP CONTENT CRAWLER
Problems with crawling an internet site
- Not a straightforward process, requires superior programming abilities
- Every web site is totally different and wishes tailor-made crawling implementation
- Not simply each web site is totally different, but additionally pages of a single web site can differ
- Pages and their supply codes have to be investigated intensively to provide you with a crawling plan
- Knowing easy methods to save sure info in a particular place in WordPress requires information concerning the inner construction of WordPress and the way WordPress works
- If sure info needs to be saved into a particular subject outlined by a 3rd-celebration plugin, one ought to modify the crawling implementation after researching for hours about easy methods to save that info
- One ought to learn about how HTML works and easy methods to extract sure elements from HTML code
- One ought to deal with all doable inconsistencies that could be within the supply codes of internet sites to supply a strong resolution that can maintain working
- What if the posts have to be shared in common time intervals?
- What if you wish to crawl new posts added to an internet site after a while?
- What about translating the posts from one language to a different?
- What if the posts must be paraphrased to supply a greater SEO for the web site?
- What if some info shouldn’t be retrieved?
- What if sure info needs to be modified to make it appropriate to your website?
- What if one other website must be crawled, not only one?
- What if that different website wants a distinct crawling plan?
- What if it’s worthwhile to login to the web site to crawl it?
- What if the web site modifications its supply code?
- What if you wish to replace the crawled posts by recrawling them from the unique web site?
- What if you wish to be sure that if the knowledge is retrieved precisely as you need it earlier than mechanically posting the posts to your web site?
- What if you wish to guarantee your website’s safety by ensuring no malicious-code-executing code results in your website?
- And many extra what-ifs that you just won’t even think about except you come throughout them
Our imaginative and prescient and mission
We consider that sturdy, dependable, and automated crawling capabilities needs to be accessible for anybody. We wish to democratize this subject by letting anybody have these capabilities, not simply builders. With this function, we purpose to supply a plugin that you’ll fall in love with and really feel at house when utilizing it. To let it accessible by anybody, we make the plugin low-value and simple-to-use. We don’t implement the options simply to make gross sales. We plan and execute for the longer term. We all the time hearken to your suggestions and make required modifications accordingly. We assume that WordPress plugins needs to be developed with enterprise-degree care. So, we intensively take a look at the plugin earlier than every launch with automated finish-to-finish UI assessments, at the moment over 1700 assessments, that run in many various environments within the cloud for a complete of over 40 hours to make sure the plugin is suitable together with your server and WordPress environments and also you, our useful prospects, get the standard and reliability you deserve.
How we resolve these issues
We have been creating WP Content Crawler for almost 4 years such that we’ve come throughout almost all of the what-ifs. Working with our prospects and listening to their wants, we offer sturdy and dependable options to those issues. We consider that one ought to simply present from which website the knowledge needs to be retrieved and what info needs to be retrieved from that web site after which begin crawling that website, with out worrying concerning the complicated behind-the-scenes operations.
To make it accessible to anybody, we offer an in depth on-line documentation that incorporates not simply the outline of the settings however easy methods to use the settings to realize your targets. Sometimes you won’t really feel like studying the documentation. We additionally present interactive step-by-step guides which are accessible within the plugin, only one click on away. You can begin the interactive guides displaying you step-by-step how you are able to do sure issues any time and from any step you need.
One of essentially the most distinctive options of WP Content Crawler is the flexibility to take a look at almost any configuration. By this manner, you’ll not come throughout any surprises after you allow automated crawling. When testing, the errors associated to your settings are proven so as to repair them earlier than they trigger any issues.
WP Content Crawler has so many options that even we have no idea what number of of them are there. You can mechanically crawl, replace, and delete the posts, you may translate posts, spin posts, you may even outline what fields have to be translated or spun if you don’t want all of them modified. You can discover-change almost something. You can assign some info from the goal put up to a brief code and place that info anyplace within the put up. You can save WooCommerce merchandise. You can save particulars for third-celebration plugins that we don’t even know they exist. The options of the plugin are designed such that you just really feel that you’re in management while you use them. We make them as versatile as doable to make them suit your wants. When designing new options, we all the time understand that you would possibly want a extra superior model of that characteristic and we design the options accordingly. We make sure that the options and all the code of the plugin are maintainable and extendable in order that we will all the time enhance the plugin.
CHANGELOG
Changelog is kept in the documentation site. Click here to see the changelog.
