Hi there,
I'm using Feed Api to aggregate a few blogs and want to grab the photos that accompany the posts as well.
I've taught myself the PHP to get the files, parse them and return the image URLs and have it all themed and looking nice.
But I need to store this data with the feed items at the time of "capture" rather than calling it everytime I reload the page and hotlinking the images. Not cool.
Feed API appears to have hooks that will let me create a processor that can add specific scraping functions like this to specific feeds.
But there's no where on the internet that's explaining how to do this.
I understand that most people using Drupal are Drupal pro's. That's why Drupal documentation like the api just consists of a lists of functions and the code for those functions...no actual examples.
But could someone write up a walk through/example for how a scraping processor like this could be made? I'd be happy to help make sure it covers all the necessary points and so on. I just think a lot of people, myself included would find it useful to be able to expand FEED API in this way.
Comments
I don't have the time to
I don't have the time to give you a walk through, but looking at http://drupal.org/project/feedapi_comments could be a good start to figure out what you're trying to do. this project contains a parser, a processor and scraping code based on SimpleXML/XPath.
http://www.twitter.com/lx_barth
http://www.twitter.com/lxbarth
Hi Alex, Yes, it looks like
Hi Alex,
Yes, it looks like this module does something similar to what I'd like to do, scraping a specific type of content from specific sites that feed_api aggregates.
So a custom module's the only approach here? I was hoping there'd be a more plug and play set up. I've never written a database schema and was hoping I'd be able to avoid having to learn the ins and outs of how drupal saves stuff to the database and so on.
Question: With this module, the parser_comments is the only piece I'd need to look at, right? Or is that just where the HTML is parsed and the other module is what does all the storage and tricky Drupal interaction?
Thanks for the suggestion. I'm not sure I've got the chops to figure out how all this works though.
Feed Scraper
Hi,
I've been working on a module that extract content from raw HTML contained into syndication feeds. It is called Feed Scraper, here its project page:
http://drupal.org/project/feedapi_scraper
As you can see from this screenshot in attachment it uses xpath query to scrape out the information you are interested in. It also allows to plug in other parsers as well, a regular expression one is already available with the module.
After defining the scraper you are then able to map it using Feed Element Mapper module. Have a look to it, every feedback is highly appreciated.