Webbots, Spiders, and Screen Scrapers: A Guide to Developing Internet Agents with PHP/CURL Review

Average Reviews:

(More customer reviews)"Webbots, Spiders, adn Screen Scrapers" is a solid book for building basic scripts to do web scraping. Michael Schrenk goes covers the "should you do this" aspect very well, and devotes much of the book to these kinds of topics. On that reason alone I give him major kudos, "just because you CAN do a thing, doesn't mean you SHOULD."
Technically the book and examples are very basic and beginner level. All code is procedural and has absolutely no references to object oriented programming at all. This is great for a simple project, but building anything larger than a targetted webbot or two is beyond the scope of this book.
I was very dismayed at Mr. Schrenk's opinion of regular expressions:
"The use of regular expressions is a parsing language in itself, and most modern programming languages support aspects of regular expressions. In the right hands, regular expressions are also useful for parsing and substituting text; however, they are famous for thier sharp learning curve and cryptic syntax. I avoid regular expressions whenever possible."
This disregard for regular expressions effectively wipes out a powerful toolset for budding developers. Regular expressions are no harder to learn than PHP. The reasons for his disdain for them is also flawed:
"The regular expression engine used by PHP is not as efficient as engines used in other languages, and is certainly less efficient than PHP's built-in functions for parsing HTML."
PHP uses the same regular expression engine used (very effectively) in PERL with the use of the preg_* functions. There has been many studies that show preg_* style expressions outperform basic text matching in PHP. In this assesment the author is terribly wrong.
The book does a great job of explaining how to make single use scripts for scraping, but never how to create a larger infrastructure. There is no focus on creating multi process engines with pcntl_fork(), or proc_open(), these are critical for scaling web scraping applications. A single script scraping a few hundred websites on a single thread would take ages over a multi-threaded engine.
If you are looking to break into web scraping and not sure where to start, this is likely the best (and possibly only) book on the market. If you are intermediate or advanced you will quickly question the author's logic and see that scaling will become the number one issue you have to over come.

Click Here to see more reviews about: Webbots, Spiders, and Screen Scrapers: A Guide to Developing Internet Agents with PHP/CURL


The Internet is bigger and better than what a mere browser allows. Webbots, Spiders, and Screen Scrapers is for programmers and businesspeople who want to take full advantage of the vast resources available on the Web. There's no reason to let browsers limit your online experience-especially when you can easily automate online tasks to suit your individual needs.

Learn how to write webbots and spiders that do all this and more:
Programmatically download entire websites Effectively parse data from web pages Manage cookies Decode encrypted files Automate form submissions Send and receive email Send SMS alerts to your cell phone Unlock password-protected websites Automatically bid in online auctionsExchange data with FTP and NNTP servers
Sample projects using standard code libraries reinforce these new skills. You'll learn how to create your own webbots and spiders that track online prices, aggregate different data sources into a single web page, and archive the online data you just can't live without. You'll learn inside information from an experienced webbot developer on how and when to write stealthy webbots that mimic human behavior, tips for developing fault-tolerant designs, and various methods for launching and scheduling webbots. You'll also get advice on how to write webbots and spiders that respect website owner property rights, plus techniques for shielding websites from unwanted robots.

As a bonus, visit the author's website to test your webbots on sample target pages, and to download the scripts and code libraries used in the book.

Some tasks are just too tedious-or too important!- to leave to humans. Once you've automated your online life, you'll never let a browser limit the way you use the Internet again.

Click here for more information about Webbots, Spiders, and Screen Scrapers: A Guide to Developing Internet Agents with PHP/CURL

Read More...

Spidering Hacks Review

Average Reviews:

(More customer reviews)The `Hacks' series from O'Reilly seems to be breeding as fast as virii in a Windows network - every time you turn around another one. While the writing and editing have remained high some such as `eBay Hacks' have not really had great material. `Spidering Hacks' is an improvement almost back to the quality I remember in the last contribution from Calishain, `Google Hacks'.
She and Kevin Hemenway have taken a fairly complex topic, spidering and scraping web sites and reduced it to manageable chunks in their hundred hacks. The writing has the same light, readable feel you can quickly grow to expect from O'Reilly. Certainly I have never found myself faulting their editing.
There are some caveats. It seems that O'Reilly and Dornfest (the Editor of this book and the series) have fallen in love with having a hundred hacks and little in the way of an introduction. I think this may have been a better book if it was done as 90 `hacks' and had a much larger introduction as the first chapters hacks are all too light and more truly introductory material such as how a HTML page is built and how to properly register your spider. Given that only someone with a fair amount of web knowledge is going to consider spidering a website in the first place then this early material is way too slight. From Hack 9 on it quickly gets down to useful and informative chunks in each and no longer feels `lightweight'.
This may be a reflection on trying to extend the `Hacks' series into places it has to be forced. While the format worked well for Google and Amazon I felt the entire topic of eBay too light for a topic in this series and perhaps spidering is too heavy or complex. If this book had been written in a more traditional format some of my complaints would disappear.
All the examples are in Perl and the serious part of the book starts with examples using LWP::Simple to grab a page before going on to LWP::UserAgent and much more complex requests using authentication, custom headers and posting form data. It also covers using curl and wget.
Then it gets down to the nitty gritty of scraping using HTML:Treebuilder and HTML:TokeParser. This is all further expanded through the next few hacks until starting at Hack 39 through to 89 there are a good series of examples (perhaps a few too many). Finally there are two chapters on maintaining your collection and `Giving Back To The World' which tells how to make it easy to scrape your site and using RSS.
O'Reilly have a page for the book with ten example hacks, index, Table of Contents and errata and you can also visit hacks.oreilly.com for the same ten hacks with the possibility of more being added.
As a whole this volume seems a little thin. If you've been doing the maths then you've realised that only about thirty of the hundred hacks actually give any details on building and running a serious web spider. Sure, a number of the examples provide good information on how to perform various tasks and some of the last eleven hacks are good to know but in all the book feels like it lacks solid information throughout. A bit more information on various crawling and page parsing techniques would have been good.
After that criticism I'm now surprising myself, I'm going to recommend this book. This isn't a large field and when you consider that most other books on writing spiders and crawlers are less than practical and more than expensive "Spidering Hacks" has many good points. It's written for the practical Perl programmer, it examines several methods and gives lots of examples and while not cheap it's certainly inexpensive. Given that I found it both useful and inspiring the complaints above may be a little like nitpicking. I should also say that I found this volume immensely useful in writing my own spider and scraper (it gets a list of new books from the web sites of several publishers.) I have to be honest and admit that there are three publishers, O'Reilly, Addison Wesley and Prentice Hall, from whom I expect a decent standard and criticise a little harder when they move from that norm. If this book had come from SAMS or Wrox I may well have not looked quite so hard for flaws and been a little more generous in my treatment of the ones I found.
That said, I recommend this book to you if you want a practical introduction to building a web spider in Perl.

Click Here to see more reviews about: Spidering Hacks


The Internet, with its profusion of information, has made us hungry for ever more, ever better data. Out of necessity, many of us have become pretty adept with search engine queries, but there are times when even the most powerful search engines aren't enough. If you've ever wanted your data in a different form than it's presented, or wanted to collect data from several sites and see it side-by-side without the constraints of a browser, then Spidering Hacks is for you.

Spidering Hacks takes you to the next level in Internet data retrieval--beyond search engines--by showing you how to create spiders and bots to retrieve information from your favorite sites and data sources. You'll no longer feel constrained by the way host sites think you want to see their data presented--you'll learn how to scrape and repurpose raw data so you can view in a way that's meaningful to you.

Written for developers, researchers, technical assistants, librarians, and power users, Spidering Hacks provides expert tips on spidering and scraping methodologies. You'll begin with a crash course in spidering concepts, tools (Perl, LWP, out-of-the-box utilities), and ethics (how to know when you've gone too far: what's acceptable and unacceptable). Next, you'll collect media files and data from databases. Then you'll learn how to interpret and understand the data, repurpose it for use in other applications, and even build authorized interfaces to integrate the data into your own content.By the time you finish Spidering Hacks, you'll be able to:

Aggregate and associate data from disparate locations, then store and manipulate the data as you like
Gain a competitive edge in business by knowing when competitors' products are on sale, and comparing sales ranks and product placement on e-commerce sites
Integrate third-party data into your own applications or web sites
Make your own site easier to scrape and more usable to others
Keep up-to-date with your favorite comics strips, news stories, stock tips, and more without visiting the site every day
Like the other books in O'Reilly's popular Hacks series, Spidering Hacks brings you 100 industrial-strength tips and tools from the experts to help you master this technology.If you're interested in data retrieval of any type, this book provides a wealth of data for finding a wealth of data.

Click here for more information about Spidering Hacks

Read More...

Bot Programming: Intelligent Automation For Windows Applications And Games Review

Average Reviews:

(More customer reviews)I mainly purchased this book as a starting point to explore UI automation techniques. Unfortunately this is really a poorly written beginning Visual Basic.Net programming book which uses UI automation as a basis for presenting example code. Anyone without prevous programming experience could not extend the code much beyond what was presented without additional information due to the significant holes in the beginning programmers knowledge base that the author neglects.
The book appears to have been written for individuals with no programming experience since the author felt the need to spend the first few pages of each chapter providing screen shots of each property that needed to be modified in the properties window during the set up of the form for the new project. There is no discussion of proper programming techniques or basic information such as discussions of data types or flow control yet after the code listing the author discusses the code line by line to the extent that he talks about what the Else stmt and End If statements are for.
As an intermediate VB.net/C# programmer the level of detail was frustrating to deal with. I found myself skipping the first 2-3 pages of each chapter and skimming the rest looking for useful content. The code was presented poorly and didn't follow the standard code presentation / discussion format that typical programming books follow. Usually a book will only discuse the sections of the code important to the concept being explained rather than every single line. Alot of the details in the line by line discussion could have been more streamlined by properly commenting the code.
In respect to the useful Windows API information presented, it's more of a cut and paste code snippets book that includes a cursory review programming techniques without the discussion of why the code is written in this manner, alternative ways it can be accomplished or what is happening behind the scenes(See Effective GUI Testing Automation which is referenced below for detailed explainations). It teaches poor programming techniques such as programmatic use of the print screen key to perform a screen capture making use of the users clipboard. The correct way would be using gdi+ to directly store the the capture to a bitmap object leaving previously stored clipboard information accessible to the user. A program erasing the user's clipboard data is a pet peeve of mine...
The book itself appeared to need an additional review process. There were a couple of times where I had to reread a sentence to understand the authors intent since a word was obviously missing. I can't make any statements about the accuracy of the code since I did my coding in C#. Mainly I just used the Appendix A which is a listing the main recipes covered to look up Windows API functions on pinvoke(dot)net which is a great free resouce.
The author provides for download an application called the Automation Tool Helper. Unfortunately it's only available on lulu.com so you must provide your email address to them to be able to download it. In addition, the application is poorly thought out adding unnecessary steps to the development cycle. It allows you to enter a process name and get a list of processes that match, providing details such as MainTitleCaption, AppID, x and y position and height and width. It will take a screen capture of the application but then requires you to open MS Paint to determine the coordinates of a particular pixel. You must then return to the authors application, enter in the coordinates into the provided text boxes to return the RGB hex value. This could have just as easily been implemented so that the capture was dropped into a picture box. As you moved the mouse over the picture box you would be provided the x,y coordinates within the picture box and the RGB hex value automatically. This would remove the need to utilize MS Paint at all since a zoom function could easily be added. Also, rather than having to rely on the taskmanager to determine process names, the author could have populated a list of all active process names where clicking on one then provided you the same list of all matching processes with that process name. The first thing that I did after reading this book was rewrite the Automation Tool Helper to include all the improvements described and a couple more I thought of as I worked on it.
For the author to supply such a poorly designed development tool, the inefficient programming style and his indication in the Acknowledgment that his knowledge of bot programming was gained one question at a time on Experts-Exchange leads me to believe that the author is not an expert in this field that should not be relied upon as a source of the correct way things should be done. I would have also liked to have seen a section at the end of the book with references to sources of additional information, both online and in print. On a whole the book has been an adequate place to start learning about user interface automation while I waited for my other books to be delivered by amazon. Mainly it functioned as a tool to let me know what keywords I should be using in google to get additional detailed information. For the intermediate programmer I would highly recommend Effective GUI Testing Automation: Developing an Automated GUI Testing Tool and also .NET Test Automation Recipes: A Problem-Solution Approach (Expert's Voice in .NET) which are not specifically about application automation for purposes other than testing, but have similar concepts discussed in a much more thorough manner.

Click Here to see more reviews about: Bot Programming: Intelligent Automation For Windows Applications And Games

Find your love for botting, with step by step instruction in Visual Basic .NET creating Bots.The Following Windows Automations Are Covered:
Left And Right Mouse Clicks With Millisecond Timing
Set Focus On Specific Applications
Application Specific Mouse Clicks With Millisecond Timing
Send Individual Key Strokes With Timing Control
Send Multiple Keys Stokes Without Timing
Starting Remote Applications
Move Remote Applications
Resize Remote Applications
Run Automated Bots With Windows Scheduled Tasks
Perform Screen Captures
Use Pixel Recognition For Intelligent Decisions
Create Character Bots For Leveling Up In Games
Create Character Bots For Multiple Screen Games
Applications and Games Botted:
Windows Notepad
Windows Calculator
Windows Explorer
Firefox
3D Pinball for Windows
Flash Games World Of Pain And Hero's Arms

Click here for more information about Bot Programming: Intelligent Automation For Windows Applications And Games

Read More...