Summary

Fixed a critical bug in AsyncUrlSeeder where _resolve_head() was incorrectly returning redirect targets without verifying they were alive. This could cause dead URLs to be treated as valid during URL discovery.
#1603
Key Changes:

• Added verify_redirect_targets parameter to AsyncUrlSeeder.__init__() (default: True)
• Modified _resolve_head() to conditionally verify redirect targets based on the parameter
• Enhanced documentation with parameter descriptions
• Added comprehensive test suite for regression testing

Backward Compatibility: Existing code continues to work with improved behavior by default. Users can set verify_redirect_targets=False to restore the previous behavior if needed.

List of files changed and why

• crawl4ai/async_url_seeder.py - Core bug fix and parameter addition
• tests/test_async_url_seeder.py - Added unit tests for both verification modes
• test_scripts/test_async_url_seeder_fixes.py - Comprehensive demo/test suite for all fixes
• test_scripts/README.md - Documentation for test scripts

How Has This Been Tested?

Created comprehensive test suite covering:

• ✅ Dead redirect targets are filtered out (default behavior)
• ✅ Legacy behavior preserved when verify_redirect_targets=False
• ✅ Valid redirects still work correctly
• ✅ Direct URL validation unchanged
• ✅ Context manager functionality preserved
• ✅ All existing AsyncUrlSeeder functionality works
• ✅ Integration with SeedingConfig.live_check parameter

Run the test suite with: python test_scripts/test_async_url_seeder_fixes.py

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes