- Add proper file seeking to end of file before resuming downloads
- Improve range request logic to only apply when file_size > 0
- Set initial progress bar position for resumed downloads
- Add file.flush() to ensure data is written before hash verification
- Conditionally show download stats only when bytes were downloaded
Fixes regression from v0.1.0 where interrupted downloads would fail hash verification due to corrupted file content when resumed.
- Combine XML URL generation and fetching into single function
- Consolidate multi-step XML derivation, validation, and parsing flow
- Reduce intermediate error handling steps for better readability
- Simplify XML metadata processing from discrete steps to unified operation
- Relocate signal handler setup from main to downloader module
- Encapsulate signal handling as internal concern of download operations
- Simplify main function by removing signal-related complexity
- Reduce coupling between application initialization and download control
- Create new constants.rs module for centralised configuration
- Move USER_AGENT, HTTP_TIMEOUT, and URL_PATTERN from main.rs
- Update imports in main.rs to use constants module
- Add constants module export to lib.rs
- Consolidate HTTP timeout constants to single HTTP_TIMEOUT value
- Update tests to compile regex locally for validation
- Remove Unused Imports and Comments
Created a new `downloader` module (`src/downloader.rs`) and moved all functions related to file downloading, content streaming, MD5 hashing, file existence checks, and directory creation from `main.rs` into this new module.
This change includes:
- Moving functions: `download_file`, `download_file_content`, `check_existing_file`, `verify_downloaded_file`, `calculate_md5`, `prepare_file_for_download`, and `ensure_parent_directories`.
- Relocating relevant constants (`BUFFER_SIZE`, `LARGE_FILE_THRESHOLD`) to `downloader.rs`.
- Updating `main.rs` to use `downloader::download_file` and removing the now redundant local functions and associated unused imports.
- Adding `pub mod downloader;` to `lib.rs`.
This refactoring improves code organisation by centralising file operations, enhancing maintainability, and promoting better separation of concerns.
Refactored XML handling by:
- Moving `XmlFiles` and `XmlFile` struct definitions from `main.rs` to a new `src/archive_metadata.rs` module.
- Updating `main.rs` to import these structures from the new module using `ia_get::archive_metadata`.
This commit improves the structure and maintainability of the `ia-get` by extracting general utility functions from `src/main.rs` into a new dedicated `src/utils.rs` module. This centralises common helper functions and slims down `main.rs`.
Extract hardcoded values to named constants for better maintainability and readability. This includes:
- BUFFER_SIZE for file operations
- LARGE_FILE_THRESHOLD for progressive hash calculation
- USER_AGENT for HTTP requests
- DEFAULT_HTTP_TIMEOUT for HTTP request timeouts
- URL_CHECK_TIMEOUT for accessibility checks
- SPINNER_TICK_INTERVAL for UI feedback
- PATTERN for URL validation
This change improves code clarity and makes configuration changes easier by centralising magic values in one place.
This enhances user experience by providing visual feedback during potentially
slow network operations.
- Add animated braille spinner for feedback during URL processing
- Show progress through URL validation, XML metadata fetching and parsing
- Replace spinner with green tick when processing completes successfully
- Improve error display with appropriate icons for failed operations
- Provide file count summary upon successful initialisation
- Show download size, duration, and transfer speed when downloads finish
- Add helper functions for formatting sizes, durations, and transfer rates
- Update download_file_content to track and report download statistics
- Use consistent UI elements with existing tree-like structure
- Extract file download logic into separate functions
- download_file: main entry point for file downloads
- check_existing_file: verify if file exists with correct hash
- ensure_parent_directories: create directories for downloads
- prepare_file_for_download: set up file for writing
- download_file_content: perform the actual download with progress
- verify_downloaded_file: validate downloaded file hash
- Optimize URL accessibility checking
- Modify is_url_accessible to accept an existing client
- Add timeouts to prevent hanging on slow connections
- Update call sites to reuse the client instance
These changes improve code modularity, maintainability and performance without changing user-facing behaviour.
- Implement Ctrl+C signal handler with Arc<AtomicBool> for thread-safe state
- Add signal interruption checks in download loops and MD5 calculation
- Preserve download progress when interrupted, allowing seamless resume
- Provide clear user feedback on interruption with resume instructions
- Ensure progress bars are properly cleaned up on signal interruption
This enables users to gracefully cancel long-running downloads while maintaining the ability to resume from where they left off by running the same command again.
- Create dedicated error module with custom error types
- Use thiserror for improved error messages and derivation
- Add specific error variants for different failure modes
- Implement From traits for automatic error conversion
- Replace generic error types with custom Result<T>
- Update function signatures for consistent error handling
- Improve error messages for better user feedback
This change follows Rust best practices for error handling and makes debugging easier with more descriptive error messages. It also provides proper error propagation using the ? operator while maintaining type safety throughout the application.
This change ensures memory usage remains constant regardless of file size, allowing the application to efficiently verify multi-gigabyte files from Internet Archive without excessive RAM consumption.
- Implement chunked file reading to prevent excessive memory usage
- Add progress bar for large file hash calculation (>100MB)
- Use BufReader with fixed buffer size (8KB) for efficient I/O
- Display visual feedback during hash verification of large files
- Fix memory issue where entire file was loaded into RAM
The issue was introduced when upgrading serde-xml-rs to 0.8.0 which required explicit handling of attributes vs elements in the XML structure.
- Adjust XML parsing to handle nested elements correctly
- Make source attribute optional with proper default handling
- Add safer MD5 hash verification with proper None handling
- Update struct documentation to match actual XML format
Bug: archive.org uses XML attributes for name/source but nested elements for metadata like md5/mtime, causing deserialization failures.
Updates dependencies to their latest compatible versions, including clap, anstyle, and windows-sys, ensuring compatibility and potentially incorporating bug fixes and performance improvements from upstream dependencies.
BREAKING CHANGE: cargo.lock version is updated from 3 to 4
- Replace old clap builder API with modern derive macros for CLI parsing
- Fix is_url_accessible() to use HEAD requests and proper error propagation
- Add comprehensive documentation comments for all functions and structs
- Add unit tests for URL pattern validation with valid/invalid test cases
- Improve error handling by replacing panic! with proper exit codes
- Set user-agent header to "ia-get" for better internet citizenry
- Update dependencies to use clap 4.0 with derive features
Co-authored-by: Daniel Dewberry <[email protected]>
Implements suggestions from podcast listener feedback received after Linux Matters Episode 20, addressing AI-generated code deficits with proper Rust ecosystem practices and modern idioms.
Close: #7
Adds useful tools and environment variables to improve the rust development experience within the nix environment. This includes tools like git, gh, nodejs, cargo-watch, cargo-edit, clippy, and rustfmt.
It also configures a shell hook to provide information and sets RUST_BACKTRACE to 1 for easier debugging.
Adds categories, description, keywords, repository, and readme to the Cargo.toml. This provides more information about the crate on crates.io and helps with discovery.
Martin Wimpress