This commit enhances the user interface by adding coloured glyphs/symbols instead of emoji.
- Added the coloured crate as a dependency.
- Modified downloader.rs to incorporate coloured output for download status, MD5 hashing, and error messages.
- Updated main.rs to use colours in spinner messages and validation feedback.
- Modified utils.rs to format progress bar messages with colours.
- Reduced the LARGE_FILE_THRESHOLD in downloader.rs from 16MB to 2MB, to trigger hashing progress bar more frequently.
- Adds a comment explaining why the unwrap operation on the URL identifier is considered safe.
- Clarifies that the URL is validated before this operation, ensuring a valid identifier segment exists.
- Improves archive.org URL validation to ensure the identifier after "/details/" is not empty.
- Updates error message for invalid URLs to provide a more descriptive explanation of the expected format.
- Streamlines error handling in the main function by directly displaying the error message from the validation function.
Removes direct calls to `std::process::exit` from `fetch_xml_metadata` and error handling blocks within `main`. These functions now propagate errors by returning `Result`.
This change improves testability and aligns with idiomatic Rust error handling, allowing `main` to be the single point of exit for the application on error. Spinner messages for context are retained.
The `get_xml_url` function has been updated to ensure consistent identifier extraction from archive.org details URLs, regardless of the presence of a trailing slash. This is achieved by trimming any trailing slash before splitting the URL to find the identifier.
The corresponding unit tests in `check_get_xml_url` have been revised to accurately reflect and verify this behavior, covering cases with and without trailing slashes for various URL patterns. This ensures the correct formation of the XML metadata URL.
- Add proper file seeking to end of file before resuming downloads
- Improve range request logic to only apply when file_size > 0
- Set initial progress bar position for resumed downloads
- Add file.flush() to ensure data is written before hash verification
- Conditionally show download stats only when bytes were downloaded
Fixes regression from v0.1.0 where interrupted downloads would fail hash verification due to corrupted file content when resumed.
- Combine XML URL generation and fetching into single function
- Consolidate multi-step XML derivation, validation, and parsing flow
- Reduce intermediate error handling steps for better readability
- Simplify XML metadata processing from discrete steps to unified operation
- Relocate signal handler setup from main to downloader module
- Encapsulate signal handling as internal concern of download operations
- Simplify main function by removing signal-related complexity
- Reduce coupling between application initialization and download control
- Create new constants.rs module for centralised configuration
- Move USER_AGENT, HTTP_TIMEOUT, and URL_PATTERN from main.rs
- Update imports in main.rs to use constants module
- Add constants module export to lib.rs
- Consolidate HTTP timeout constants to single HTTP_TIMEOUT value
- Update tests to compile regex locally for validation
- Remove Unused Imports and Comments
Created a new `downloader` module (`src/downloader.rs`) and moved all functions related to file downloading, content streaming, MD5 hashing, file existence checks, and directory creation from `main.rs` into this new module.
This change includes:
- Moving functions: `download_file`, `download_file_content`, `check_existing_file`, `verify_downloaded_file`, `calculate_md5`, `prepare_file_for_download`, and `ensure_parent_directories`.
- Relocating relevant constants (`BUFFER_SIZE`, `LARGE_FILE_THRESHOLD`) to `downloader.rs`.
- Updating `main.rs` to use `downloader::download_file` and removing the now redundant local functions and associated unused imports.
- Adding `pub mod downloader;` to `lib.rs`.
This refactoring improves code organisation by centralising file operations, enhancing maintainability, and promoting better separation of concerns.
Refactored XML handling by:
- Moving `XmlFiles` and `XmlFile` struct definitions from `main.rs` to a new `src/archive_metadata.rs` module.
- Updating `main.rs` to import these structures from the new module using `ia_get::archive_metadata`.
This commit improves the structure and maintainability of the `ia-get` by extracting general utility functions from `src/main.rs` into a new dedicated `src/utils.rs` module. This centralises common helper functions and slims down `main.rs`.
Extract hardcoded values to named constants for better maintainability and readability. This includes:
- BUFFER_SIZE for file operations
- LARGE_FILE_THRESHOLD for progressive hash calculation
- USER_AGENT for HTTP requests
- DEFAULT_HTTP_TIMEOUT for HTTP request timeouts
- URL_CHECK_TIMEOUT for accessibility checks
- SPINNER_TICK_INTERVAL for UI feedback
- PATTERN for URL validation
This change improves code clarity and makes configuration changes easier by centralising magic values in one place.
This enhances user experience by providing visual feedback during potentially
slow network operations.
- Add animated braille spinner for feedback during URL processing
- Show progress through URL validation, XML metadata fetching and parsing
- Replace spinner with green tick when processing completes successfully
- Improve error display with appropriate icons for failed operations
- Provide file count summary upon successful initialisation
- Show download size, duration, and transfer speed when downloads finish
- Add helper functions for formatting sizes, durations, and transfer rates
- Update download_file_content to track and report download statistics
- Use consistent UI elements with existing tree-like structure
- Extract file download logic into separate functions
- download_file: main entry point for file downloads
- check_existing_file: verify if file exists with correct hash
- ensure_parent_directories: create directories for downloads
- prepare_file_for_download: set up file for writing
- download_file_content: perform the actual download with progress
- verify_downloaded_file: validate downloaded file hash
- Optimize URL accessibility checking
- Modify is_url_accessible to accept an existing client
- Add timeouts to prevent hanging on slow connections
- Update call sites to reuse the client instance
These changes improve code modularity, maintainability and performance without changing user-facing behaviour.
- Implement Ctrl+C signal handler with Arc<AtomicBool> for thread-safe state
- Add signal interruption checks in download loops and MD5 calculation
- Preserve download progress when interrupted, allowing seamless resume
- Provide clear user feedback on interruption with resume instructions
- Ensure progress bars are properly cleaned up on signal interruption
This enables users to gracefully cancel long-running downloads while maintaining the ability to resume from where they left off by running the same command again.
- Create dedicated error module with custom error types
- Use thiserror for improved error messages and derivation
- Add specific error variants for different failure modes
- Implement From traits for automatic error conversion
- Replace generic error types with custom Result<T>
- Update function signatures for consistent error handling
- Improve error messages for better user feedback
This change follows Rust best practices for error handling and makes debugging easier with more descriptive error messages. It also provides proper error propagation using the ? operator while maintaining type safety throughout the application.
This change ensures memory usage remains constant regardless of file size, allowing the application to efficiently verify multi-gigabyte files from Internet Archive without excessive RAM consumption.
- Implement chunked file reading to prevent excessive memory usage
- Add progress bar for large file hash calculation (>100MB)
- Use BufReader with fixed buffer size (8KB) for efficient I/O
- Display visual feedback during hash verification of large files
- Fix memory issue where entire file was loaded into RAM
The issue was introduced when upgrading serde-xml-rs to 0.8.0 which required explicit handling of attributes vs elements in the XML structure.
- Adjust XML parsing to handle nested elements correctly
- Make source attribute optional with proper default handling
- Add safer MD5 hash verification with proper None handling
- Update struct documentation to match actual XML format
Bug: archive.org uses XML attributes for name/source but nested elements for metadata like md5/mtime, causing deserialization failures.
dependabot[bot]