diff --git a/README.md b/README.md index 8f2546e..e59f0a9 100644 --- a/README.md +++ b/README.md @@ -90,6 +90,171 @@ The `import` command also supports input in the [rosinstall file format](http:// Only for this command vcs2l supports the pseudo clients `tar` and `zip` which fetch a tarball / zipfile from a URL and unpack its content. For those two types the `version` key is optional. If specified only entries from the archive which are in the subfolder specified by the version value are being extracted. +### Import with extends functionality + +The `vcs import` command supports an `extends` key at the top level of the YAML file. The value of that key is a path or URL to another YAML file which is imported first. +This base file can itself also contain the key to chain multiple files. The extension to this base file is given precedence over the parent in case of duplicate repository entries. + +#### Normal Extension + +For instance, consider the following two files: + +- **`base.repos`**: contains three repositories `vcs2l`, `immutable/hash` and `immutable/tag`, checked out at specific versions. + + ```yaml + --- + repositories: + vcs2l: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: main + immutable/hash: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 377d5b3d03c212f015cc832fdb368f4534d0d583 + immutable/tag: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 1.1.3 + ``` + +- **`base_extension.repos`**: extends the base file and overrides the version of `immutable/hash` and `immutable/tag` repositories. + + ```yaml + --- + extends: base.repos + repositories: + immutable/hash: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 25e4ae2f1dd28b0efcd656f4b1c9679d8a7d6c22 + immutable/tag: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 1.1.5 + ``` +The resulting extension import would import vcs2l at version `main`, `immutable/hash` at version `25e4ae2` and `immutable/tag` at version `1.1.5`. + +#### Multiple Extensions + +The `extends` key also supports a list of files to extend from. The files are imported in the order they are specified and the precedence is given to the last file in case of duplicate repository entries. + +For instance, consider the following three files: + +- **`base_1.repos`**: contains two repositories `vcs2l` and `immutable/hash`, checked out at `1.1.3`. + + ```yaml + --- + repositories: + immutable/hash: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: e700793cb2b8d25ce83a611561bd167293fd66eb # 1.1.3 + vcs2l: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 1.1.3 + ``` + +- **`base_2.repos`**: contains two repositories `vcs2l` and `immutable/hash`, checked out at `1.1.4`. + + ```yaml + --- + repositories: + immutable/hash: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 2c7ff89d12d8a77c36b60d1f7ba3039cdd3f742b # 1.1.4 + vcs2l: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 1.1.4 + ``` + +- **`multiple_extension.repos`**: extends both base files and overrides the version of `vcs2l` repository. + + ```yaml + --- + extends: + - base_1.repos # Lower priority + - base_2.repos # Higher priority + repositories: + vcs2l: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 1.1.5 + ``` + +The resulting extension import would import `immutable/hash` at version `1.1.4` (from `base_2.repos`) and `vcs2l` at version `1.1.5`. + +Duplicate file names in the `extends` list are not allowed and would raise the following error: + +```bash +Duplicate entries found in extends in file: /multiple_extension.repos +``` + +#### Circular Loop Protection + +In order to avoid infinite loops in case of circular imports the tool detects already imported files and raises an error if such a file is encountered again. + +For instance, consider the following two files: + +- **`loop_base.repos`**: extends the `loop_extension.repos` file, and contains two repositories `vcs2l` and `immutable/tag`. + + ```yaml + --- + extends: loop_extension.repos + repositories: + vcs2l: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: main + immutable/tag: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 1.1.3 + ``` + +- **`loop_extension.repos`**: extends the `loop_base.repos` file, and modifies the version of `immutable/tag` with `1.1.5`. + + ```yaml + --- + extends: loop_base.repos + repositories: + immutable/tag: + type: git + url: https://github.com/ros-infrastructure/vcs2l.git + version: 1.1.5 + ``` +The resulting extension import would prevent the download and raise the following error: + +```bash +Circular import detected: /loop_extension.repos +``` + +#### File path behaviour + +Currently there are two ways to specify the path to the repository file passed to `vcs import`: + +1. **Recommended**: Using `--input`. + + * For instance: `vcs import --input my.repos ` + + * The extended files are searched relative to the file containing the `extends` key. + + * You do not require to be in the same directory as `my.repos` to run the command. + +2. Using the input redirection operator `<` to pass a local file path via `stdin`. + + * For instance: `vcs import < my.repos ` + + * The extended files are searched relative to the current working directory. + + * Therefore, you have to be in the **same** directory as `my.repos` to run the command. + + The files being directly extended by the file provided through `stdin` are relative to the current working directory. + Any other file being extended is relative to the file extending it. + ### Delete set of repositories The `vcs delete` command removes all directories of repositories which are passed in via `stdin` in YAML format. diff --git a/test/__init__.py b/test/__init__.py index 53174f9..a2ed96b 100644 --- a/test/__init__.py +++ b/test/__init__.py @@ -12,6 +12,8 @@ import yaml +from .repos_factory import generate_extends_repos + def to_file_url(path): return Path(path).as_uri() @@ -65,6 +67,13 @@ def setUpClass(cls): ('commit', '--quiet', '--allow-empty', '-m', '0.1.27'), ('tag', '0.1.27', '-m', '0.1.27'), ('commit', '--quiet', '--allow-empty', '-m', "codin' codin' codin'"), + # Additional commits and tags for extends testing + ('commit', '--quiet', '--allow-empty', '-m', '1.1.3'), + ('tag', '1.1.3', '-m', '1.1.3'), + ('commit', '--quiet', '--allow-empty', '-m', '1.1.4'), + ('tag', '1.1.4', '-m', '1.1.4'), + ('commit', '--quiet', '--allow-empty', '-m', '1.1.5'), + ('tag', '1.1.5', '-m', '1.1.5'), ): subprocess.check_call( [ @@ -75,6 +84,20 @@ def setUpClass(cls): env=cls._git_env, ) + # Capture commit hashes for the extends testing tags + cls._tag_hashes = {} + for tag in ('1.1.3', '1.1.4', '1.1.5'): + hash_val = ( + subprocess.check_output( + [cls._git, 'rev-parse', tag + '^{commit}'], + cwd=gitrepo_path, + env=cls._git_env, + ) + .decode() + .strip() + ) + cls._tag_hashes[tag] = hash_val + # Create the archive stage archive_path = os.path.join(cls.temp_dir.name, 'archive_dir') os.mkdir(archive_path) @@ -131,6 +154,20 @@ def setUpClass(cls): with open(cls.repos_file_path, 'wb') as f: yaml.safe_dump({'repositories': repos}, f, encoding='utf-8') + # Generate extends repos files for inheritance testing. + # Extension files reference staged.repos directly as their base. + gitrepo_url = to_file_url(gitrepo_path) + extends_paths = generate_extends_repos( + cls.temp_dir.name, gitrepo_url, cls._tag_hashes + ) + cls.staged_extension_repos_path = extends_paths['staged_extension'] + cls.staged_extension_2_repos_path = extends_paths['staged_extension_2'] + cls.staged_multiple_extension_repos_path = extends_paths[ + 'staged_multiple_extension' + ] + cls.loop_base_repos_path = extends_paths['loop_base'] + cls.loop_extension_repos_path = extends_paths['loop_extension'] + @classmethod def tearDownClass(cls): cls.repos_file_path = None diff --git a/test/repos_factory.py b/test/repos_factory.py new file mode 100644 index 0000000..0fffd62 --- /dev/null +++ b/test/repos_factory.py @@ -0,0 +1,111 @@ +"""Factory for generating .repos files used in tests.""" + +import os + +import yaml + + +def _dump(data: dict, path: str): + """Write a YAML repos file.""" + with open(path, 'wb') as f: + yaml.safe_dump(data, f, encoding='utf-8') + + +def _git_repo(url: str, version: str) -> dict: + return {'type': 'git', 'url': url, 'version': version} + + +def generate_extends_repos(temp_dir: str, git_url: str, tag_hashes: dict) -> dict: + """Generate all extends .repos files for inheritance testing. + + The generated files use `staged.repos` present in `temp_dir` as + the base file that extension repos inherit from via `extends:`. + + Args: + temp_dir: Path to the temporary directory that already contains + `staged.repos`. + git_url: file:// URL to the local staged git repository. + tag_hashes: Dict mapping tag names ('1.1.3', '1.1.4', '1.1.5') + to their commit hashes. + + Returns: + Dict with paths to all generated repos files: + - staged_extension + - staged_extension_2 + - staged_multiple_extension + - loop_base + - loop_extension + """ + paths = {} + + # staged_extension.repos — extends staged.repos, overrides with tag 1.1.3 + paths['staged_extension'] = os.path.join(temp_dir, 'staged_extension.repos') + _dump( + { + 'extends': 'staged.repos', + 'repositories': { + 'immutable/hash': _git_repo(git_url, tag_hashes['1.1.3']), + 'immutable/tag': _git_repo(git_url, 'tags/1.1.3'), + 'vcs2l': _git_repo(git_url, '1.1.3'), + }, + }, + paths['staged_extension'], + ) + + # staged_extension_2.repos — extends staged.repos, overrides with tag 1.1.4 + paths['staged_extension_2'] = os.path.join(temp_dir, 'staged_extension_2.repos') + _dump( + { + 'extends': 'staged.repos', + 'repositories': { + 'immutable/hash': _git_repo(git_url, tag_hashes['1.1.4']), + 'immutable/tag': _git_repo(git_url, 'tags/1.1.4'), + 'vcs2l': _git_repo(git_url, '1.1.4'), + }, + }, + paths['staged_extension_2'], + ) + + # staged_multiple_extension.repos — extends both extensions + paths['staged_multiple_extension'] = os.path.join( + temp_dir, 'staged_multiple_extension.repos' + ) + _dump( + { + 'extends': [ + 'staged_extension.repos', + 'staged_extension_2.repos', + ], + 'repositories': { + 'immutable/tag': _git_repo(git_url, 'tags/1.1.5'), + 'vcs2l': _git_repo(git_url, 'heads/main'), + }, + }, + paths['staged_multiple_extension'], + ) + + # loop_extension.repos / loop_base.repos — circular import pair + paths['loop_extension'] = os.path.join(temp_dir, 'loop_extension.repos') + _dump( + { + 'extends': 'loop_base.repos', + 'repositories': { + 'vcs2l': _git_repo(git_url, '1.1.3'), + }, + }, + paths['loop_extension'], + ) + + paths['loop_base'] = os.path.join(temp_dir, 'loop_base.repos') + _dump( + { + 'extends': 'loop_extension.repos', + 'repositories': { + 'vcs2l': _git_repo(git_url, 'heads/main'), + 'immutable/tag': _git_repo(git_url, 'tags/1.1.3'), + }, + }, + paths['loop_base'], + ) + + return paths diff --git a/test/test_commands.py b/test/test_commands.py index 151deef..96bf555 100644 --- a/test/test_commands.py +++ b/test/test_commands.py @@ -11,6 +11,11 @@ from vcs2l.util import rmtree from . import StagedReposFile, StagedReposFile2, to_file_url +from .test_utils import ( + assert_base_repos_imported, + assert_git_at_commit, + assert_git_at_tag, +) sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) @@ -320,6 +325,63 @@ def test_deletion(self): finally: rmtree(workdir) + def test_import_extends(self): + """Test import with extends functionality.""" + workdir = os.path.join(TEST_WORKSPACE, 'import_extends') + os.makedirs(workdir) + try: + run_command( + 'import', + ['--input', self.staged_extension_repos_path, '.'], + subfolder='import_extends', + ) + # Verify base repos from staged.repos were imported + assert_base_repos_imported(workdir) + + # Verify overridden repos are at the correct version (1.1.3) + assert_git_at_commit( + os.path.join(workdir, 'immutable', 'hash'), + self._tag_hashes['1.1.3'], + ) + assert_git_at_tag( + os.path.join(workdir, 'immutable', 'tag'), + '1.1.3', + ) + finally: + rmtree(workdir) + + def test_import_extends_loop(self): + """Test import with extends functionality that creates a circular import.""" + with self.assertRaises(subprocess.CalledProcessError) as e: + run_command('import', ['--input', self.loop_extension_repos_path, '.']) + self.assertIn(b'Circular import detected:', e.exception.output) + + def test_import_multiple_extends(self): + """Test import with multiple extends functionality.""" + workdir = os.path.join(TEST_WORKSPACE, 'import_multiple_extends') + os.makedirs(workdir) + try: + run_command( + 'import', + ['--input', self.staged_multiple_extension_repos_path, '.'], + subfolder='import_multiple_extends', + ) + assert_base_repos_imported(workdir) + + # Verify the highest-priority extension overrides (1.1.4 from + # staged_extension_2.repos) for immutable/hash + assert_git_at_commit( + os.path.join(workdir, 'immutable', 'hash'), + self._tag_hashes['1.1.4'], + ) + # Verify the top-level file overrides immutable/tag to 1.1.5 + assert_git_at_tag( + os.path.join(workdir, 'immutable', 'tag'), + '1.1.5', + ) + finally: + rmtree(workdir) + def test_validate(self): output = run_command('validate', ['--input', self.repos_file_path]) expected = get_expected_output('validate') diff --git a/test/test_utils.py b/test/test_utils.py new file mode 100644 index 0000000..beb5f12 --- /dev/null +++ b/test/test_utils.py @@ -0,0 +1,39 @@ +"""Utilities for vcs2l tests.""" + +import os +import subprocess + + +def assert_base_repos_imported(workdir: str): + """Assert that the base repos from staged.repos exist in workdir.""" + assert os.path.isdir(os.path.join(workdir, 'immutable', 'hash_tar')) + assert os.path.isdir(os.path.join(workdir, 'immutable', 'hash_zip')) + assert os.path.isdir(os.path.join(workdir, 'without_version')) + + +def assert_git_at_commit(repo_dir: str, expected_hash: str): + """Assert that a git repo's HEAD is at the expected commit hash.""" + actual = ( + subprocess.check_output( + ['git', 'rev-parse', 'HEAD'], + cwd=repo_dir, + stderr=subprocess.STDOUT, + ) + .decode() + .strip() + ) + assert actual == expected_hash, f'Expected {expected_hash}, got {actual}' + + +def assert_git_at_tag(repo_dir: str, expected_tag: str): + """Assert that a git repo's HEAD is at the expected tag.""" + actual = ( + subprocess.check_output( + ['git', 'describe', '--tags', '--exact-match'], + cwd=repo_dir, + stderr=subprocess.STDOUT, + ) + .decode() + .strip() + ) + assert actual == expected_tag, f'Expected tag {expected_tag}, got {actual}' diff --git a/vcs2l/commands/import_.py b/vcs2l/commands/import_.py index 54af290..d07cb81 100644 --- a/vcs2l/commands/import_.py +++ b/vcs2l/commands/import_.py @@ -11,6 +11,7 @@ from vcs2l.clients.none import NoneClient from vcs2l.clients.vcs_base import run_command from vcs2l.commands.command import Command, add_common_arguments +from vcs2l.errors import CircularImportError from vcs2l.executor import ansi, execute_jobs, output_repositories, output_results from vcs2l.streams import set_streams @@ -87,23 +88,94 @@ def file_or_url_type(value): return request.Request(value, headers={'User-Agent': 'vcs2l/' + vcs2l_version}) -def get_repositories(yaml_file): +def load_yaml_file(yaml_file): + """Load and parse a YAML file.""" try: - root = yaml.safe_load(yaml_file) + return yaml.safe_load(yaml_file) except yaml.YAMLError as e: - raise RuntimeError('Input data is not valid yaml format: %s' % e) + raise RuntimeError('Input data is not valid yaml format: %s' % e) from e + +def get_repositories_from_root(root): + """Extract repositories from the parsed YAML root object.""" try: repositories = root['repositories'] return get_repos_in_vcs2l_format(repositories) except KeyError as e: - raise RuntimeError('Input data is not valid format: %s' % e) + raise RuntimeError('Input data is not valid format: %s' % e) from e except TypeError as e: # try rosinstall file format try: return get_repos_in_rosinstall_format(root) except Exception: - raise RuntimeError('Input data is not valid format: %s' % e) + raise RuntimeError('Input data is not valid format: %s' % e) from e + + +def get_repositories(yaml_file, visited_files=None): + """Recursively get repositories from a YAML file, handling inheritance.""" + if visited_files is None: + visited_files = set() + + # Get absolute path to handle relative paths consistently + current_file_path = os.path.abspath(yaml_file.name) + + if current_file_path in visited_files: + raise CircularImportError(f'Circular import detected: {current_file_path}') + + visited_files.add(current_file_path) + + try: + root = load_yaml_file(yaml_file) + + combined_repos = {} + + if 'extends' in root: + parent_files = root['extends'] + # Convert single file to list for consistent processing + if isinstance(parent_files, str): + parent_files = [parent_files] + + # Check for duplicate entries in extends + if len(parent_files) != len(set(parent_files)): + raise RuntimeError( + f'Duplicate entries found in extends in file: {current_file_path}' + ) + + for parent_file in parent_files: + # If absolute path is not valid, try relative to current file + if not os.path.isabs(parent_file): + current_dir = os.path.dirname(current_file_path) + parent_file = os.path.join(current_dir, parent_file) + + if not os.path.exists(parent_file): + raise RuntimeError(f'Parent file not found: {parent_file}') + + try: + # Recursively get repositories from parent file + with open(parent_file, 'r', encoding='utf-8') as parent_f: + parent_repos = get_repositories(parent_f, visited_files.copy()) + combined_repos.update(parent_repos) + + except CircularImportError: + raise + + except Exception as e: + raise RuntimeError( + f'Error reading parent file {parent_file}: \n{str(e)}' + ) from e + + current_repos = get_repositories_from_root(root) + combined_repos.update(current_repos) + + return combined_repos + + except FileNotFoundError as e: + raise RuntimeError(f'File not found: {yaml_file}') from e + except yaml.YAMLError as e: + raise RuntimeError(f'Error parsing YAML file {yaml_file}: {str(e)}') from e + finally: + # Remove current file from visited set when leaving this call + visited_files.discard(current_file_path) def get_repos_in_vcs2l_format(repositories): diff --git a/vcs2l/errors.py b/vcs2l/errors.py index fc7ba30..fe88a1b 100644 --- a/vcs2l/errors.py +++ b/vcs2l/errors.py @@ -21,3 +21,10 @@ def __init__(self, min_version: str = '3.6'): f'vcs2l requires Python {min_version} or higher.' ) super().__init__(message) + + +class CircularImportError(Vcs2lError): + """Raised when a circular import is detected.""" + + def __init__(self, message: str = 'Circular import detected.'): + super().__init__(message)