Project Folder structure

[dblek-stx]

While trying to work on a project that has a specific folder structure (ie. models is a folder and inside there are other files for each model), DeepNext causes an error in create_action_plan

deep_next.core.steps.action_plan.action_plan.ActionPlanValidationError: Target file '/path/to/project/app/models' is a directory, not a file

There should be a checker that looks into the folder and works out the structure of where the models/serializers etc. are in a project.

[stxpatryk]

Good one! I guess it's not the issue with project folder structure but with the LLM that is unable to select single file from directory. Instead of specifying one file it goes for the whole dir which is unexpected and causes the error. As you can see it's ActionPlanValidationError so the issue with "action plan" specifically.

After quick code base analysis this is where the error is raised:

def _validate_paths(action_plan: ActionPlan, root_path: Path) -> ActionPlan:
    """Validates the action plan.

    Raises:
        ActionPlanValidationError: If the action plan is invalid.
    """
    if not action_plan.ordered_steps:
        raise ActionPlanValidationError("Action has no steps.")

    for step in action_plan.ordered_steps:
        try:
            step.target_file = try_to_resolve_path(step.target_file, root_path)
        except FileNotFoundError as e:
            raise ActionPlanValidationError(str(e))

        if step.target_file.is_dir():
            raise ActionPlanValidationError(
                f"Target file '{step.target_file}' is a directory, not a file"
            )

        if not step.target_file.is_relative_to(root_path):
            raise ActionPlanValidationError(
                f"Target file '{step.target_file}' is not part of project dir "
                f"'{root_path}'"
            )

    return action_plan

specifically if step.target_file.is_dir(). This is validation for the LLM output.

Do You have an idea how you would resolve the issue?

[dblek-stx]

I'm thinking of running this recursively to check if its a folder, then try to go inside the folder and check the files and folders inside... the folder/file case should not be an issue...

another resolve would be FIRST get the whole project structure and then start the analysis from that

[stxpatryk]

It is rather prompt related thing. If you go and see the prompt for action plan or expected output model, we require single step to have single related file:

class Step(BaseModel):
    title: str = Field(description="High level step overview.")
    description: str = Field(description="Detailed step description.")
    target_file: Path = Field(description="Absolute path to the file to be modified.")


class ActionPlan(BaseModel):
    reasoning: str
    ordered_steps: list[Step]

LLM returns a directory instead of file. This is something to be either done via the prompt (emphasis that we require single file per single step) or unpacking the directory e.g. if directory has multiple files then populate action plan for each file copy-pasting the description. The second seems to be error prone.

I'd suggest to double check the prompt. Or use some engineering to improve the LLM output. This is LLM realted issue with output quality from my perspective.

Using some follow-up recursive checks might be overkill. And refering to second idea - project structure is already provided via the project_knowledge

@retry(
    stop=stop_after_attempt(3),
    retry=retry_if_exception_type((OutputParserException, ActionPlanValidationError)),
    reraise=True,
)
def create_action_plan(
    root_path: Path,
    issue_statement: str,
    existing_code_context: ExistingCodeContext,
    project_knowledge: str,
) -> ActionPlan:
    """Creates structured and dependency-ordered action plan for solving the issue."""
    parser = PydanticOutputParser(pydantic_object=ActionPlan)

    prompt = ChatPromptTemplate.from_messages(
        [
            ("system", _Prompt.role),
            ("human", _Prompt.issue_statement),
            ("human", _Prompt.project_knowledge),
            ("human", _Prompt.existing_code_snippet),
            ("human", _Prompt.output_requirements),
            ("ai", _Prompt.ai_asks_for_example),
            ("human", _Prompt.example_action_plan),
        ],
    ).partial(
        format_instructions=parser.get_format_instructions(),
        example_action_plan=example.action_plan,
    )

    action_plan = (prompt | _create_llm() | parser).invoke(
        {
            "issue_statement": issue_statement,
            "project_knowledge": project_knowledge,
            "existing_code_snippets": existing_code_context.dump(),
        }
    )

    return _validate_paths(action_plan, root_path)

What do you think?

[dblek-stx]

I am actually testing DeepNext's implementation - it actually had the same thought process of recursively creating the folder structure but after changes it failed :) I will try to rephrase my prompt and try again :)

[stxpatryk]

can You share DeepNext PR? I'm curious