Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Better intermediate output #1039

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
aantn wants to merge 2 commits into master
base: master
Choose a base branch
Loading
from better-intermediate-output
Draft

Better intermediate output #1039

aantn wants to merge 2 commits into master from better-intermediate-output

Conversation

Copy link
Contributor

@aantn aantn commented Oct 7, 2025

No description provided.

Copy link
Contributor

coderabbitai bot commented Oct 7, 2025
edited
Loading

Walkthrough

Adds a new "Communication during investigation" section to the general instructions template, defining when and how to pause and share findings during tool batches and task progression, emphasizing concise, critical updates and next steps. No public/exported interfaces are changed.

Changes

Cohort / File(s) Summary of Changes
Prompt guidance updates
holmes/plugins/prompts/_general_instructions.jinja2
Introduces a "Communication during investigation" section with standardized pause-and-update guidance: format, scope (critical discoveries), conciseness, and next-step expectations; inserted at multiple points within the general investigation/task-management instructions.

Sequence Diagram(s)

sequenceDiagram
 autonumber
 actor Investigator
 participant ToolRunner as Tool Runner
 participant Stakeholder as Stakeholder
 Investigator->>ToolRunner: Execute tool batch
 ToolRunner-->>Investigator: Results
 note over Investigator,Stakeholder: New step: standardized pause-and-update
 Investigator->>Stakeholder: Communicate concise findings + critical items + next steps
 alt More work needed
 Investigator->>ToolRunner: Proceed with next tasks/batch
 else Sufficient findings
 Investigator->>Stakeholder: Finalize/await direction
 end
Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Pre-merge checks and finishing touches

❌ Failed checks (1 warning, 1 inconclusive)
Check name Status Explanation Resolution
Description Check ⚠️ Warning No pull request description was provided, so there is no information summarizing the addition of the new communication guidelines or the context of the changes. This absence means the description does not describe any part of the changeset. Please add a pull request description that summarizes the addition of the "Communication during investigation" section and explains the purpose and scope of the new guidelines.
Title Check ❓ Inconclusive The title "Better intermediate output" is too vague to convey the main change of adding a new "Communication during investigation" section and associated guidelines. It does not reference the structured update approach introduced in the changeset or indicate what specific improvement is being made. Consequently, the title does not adequately summarize the primary focus of the pull request. Please revise the title to clearly reflect the addition of communication guidelines for intermediate investigation outputs, for example "Add communication guidelines for intermediate investigation outputs."
✅ Passed checks (1 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changes. Docstring coverage check skipped.
✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch better-intermediate-output

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 7a4fe4c and 95fcf6c.

📒 Files selected for processing (1)
  • holmes/plugins/prompts/_general_instructions.jinja2 (1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
holmes/plugins/prompts/**/*.jinja2

📄 CodeRabbit inference engine (CLAUDE.md)

Prompts must be stored as Jinja2 templates under holmes/plugins/prompts/{name}.jinja2

Files:

  • holmes/plugins/prompts/_general_instructions.jinja2
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
  • GitHub Check: llm_evals
  • GitHub Check: Pre-commit checks
🔇 Additional comments (1)
holmes/plugins/prompts/_general_instructions.jinja2 (1)

68-76: LGTM! Clear communication guidelines.

This section provides valuable guidance for providing timely updates during investigations. The instructions are clear, concise, and include a helpful example. The placement between task management and tool calls is logical.

* Format: Brief summary of discoveries + what you're checking next
* Example: "Found database connection pool at 95% with slow queries. Checking payment service dependencies..."
* Don't add extra communication points - only enhance existing ones with findings
* Focus on critical discoveries (errors, bottlenecks, root causes) not routine checks
Copy link
Contributor

@coderabbitai coderabbitai bot Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Minor grammar fix needed.

Missing comma before "not" for clarity.

Apply this diff:

-* Focus on critical discoveries (errors, bottlenecks, root causes) not routine checks
+* Focus on critical discoveries (errors, bottlenecks, root causes), not routine checks
📝 Committable suggestion

!!️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
* Focus on critical discoveries (errors, bottlenecks, root causes) not routine checks
* Focus on critical discoveries (errors, bottlenecks, root causes), not routine checks
🤖 Prompt for AI Agents
In holmes/plugins/prompts/_general_instructions.jinja2 around line 74, the
sentence "* Focus on critical discoveries (errors, bottlenecks, root causes) not
routine checks" is missing a comma for clarity; insert a comma after the closing
parenthesis so it reads "* Focus on critical discoveries (errors, bottlenecks,
root causes), not routine checks".

Copy link
Contributor

github-actions bot commented Oct 7, 2025

Results of HolmesGPT evals

  • ask_holmes: 32/35 test cases were successful, 1 regressions, 1 setup failures
Test suite Test case Status
ask 01_how_many_pods
ask 02_what_is_wrong_with_pod
ask 04_related_k8s_events
ask 05_image_version
ask 09_crashpod
ask 10_image_pull_backoff
ask 110_k8s_events_image_pull
ask 11_init_containers
ask 13a_pending_node_selector_basic
ask 14_pending_resources
ask 15_failed_readiness_probe
ask 17_oom_kill
ask 19_detect_missing_app_details
ask 20_long_log_file_search
ask 24_misconfigured_pvc
ask 24a_misconfigured_pvc_basic
ask 28_permissions_error 🚧
ask 39_failed_toolset
ask 41_setup_argo
ask 42_dns_issues_steps_new_tools ⚠️
ask 43_current_datetime_from_prompt
ask 45_fetch_deployment_logs_simple
ask 51_logs_summarize_errors
ask 53_logs_find_term
ask 54_not_truncated_when_getting_pods
ask 59_label_based_counting
ask 60_count_less_than
ask 61_exact_match_counting
ask 63_fetch_error_logs_no_errors
ask 79_configmap_mount_issue
ask 83_secret_not_found
ask 86_configmap_like_but_secret
ask 93_calling_datadog[0]
ask 93_calling_datadog[1]
ask 93_calling_datadog[2]

Legend

  • ✅ the test was successful
  • :minus: the test was skipped
  • ⚠️ the test failed but is known to be flaky or known to fail
  • 🚧 the test had a setup failure (not a code regression)
  • 🔧 the test failed due to mock data issues (not a code regression)
  • 🚫 the test was throttled by API rate limits/overload
  • ❌ the test failed and should be fixed before merging the PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

AltStyle によって変換されたページ (->オリジナル) /