feat(mdm): analyze for brew & python packages#19
feat(mdm): analyze for brew & python packages#19shubham-stepsecurity wants to merge 1 commit intostep-security:mainfrom
Conversation
46bed04 to
143218c
Compare
143218c to
84795ab
Compare
There was a problem hiding this comment.
Pull request overview
Adds Homebrew and Python inventory collection to Dev Machine Guard, expanding both community-mode output (pretty/HTML/JSON) and enterprise telemetry payloads to include package managers, installed packages, and project-level package data.
Changes:
- Extend community scan results and renderers to include Homebrew formulae/casks and Python package managers/packages/projects.
- Extend enterprise telemetry payload schema to include Homebrew/Python scan data and new performance metric fields.
- Add new detectors/scanners for Homebrew + Python, plus tighten MCP enterprise content filtering and add related tests.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/telemetry/telemetry.go | Adds brew/python fields to enterprise telemetry payload and runs brew/python scans. |
| internal/scan/scanner.go | Adds brew/python scanning in community mode and includes results in ScanResult/Summary. |
| internal/output/pretty.go | Prints brew/python summary and detailed sections (brew, python, node projects). |
| internal/output/html.go | Expands HTML report to show node projects, brew, python; adds collapsible sections. |
| internal/model/model.go | Extends ScanResult/Summary; introduces shared project/package models and new telemetry scan result types. |
| internal/detector/brew.go | Detects Homebrew installation and parses installed formulae/casks. |
| internal/detector/brewscan.go | Enterprise-style brew scanning producing base64-encoded raw output. |
| internal/detector/brew_test.go | Adds unit tests for brew detection/listing/scanning. |
| internal/detector/pythonpm.go | Detects Python package managers; parses pip JSON package listing. |
| internal/detector/pythonscan.go | Enterprise-style python scanning producing base64-encoded raw output. |
| internal/detector/pythonproject.go | Detects python projects with venvs and enumerates venv packages. |
| internal/detector/pythonpm_test.go | Adds unit tests for python PM detection/version parsing and python project detection. |
| internal/detector/nodeproject.go | Changes node project detection to return project+deps details (not just counts). |
| internal/detector/mcp.go | Prevents leaking secrets by omitting config content when filtering fails. |
| internal/detector/mcp_test.go | Adds tests verifying enterprise MCP config filtering and non-JSON/invalid handling. |
| internal/config/config.go | Persists and configures enable/disable flags for brew/python scans. |
| internal/cli/cli.go | Adds CLI flags for enabling/disabling brew/python scans and updates help text. |
| cmd/stepsecurity-dev-machine-guard/main.go | Wires config file values into CLI config for brew/python flags. |
| examples/sample-output.json | Updates example output schema to include node projects, brew, and python sections. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| log.Progress("Detecting Homebrew...") | ||
| brewDetector := detector.NewBrewDetector(exec) | ||
| brewPkgMgr = brewDetector.DetectBrew(ctx) | ||
| if brewPkgMgr != nil { | ||
| log.Progress(" Found: Homebrew v%s at %s", brewPkgMgr.Version, brewPkgMgr.Path) | ||
| brewScanner := detector.NewBrewScanner(exec, log) | ||
| if r, ok := brewScanner.ScanFormulae(ctx); ok { |
There was a problem hiding this comment.
When running under LaunchDaemon/root, using exec.LookPath("brew") (and later running brew commands) will use root's PATH, which often does not include /opt/homebrew/bin. This can cause Homebrew to be incorrectly reported as “not found” even though it's installed for the logged-in user. Consider delegating brew detection/commands via exec.RunAsUser(loggedInUsername, "which brew" / "brew ...") similarly to NodeScanner, or fallback-check common brew locations (e.g., /opt/homebrew/bin/brew, /usr/local/bin/brew).
| ExtensionsCount int `json:"extensions_count"` | ||
| NodePackagesScanMs int64 `json:"node_packages_scan_ms"` | ||
| NodeGlobalPkgsCount int `json:"node_global_packages_count"` | ||
| NodeProjectsCount int `json:"node_projects_count"` | ||
| BrewFormulaeCount int `json:"brew_formulae_count"` | ||
| BrewCasksCount int `json:"brew_casks_count"` | ||
| PythonGlobalPkgsCount int `json:"python_global_packages_count"` | ||
| PythonProjectsCount int `json:"python_projects_count"` |
There was a problem hiding this comment.
These new PerformanceMetrics fields (brew/python counts) are introduced here, but the payload construction later in Run() still only sets the existing Node/extension metrics. As a result, the new brew/python metrics will remain zero in telemetry uploads. Either wire them into the metrics assignment or remove them until they're populated.
| ExtensionsCount int `json:"extensions_count"` | |
| NodePackagesScanMs int64 `json:"node_packages_scan_ms"` | |
| NodeGlobalPkgsCount int `json:"node_global_packages_count"` | |
| NodeProjectsCount int `json:"node_projects_count"` | |
| BrewFormulaeCount int `json:"brew_formulae_count"` | |
| BrewCasksCount int `json:"brew_casks_count"` | |
| PythonGlobalPkgsCount int `json:"python_global_packages_count"` | |
| PythonProjectsCount int `json:"python_projects_count"` | |
| ExtensionsCount int `json:"extensions_count"` | |
| NodePackagesScanMs int64 `json:"node_packages_scan_ms"` | |
| NodeGlobalPkgsCount int `json:"node_global_packages_count"` | |
| NodeProjectsCount int `json:"node_projects_count"` |
| func (d *PythonProjectDetector) listInDir(dir string) []model.ProjectInfo { | ||
| ctx := context.Background() | ||
| seen := make(map[string]bool) | ||
| var projects []model.ProjectInfo | ||
| _ = filepath.WalkDir(dir, func(path string, entry os.DirEntry, err error) error { |
There was a problem hiding this comment.
listInDir() creates a context.Background() and then can execute up to maxPythonProjects (1000) separate pip list commands (15s timeout each). In the worst case this can turn a scan into hours of subprocess work and it also ignores cancellation/deadlines from the caller. Consider threading the caller ctx through ListProjects/listInDir, and adding tighter limits (e.g., cap the number of venvs for which packages are enumerated, or make package enumeration optional/summary-only).
| pkgs := make([]model.PackageDetail, 0, total) | ||
| for name, version := range pkg.Dependencies { | ||
| pkgs = append(pkgs, model.PackageDetail{Name: name, Version: version}) | ||
| } | ||
| for name, version := range pkg.DevDependencies { | ||
| pkgs = append(pkgs, model.PackageDetail{Name: name, Version: version}) | ||
| } | ||
| return pkgs |
There was a problem hiding this comment.
readPackageJSONDeps() iterates over map keys (dependencies/devDependencies), which makes package ordering nondeterministic in Go. Since this feeds Pretty/HTML output and JSON, the same project can produce different package order across runs. Consider sorting the resulting []PackageDetail (e.g., by Name) before returning to ensure stable output.
| // ListProjects returns Node.js project paths with their detected package manager | ||
| // and the dependencies listed in package.json. | ||
| func (d *NodeProjectDetector) ListProjects(searchDirs []string) []model.ProjectInfo { | ||
| var projects []model.ProjectInfo | ||
| for _, dir := range searchDirs { | ||
| count += d.countInDir(dir) | ||
| if count >= maxNodeProjects { | ||
| return maxNodeProjects | ||
| projects = append(projects, d.listInDir(dir)...) | ||
| if len(projects) >= maxNodeProjects { | ||
| return projects[:maxNodeProjects] | ||
| } | ||
| } | ||
| return count | ||
| return projects | ||
| } |
There was a problem hiding this comment.
NodeProjectDetector now has new behavior (filtering by node_modules / Yarn PnP, and parsing dependencies from package.json), but there are no unit tests covering it. Given other detectors in this package are tested, please add a nodeproject_test.go covering at least: (1) detection of a project with node_modules, (2) Yarn Berry PnP detection via .pnp.cjs, and (3) dependency parsing from package.json.
| // ScanGlobalPackages runs pip3/conda/uv list and returns raw base64-encoded results. | ||
| func (s *PythonScanner) ScanGlobalPackages(ctx context.Context) []model.PythonScanResult { | ||
| var results []model.PythonScanResult | ||
|
|
||
| for _, spec := range pythonScanSpecs { | ||
| binPath, err := s.exec.LookPath(spec.binary) | ||
| if err != nil { | ||
| continue | ||
| } | ||
|
|
||
| s.log.Progress(" Checking %s global packages...", spec.name) | ||
| version := s.getVersion(ctx, spec.binary, spec.versionCmd) | ||
|
|
||
| start := time.Now() | ||
| args := spec.listArgs | ||
| stdout, stderr, exitCode, _ := s.exec.RunWithTimeout(ctx, 60*time.Second, spec.binary, args...) | ||
| duration := time.Since(start).Milliseconds() | ||
|
|
||
| errMsg := "" | ||
| if exitCode != 0 { | ||
| errMsg = spec.binary + " list command failed" | ||
| } | ||
|
|
||
| results = append(results, model.PythonScanResult{ | ||
| PackageManager: spec.name, | ||
| PMVersion: version, | ||
| BinaryPath: binPath, | ||
| RawStdoutBase64: base64.StdEncoding.EncodeToString([]byte(stdout)), | ||
| RawStderrBase64: base64.StdEncoding.EncodeToString([]byte(stderr)), | ||
| Error: errMsg, | ||
| ExitCode: exitCode, | ||
| ScanDurationMs: duration, | ||
| }) | ||
| } | ||
|
|
||
| return results | ||
| } |
There was a problem hiding this comment.
PythonScanner (enterprise-mode raw/base64 output) is new but has no unit tests. Please add tests using executor.NewMock to cover: tool not found (skipped), successful scan capturing stdout/stderr/exitCode, and timeout/non-zero exitCode behavior so telemetry parsing stays stable.
| <div class="section"> | ||
| <h2>AI Agents and Tools <span class="count">{{.Summary.AIAgentsAndToolsCount}}</span></h2> | ||
| <div class="section-header" onclick="toggleSection(this)"> | ||
| <h2>AI Agents and Tools <span class="count">{{.Summary.AIAgentsAndToolsCount}}</span></h2> | ||
| <span class="toggle">▼</span> | ||
| </div> | ||
| <div class="section-body"> | ||
| <table> |
There was a problem hiding this comment.
The new collapsible section headers are clickable
What does this PR do?
Type of change
Testing
./stepsecurity-dev-machine-guard --verbose./stepsecurity-dev-machine-guard --json | python3 -m json.toolmake lintmake testRelated Issues