ImageJAI — AI Assistant for ImageJ/Fiji

A single Fiji plugin that adds a conversational AI assistant to ImageJ. Install by dragging a JAR into plugins/. Free to use with Google Gemini or local with Ollama.

Data Governance

ImageJAI includes a per-folder Privacy Posture for labs using external agent CLIs. New folders default to Pseudonymised; supervisors can set sensitive folders to On-premises, which filters the agent list to local binaries and refuses cloud-hosted Ollama tags.

"ImageJAI applies UK GDPR Art. 4(5) pseudonymisation to all outbound responses by default, differentiates microscopy pixels (downsampled, burn-in masked) from GUI screenshots (refused), refuses cloud-hosted model endpoints in On-premises posture, and emits an append-only audit trail per project."

The audit trail is written to AI_Exports/imagejai_audit.csv, and the launcher can generate a Data Handling Statement PDF for ethics or grant paperwork. See docs/data-governance/README.md for the biologist-facing workflow and limits.

Features

• Natural language control — "Open the blobs image", "Apply a Gaussian blur with sigma 2", "Count all cells"
• Vision reasoning — AI can see your images and suggest appropriate analyses
• Multi-step pipelines — "Analyze all neurons in this confocal stack" generates and executes a complete workflow
• 6 specialist agents — Segmentation, Measurement, Visualization, Statistics, Hypothesis-Driven Analysis, and General
• Parameter exploration — Tries multiple thresholds/parameters, measures quantitatively, recommends the best
• Teaching mode — Watches your manual actions, detects patterns, suggests automation
• Knowledge base — 90+ indexed macro functions and 30+ tips/recipes including neuroscience workflows
• Script generator — Creates installable Groovy/Jython plugins with GUI dialogs on demand
• Live monitoring — Warns about saturated pixels, uncalibrated images, memory pressure
• Hypothesis-driven analysis — State a scientific hypothesis, AI designs the complete analysis plan
• Cross-tool integration — Optional Python/R script execution for advanced statistics
• TCP command server — Optional TCP server (port 7746) for external agent access (CLI agents and scripts). Off by default.
• Plugin argument discovery — Probe any plugin's dialog to learn exact macro syntax before using it
• Dialog interaction — Read, fill, and click any open dialog (buttons, checkboxes, dropdowns, text fields, sliders)
• Progress monitoring — Track progress bar state and status line from external agents
• Groovy/Jython scripting — Run scripts inside Fiji's JVM for full Java API access

Install

1. Download imagej-ai-0.2.0.jar
2. Copy to your Fiji.app/plugins/ directory
3. Restart Fiji
4. Go to Plugins > AI Assistant
5. On first run, choose your AI backend:
   • Google Gemini (free) — get a key at ai.google.dev
   • Ollama (free, local, private) — install from ollama.ai
   • OpenAI / Compatible — any OpenAI-compatible endpoint
6. Start chatting

Requirements

• Fiji (ImageJ2) with Java 8+
• An LLM backend (Gemini API key, Ollama, or OpenAI-compatible endpoint)
• No other dependencies — everything else ships with Fiji

Usage Examples

"Open the sample blobs image"
"Apply a Gaussian blur with sigma 3"
"Segment the nuclei using the best threshold method"
"Measure mean fluorescence intensity per cell"
"Make a max projection of this z-stack"
"Add a 50 micron scale bar"
"I hypothesize that AT8 staining is increased in the SCN of CK1-mutant mice"
"Find the best threshold method for this image"
"Save this pipeline as a macro"
"What do you see in this image?"
"Make a publication-quality figure with scale bar"
"Export the results as CSV"

Build from Source

# Requires Maven 3.6+ and JDK 8+
mvn clean package -q

# Build and deploy to local Fiji
bash build.sh

Versioning

MAJOR.MINOR.PATCH — first digit for new features, second for big refactors/improvements, third for bug fixes.

Architecture

ImageJAIPlugin.java          Entry point (Plugins > AI Assistant)
├── ui/                      Swing chat panel + settings dialog
├── llm/                     LLM backends (Gemini, Ollama, OpenAI)
├── engine/                  Macro execution, state inspection, image capture
│   ├── CommandEngine        Execute macros, capture results
│   ├── StateInspector       Query open images, results, ROIs, memory
│   ├── PipelineBuilder      Multi-step workflow orchestration
│   ├── ExplorationEngine    Parameter optimization (try N methods, compare)
│   ├── ImageMonitor         Background monitoring for issues
│   ├── ScriptGenerator      Generate installable Fiji scripts
│   ├── CrossToolRunner      Run Python/R externally
│   └── TCPCommandServer     Optional TCP server for external agents (port 7746)
├── agents/                  Multi-agent specialist system
│   ├── AgentOrchestrator    Intent classification + routing
│   ├── SegmentationAgent    Thresholding, watershed, StarDist/Cellpose
│   ├── MeasurementAgent     CTCF, colocalization, particle analysis
│   ├── VisualizationAgent   LUTs, projections, montages, figures
│   ├── StatsAgent           Statistical tests, data export
│   └── HypothesisAgent      Hypothesis → analysis plan
├── knowledge/               RAG knowledge base
│   ├── MacroReference       90+ indexed ImageJ macro functions
│   ├── DocIndex             30+ tips, recipes, best practices
│   └── PromptTemplates      System prompts, response parsing
└── config/                  Settings persistence (~/.imagej-ai/)

How It Works

1. You type a message in the chat panel
2. The AgentOrchestrator classifies your intent and routes to the right specialist
3. The specialist builds a prompt with your message + current ImageJ state + relevant knowledge
4. The LLM generates a response with <macro> or <pipeline> blocks
5. The CommandEngine executes the macros on ImageJ's EDT thread
6. Results (new images, measurements, errors) are captured and fed back
7. If a macro fails, the AI self-corrects (up to 3 retries)
8. The response and any results are displayed in the chat

TCP Command Server (Advanced)

For power users who want to control ImageJ from CLI agents or custom scripts:

1. Open Settings > check "Enable TCP command server"
2. Default port: 7746 (configurable)
3. Send JSON commands over TCP:

# Test connection
echo '{"command": "ping"}' | nc localhost 7746

# Execute a macro
echo '{"command": "execute_macro", "code": "run(\"Blobs (25K)\");"}' | nc localhost 7746

# Get current state
echo '{"command": "get_state"}' | nc localhost 7746

# Capture image as base64 PNG
echo '{"command": "capture_image"}' | nc localhost 7746

# Probe a plugin's parameters
echo '{"command": "probe_command", "plugin": "Gaussian Blur..."}' | nc localhost 7746

# Check progress bar
echo '{"command": "get_progress"}' | nc localhost 7746

Available commands: ping, execute_macro, get_state, get_image_info, get_results_table, capture_image, run_pipeline, explore_thresholds, get_state_context, batch, get_log, get_histogram, get_open_windows, get_metadata, get_pixels, 3d_viewer, get_dialogs, close_dialogs, probe_command, run_script, interact_dialog, get_progress

The run_script command executes Groovy/Jython/JavaScript code directly inside Fiji's JVM — enabling access to any Java API, Swing component manipulation, and plugin internals that macros can't reach.

The probe_command command opens a plugin's dialog, reads every field (numeric, string, checkbox, dropdown with all options), derives macro argument keys, generates example macro syntax, and cancels without executing.

The interact_dialog command reads and interacts with any open dialog — list components, click buttons, set checkboxes, fill text fields, select dropdowns, adjust sliders.

The get_progress command reads the Fiji progress bar state and status line text — useful for monitoring long-running operations from external agents.

This is completely optional — the plugin works fully without it.

Agent Directory

The agent/ directory contains a complete AI agent toolkit for controlling ImageJ via the TCP server:

• ij.py — Python CLI helper for all TCP commands (macro, capture, state, script, probe, UI interaction, progress, etc.)
• pixels.py — Python-side pixel analysis (stats, cell detection, line profiles)
• scan_plugins.py — Discover all installed Fiji commands and update sites
• probe_plugin.py — Probe plugin dialogs for parameters, cache results, batch-probe
• adviser.py — Research-only analysis consultant (no TCP needed)
• auditor.py — Validate measurement results for sanity
• practice.py — Autonomous self-improvement on sample images
• train_agent.py — Train the agent on a lab's specific images
• recipes/ — YAML analysis recipes (colocalization, cell counting, CTCF, 3D rendering, etc.)
• references/ — 50+ expert reference documents covering microscopy, analysis methods, plugins, statistics, and neuroscience workflows

Context Hook (Claude Code Integration)

When using Claude Code in this project directory, a context hook (context_hook.py) automatically injects live Fiji state into the conversation via the TCP server (port 7746).

Session start (once):

• Fiji connection status
• Full list of installed Fiji commands/plugins
• Available reference documents (agent/references/)

Every message (dynamic):

• Open images — titles, dimensions, bit depth, stack info, calibration, ROI
• Results table — row count and column names
• JVM memory — used/max/free + open image count
• Progress bar — active state, percent complete, status line text
• Open dialogs — errors, warnings, prompts with text and buttons
• IJ Log — last 10 lines

Requires the TCP command server to be enabled in Fiji (Settings > Advanced > "Enable TCP command server"). Gracefully degrades when Fiji is not running.

Citing ImageJAI

If you use ImageJAI in published work, please cite it. Citation metadata is in CITATION.cff (use GitHub's "Cite this repository" button). A Zenodo DOI will be added here once the first tagged release is archived.

When you use ImageJAI to invoke specific tools, also cite the underlying methods (StarDist, Cellpose, TrackMate, etc.) as you would when using them directly.

License

BSD 3-Clause License. See LICENSE for the full text.

(Earlier ImageJAI versions shipped under BSD-2-Clause. Versions from the next tagged release onwards ship under BSD-3-Clause.)

Acknowledgements

Developed by Jamie Malcolm in the Brancaccio Lab at the UK Dementia Research Institute, Imperial College London.

This work was supported by the UK Dementia Research Institute, which receives its core funding from the UK Medical Research Council, the Alzheimer's Society, and Alzheimer's Research UK.

Built on the Fiji / ImageJ ecosystem and on third-party language-model providers (Google Gemini, OpenAI, Ollama). Users running ImageJAI with cloud LLM backends should be aware that image content and prompts are transmitted to the chosen provider; see the provider's privacy policy.

Note: this plugin contains AI components that interact with third-party LLM services. Before institutional or commercial use, confirm your data-handling requirements with your local information-governance contact.