Skip to content

Speed up busy forms, the export and submit lookups - #217

Merged
PhilReinking merged 3 commits into
mainfrom
perf/bits-159-busy-forms
Oct 10, 2026
Merged

PhilReinking merged 3 commits into
mainfrom
perf/bits-159-busy-forms

Conversation

@PhilReinking

@PhilReinking PhilReinking commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Form statistics now come from SQL counts, the CSV export writes its rows 500 submissions at a time, and one migration indexes the submit and upload lookups.

Fixes BITS-159

Heads-up: export columns follow the form's question order and always list every question, also on a form without submissions. Answers that start with = + - @, tab or CR get a leading '. Plain numbers and phone numbers like -5 or +49 30 1234567 stay as they are.

  • 10,000 submissions: the export adds about 30 MB (main: 995 MB). The test allows 64 MB on top of a request's 29 MB.
  • MySQL already indexes form_session_uploads.form_session_response_id through its foreign key, so the migration adds that index only where it's missing (SQLite).
  • MySQL then uses the two new indexes for its foreign keys, so down() first gives those keys their own index back.
  • The statistics and responses_count keep their JSON keys and values.

Tested: full Pest suite on MariaDB, new tests on SQLite too; the 6 criterion tests fail on main. Migration up, down, up on both. 10,000-row test passes under memory_limit=128M.
Not tested: a download in the browser, MySQL 8 (only MariaDB 10).
After merge: nothing, the migration runs on container start.

🤖 Generated with Claude Code

PhilReinking and others added 2 commits October 10, 2026 13:33
…okups

- Form JSON: total, completed and rate come from SQL counts, cached per model
- Blocks page: answers per option via withCount, session count as a plain count
- Export: header from the form's questions, rows written per 500 sessions with
  eager loads, cells starting with = + - @ tab or CR prefixed with '
- Migration: indexes on form_sessions(form_id, token),
  form_session_responses(form_session_id, form_block_id,
  form_block_interaction_id) and form_session_uploads(form_session_response_id)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@PhilReinking PhilReinking left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK at 45ffc64, CI green.

Checked beyond the PR text: on the same seeded data, main and this PR give the same statistics and per-question counts, also for a trashed form, open sessions and a session completed without answers. The dashboard, Summary and a real CSV download work in the browser, with emoji, quotes and line breaks intact. The migration runs up, down and up on MariaDB and SQLite. One choice on number answers, see inline.

// spreadsheet apps run a cell that starts with one of these as a formula
private function escapeFormula($value)
{
return is_string($value) && preg_match('/^[=+\-@\t\r]/', $value) ? "'".$value : $value;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plain numbers get the ' too: -5 becomes '-5 and a phone answer +49 30 1234567 becomes '+49 30 1234567. A spreadsheet then reads them as text, so a number column no longer sorts or sums. A cell made only of a sign, digits, spaces, dots and commas can't run anything, so I'd leave those alone:

return is_string($value) && preg_match('/^[=+\-@\t\r]/', $value) && ! preg_match('/^[+-]?[\d .,]+$/', $value) ? "'".$value : $value;

I tried it: -1+1, =1+1, @SUM(1) and -2+3+cmd|... still get the prefix. The prefix test would need values like +1+1 and -1+1. This goes beyond the ticket's wording, so it's a product decision.

A cell made only of a sign, digits, spaces, dots and commas can't run
anything, and with a leading ' a spreadsheet no longer sorts or sums it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@PhilReinking PhilReinking left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK at 9cce835, CI green.

The number exception holds on odd input too: whitespace, + , -. and -0 stay as they are, while a lone + or -, a tab or CR after the number, -5e3, -5% and +1-1 still get the prefix. The new test fails on 45ffc64 only for -5 and the phone number.

@PhilReinking
PhilReinking merged commit e4c184c into main Oct 10, 2026
4 checks passed
@PhilReinking
PhilReinking deleted the perf/bits-159-busy-forms branch October 10, 2026 12:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant