Update Duplicate Detection by Lightning11wins · Pull Request #77 · LightSys/centrallix

Lightning11wins · 2025-11-14T19:02:39Z

The duplicate detection project is ready to review, although (best case), there are still a couple of things blocking it from being ready to merge.

I would appreciate a full review of all changes, as there's quite a lot here. That said, some areas may require additional special attention, so I've compiled a list of all 28 TODO: Greg comments below. (Note: Some of my todos assume the reader understands various pieces of nearby context / has generally read the indicated source code.)

3 TODO: Gregs in objdrv_cluster.c
1 TODO: Greg in mtsession.md.
1 TODO: Greg in xarray.md.
1 TODO: Greg in xstring.md.

Please let me know if you have any questions, comments, or concerns about my changes and design choices.

GitHub Relationships

Depends on Upgrade centrallix-lib Tests #170
- Depends on Add centrallix-lib Utilities #130
Depends on Add centrallix-lib Utilities #130

Lightning11wins · 2025-11-14T19:04:45Z

Kardia PR.

gbeeley

First pass here, covering all of the documentation and other parts of this PR except the clusters.c and objdrv_cluster.c files. Thanks!

gbeeley · 2025-12-04T20:51:37Z

Ok, this is odd. Github crashed when I submitted this PR review. This initial review is for everything except the core of this PR (clusters.c and objdrv_cluster.c). So I'm not sure Github submitted this correctly.

Lightning11wins · 2025-12-12T00:41:08Z

FYI: I just cleaned up my todos for you, but I also updated and reorganized the lists in each PR so that they only include todos in their respective branches.

Lightning11wins · 2025-12-12T17:58:00Z

~~FYI: I am aware that most of my functions are missing the required final return; statement. I'll get to that next time I'm on this branch.~~

Just pushed a commit to fix this.

Lightning11wins · 2026-02-21T01:37:50Z

@Lightning11wins Review this PR to look for usages of DateTime that assume the Value property represents "seconds since the Epoch". This assumption is false.

Resolved

Lightning11wins · 2026-03-16T17:57:31Z

PR cleared for human review.

nboard

Overall, excellent work! I can tell you put a lot of care into making sure the code was readable and well documented.

All of requested changes are open to discussion, so please reach out to me with any questions.

nboard · 2026-01-30T19:27:52Z

+/* Centrallix Application Server System					*/
+/* Centrallix Core							*/
+/* 									*/
+/* Copyright (C) 1998-2012 LightSys Technology Services, Inc.		*/


I believe this is meant to be through the year the file was last changed (though it seems we generally do a bad job of updating it). I would at least put it through the date it was created, though.
To avoid burying you in (even more) comments I'll just mention it here.

Ahh, I think we later came to the conclusion that all licenses in modified files should be updated, but I forgot to apply that to this project.

I've applied this change to all modified files on this branch. Done.

It looks like xarray and mtsession both had a decent number of changes but never had their copyright dates updated. Was there a reason you did not want to update the dates on those?

I forgot, updated.

nboard · 2026-03-17T16:50:39Z

+	if (source_data->Strings == NULL) goto err_free;
+	data_xarray.nAlloc = 0;
+
+	/** Trim and store vectors. **/


If the second or third call to ci_xaToTrimmedArray fails, then source_data has either keys or keys and strings which never get freed, and which get ignored by the calling function on failure, and thus would be leaked.

Ah, yup, I think I see that. It should be fixed now.

It looks like this is still a problem, since the only place the contents of the strings in the arrays are freed requires the xArrays to exist. That is, if an error occurs after one or more xArray is freed, then the list itself stored in source_data is freed, but not the strings stored by the list

I added a proper cleanup goto so the memory leaks should be fixed now.

Lightning11wins · 2026-03-23T13:43:16Z

@greptileai Review this PR please. :)

greptile-apps · 2026-03-23T13:49:30Z

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains from the prior review threads.

Summary

The PR introduces duplicate-detection support built around string similarity, clustering utilities, and a new cluster object driver, while also expanding tests and shared-library helpers.

Adds cosine, Levenshtein, metaphone, sparse-vector, search, and k-means functionality.
Adds cluster schema, configuration, object-driver integration, datasets, and regression coverage.
Updates shared memory, collection, string, parser, build, and test infrastructure used by the feature.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    Input[Source object data] --> Driver[Cluster object driver]
    Driver --> Normalize[String/vector preparation]
    Normalize --> Similarity[Cosine or Levenshtein similarity]
    Similarity --> Search[Pair search]
    Similarity --> KMeans[K-means clustering]
    Search --> Results[Duplicate/query results]
    KMeans --> Results

_{Reviews (15) · Last reviewed commit: "Fix mistakes and typos."}

Lightning11wins · 2026-03-23T23:57:56Z

@greptileai Please re-review.

Lightning11wins · 2026-03-24T14:06:40Z

@greptileai Please re-review.

… manually in case new fields are added.

…ontext (which could be intentional).

… 1024.0 KiB, which should be 1.0 MiB.

…pointers, which confused the compiler.

# Conflicts: # centrallix-lib/tests/test_utils.h

# Conflicts: # centrallix-lib/tests/test_strtcpy_13.c

Wow, how many more of these are there!?

Lightning11wins assigned gbeeley and Lightning11wins Nov 14, 2025

Lightning11wins requested a review from gbeeley November 14, 2025 19:03

Lightning11wins added duplicate enhancement and removed duplicate labels Nov 14, 2025

Lightning11wins mentioned this pull request Nov 14, 2025

Update Duplicate Detection LightSys/kardia#57

Open

gbeeley previously requested changes Dec 4, 2025

View reviewed changes

Lightning11wins mentioned this pull request Dec 11, 2025

Implement Schema Verification for Expression Functions #78

Closed

Lightning11wins requested a review from gbeeley December 12, 2025 00:31

Lightning11wins added the testing Includes testing, either new tests or updates to existing tests. label Jan 9, 2026

nboard self-assigned this Feb 2, 2026

This was referenced Feb 25, 2026

Implement Responsive Client-Side Autoscaling (DRAFT) #84

Closed

Implement Responsive Client-Side Autoscaling (OLD) #85

Closed

Implement Responsive Client-Side Autoscaling #90

Open

Lightning11wins mentioned this pull request Mar 10, 2026

Apos autoscale9 slim5 greptile #91

Closed

nboard requested changes Mar 17, 2026

View reviewed changes

Lightning11wins added the ai-review Request AI review for PRs. label Mar 23, 2026

greptile-apps Bot reviewed Mar 23, 2026

View reviewed changes

Comment thread centrallix/osdrivers/objdrv_cluster.c

Comment thread centrallix/utility/double_metaphone.c Outdated

greptile-apps Bot reviewed Mar 24, 2026

View reviewed changes

Comment thread centrallix-lib/src/clusters.c Outdated

Lightning11wins added 6 commits September 23, 2026 16:06

Add a memset() for ClusterQuery even though each field is initialized…

48bbede

… manually in case new fields are added.

Fix a confusing error message.

ff9b869

Improve some wording in string_similarity.md.

a9942a0

Improve spacing in test_expfn_log_00.to.

69c0c73

Fix more typos and mistakes.

53d75d4

Fix mistakes in comments.

fe6bb37

greptile-apps Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread centrallix/osdrivers/objdrv_cluster.c

Lightning11wins added 23 commits September 23, 2026 16:38

Remove warning noise when running mssClearError() outside a session c…

1e8fe5d

…ontext (which could be intentional).

Fix a bug (with tests) that could snprintBytes() to print values like…

5abbf5a

… 1024.0 KiB, which should be 1.0 MiB.

Merge branch 'add-utils' into add-lib-tests

2503820

Add tests for the snprintBytes() bug that was recently fixed.

e0fa24f

Fix obvious mistakes and typos.

ecabce3

Merge branch 'add-lib-tests' into dups

c0491db

Document the possible memory leak if excessive recursion occurs.

3930af0

Revert changes that are not in scope for this work.

d303e42

Fix more typos and mistakes.

ec6b61d

Merge branch 'master' into add-lib-tests

dbbcab0

Improve readability.

8c1b59f

Remove an assert function that should have stayed on the dups branch.

be3fae9

Update test asserts to use bools.

4833d29

Improve performance of testing functions.

51aff7d

Fix ASSERT_STR_EQL() and ASSERT_STR_EQL_N() comparing heap and stack …

f7f3b48

…pointers, which confused the compiler.

Merge branch 'add-lib-tests' into dups

0126ea6

# Conflicts: # centrallix-lib/tests/test_utils.h

Clean up a comments.

2ee5416

Merge branch 'add-utils' into add-lib-tests

44c5bb8

# Conflicts: # centrallix-lib/tests/test_strtcpy_13.c

Fix mistakes and typos.

2452ff9

Merge branch 'add-lib-tests' into dups

40ca017

Fix mistakes and typos.

fef1f58

Fix mistakes and typos.

670e2af

Fix mistakes and typos.

24ca1e0

Wow, how many more of these are there!?

Conversation

Lightning11wins commented Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Relationships

Uh oh!

Lightning11wins commented Nov 14, 2025

Uh oh!

gbeeley left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gbeeley commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Lightning11wins commented Dec 12, 2025

Uh oh!

Lightning11wins commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Lightning11wins commented Feb 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Lightning11wins commented Mar 16, 2026

Uh oh!

nboard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Lightning11wins commented Mar 23, 2026

Uh oh!

greptile-apps Bot commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

Uh oh!

Uh oh!

Lightning11wins commented Mar 23, 2026

Uh oh!

Uh oh!

Lightning11wins commented Mar 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Lightning11wins commented Nov 14, 2025 •

edited

Loading

gbeeley commented Dec 4, 2025 •

edited

Loading

Lightning11wins commented Dec 12, 2025 •

edited

Loading

Lightning11wins commented Feb 21, 2026 •

edited

Loading

greptile-apps Bot commented Mar 23, 2026 •

edited

Loading