Add pg_clickhouse

[theory]

It dupes the clickhouse benchmark config to load data (except that column names are lowercase) and the Postgres config to run the queries. All queries push down to ClickHouse without error.

Replaces #739 by eliminating a wayward | head -n1 and makes the column names in ClickHouse lowercase.

[theory]

Output in my test:

SELECT COUNT(*) FROM hits;
Time: 6.359 ms
Time: 1.031 ms
Time: 0.927 ms
SELECT COUNT(*) FROM hits WHERE AdvEngineID <> 0;
Time: 7.311 ms
Time: 2.159 ms
Time: 1.711 ms
SELECT SUM(AdvEngineID), COUNT(*), AVG(ResolutionWidth) FROM hits;
Time: 8.554 ms
Time: 3.772 ms
Time: 3.413 ms
SELECT AVG(UserID) FROM hits;
Time: 8.894 ms
Time: 7.014 ms
Time: 4.308 ms
SELECT COUNT(DISTINCT UserID) FROM hits;
Time: 5.971 ms
Time: 3.423 ms
Time: 2.587 ms
SELECT COUNT(DISTINCT SearchPhrase) FROM hits;
Time: 7.407 ms
Time: 2.470 ms
Time: 2.089 ms
SELECT MIN(EventDate), MAX(EventDate) FROM hits;
Time: 10.218 ms
Time: 4.610 ms
Time: 4.870 ms
SELECT AdvEngineID, COUNT(*) FROM hits WHERE AdvEngineID <> 0 GROUP BY AdvEngineID ORDER BY COUNT(*) DESC;
Time: 4.711 ms
Time: 1.768 ms
Time: 2.433 ms
SELECT RegionID, COUNT(DISTINCT UserID) AS u FROM hits GROUP BY RegionID ORDER BY u DESC LIMIT 10;
Time: 6.917 ms
Time: 2.965 ms
Time: 3.116 ms
SELECT RegionID, SUM(AdvEngineID), COUNT(*) AS c, AVG(ResolutionWidth), COUNT(DISTINCT UserID) FROM hits GROUP BY RegionID ORDER BY c DESC LIMIT 10;
Time: 21.958 ms
Time: 4.241 ms
Time: 3.989 ms
SELECT MobilePhoneModel, COUNT(DISTINCT UserID) AS u FROM hits WHERE MobilePhoneModel <> '' GROUP BY MobilePhoneModel ORDER BY u DESC LIMIT 10;
Time: 6.864 ms
Time: 2.405 ms
Time: 2.021 ms
SELECT MobilePhone, MobilePhoneModel, COUNT(DISTINCT UserID) AS u FROM hits WHERE MobilePhoneModel <> '' GROUP BY MobilePhone, MobilePhoneModel ORDER BY u DESC LIMIT 10;
Time: 4.708 ms
Time: 2.197 ms
Time: 2.204 ms
SELECT SearchPhrase, COUNT(*) AS c FROM hits WHERE SearchPhrase <> '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;
Time: 5.586 ms
Time: 2.510 ms
Time: 2.261 ms
SELECT SearchPhrase, COUNT(DISTINCT UserID) AS u FROM hits WHERE SearchPhrase <> '' GROUP BY SearchPhrase ORDER BY u DESC LIMIT 10;
Time: 4.628 ms
Time: 2.099 ms
Time: 2.113 ms
SELECT SearchEngineID, SearchPhrase, COUNT(*) AS c FROM hits WHERE SearchPhrase <> '' GROUP BY SearchEngineID, SearchPhrase ORDER BY c DESC LIMIT 10;
Time: 5.060 ms
Time: 2.199 ms
Time: 2.220 ms
SELECT UserID, COUNT(*) FROM hits GROUP BY UserID ORDER BY COUNT(*) DESC LIMIT 10;
Time: 5.957 ms
Time: 2.955 ms
Time: 2.669 ms
SELECT UserID, SearchPhrase, COUNT(*) FROM hits GROUP BY UserID, SearchPhrase ORDER BY COUNT(*) DESC LIMIT 10;
Time: 12.301 ms
Time: 6.051 ms
Time: 7.826 ms
SELECT UserID, SearchPhrase, COUNT(*) FROM hits GROUP BY UserID, SearchPhrase LIMIT 10;
Time: 7.472 ms
Time: 8.860 ms
Time: 11.859 ms
SELECT UserID, extract(minute FROM EventTime) AS m, SearchPhrase, COUNT(*) FROM hits GROUP BY UserID, m, SearchPhrase ORDER BY COUNT(*) DESC LIMIT 10;
SELECT UserID, extract(minute FROM EventTime) AS m, SearchPhrase, COUNT(*) FROM hits GROUP BY UserID, m, SearchPhrase ORDER BY COUNT(*) DESC LIMIT 10;
Time: 8.453 ms
SELECT UserID, extract(minute FROM EventTime) AS m, SearchPhrase, COUNT(*) FROM hits GROUP BY UserID, m, SearchPhrase ORDER BY COUNT(*) DESC LIMIT 10;
Time: 6.062 ms
SELECT UserID, extract(minute FROM EventTime) AS m, SearchPhrase, COUNT(*) FROM hits GROUP BY UserID, m, SearchPhrase ORDER BY COUNT(*) DESC LIMIT 10;
Time: 6.511 ms
SELECT UserID FROM hits WHERE UserID = 435090932899640449;
Time: 4.108 ms
Time: 1.361 ms
Time: 1.219 ms
SELECT COUNT(*) FROM hits WHERE URL LIKE '%google%';
Time: 6.865 ms
Time: 2.345 ms
Time: 2.053 ms
SELECT SearchPhrase, MIN(URL), COUNT(*) AS c FROM hits WHERE URL LIKE '%google%' AND SearchPhrase <> '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;
Time: 6.156 ms
Time: 2.535 ms
Time: 2.064 ms
SELECT SearchPhrase, MIN(URL), MIN(Title), COUNT(*) AS c, COUNT(DISTINCT UserID) FROM hits WHERE Title LIKE '%Google%' AND URL NOT LIKE '%.google.%' AND SearchPhrase <> '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10;
Time: 7.954 ms
Time: 2.471 ms
Time: 2.330 ms
SELECT * FROM hits WHERE URL LIKE '%google%' ORDER BY EventTime LIMIT 10;
SELECT * FROM hits WHERE URL LIKE '%google%' ORDER BY EventTime LIMIT 10;
Time: 6.646 ms
SELECT * FROM hits WHERE URL LIKE '%google%' ORDER BY EventTime LIMIT 10;
Time: 4.062 ms
SELECT * FROM hits WHERE URL LIKE '%google%' ORDER BY EventTime LIMIT 10;
Time: 4.080 ms
SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY EventTime LIMIT 10;
SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY EventTime LIMIT 10;
Time: 3.481 ms
SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY EventTime LIMIT 10;
Time: 1.017 ms
SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY EventTime LIMIT 10;
Time: 0.954 ms
SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY SearchPhrase LIMIT 10;
Time: 3.899 ms
Time: 1.819 ms
Time: 1.326 ms
SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY EventTime, SearchPhrase LIMIT 10;
SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY EventTime, SearchPhrase LIMIT 10;
Time: 3.769 ms
SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY EventTime, SearchPhrase LIMIT 10;
Time: 1.175 ms
SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY EventTime, SearchPhrase LIMIT 10;
Time: 1.163 ms
SELECT CounterID, AVG(length(URL)) AS l, COUNT(*) AS c FROM hits WHERE URL <> '' GROUP BY CounterID HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
Time: 6.452 ms
Time: 2.718 ms
Time: 2.502 ms
SELECT REGEXP_REPLACE(Referer, '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(length(Referer)) AS l, COUNT(*) AS c, MIN(Referer) FROM hits WHERE Referer <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;
Time: 15.984 ms
Time: 2.735 ms
Time: 2.504 ms
SELECT SUM(ResolutionWidth), SUM(ResolutionWidth + 1), SUM(ResolutionWidth + 2), SUM(ResolutionWidth + 3), SUM(ResolutionWidth + 4), SUM(ResolutionWidth + 5), SUM(ResolutionWidth + 6), SUM(ResolutionWidth + 7), SUM(ResolutionWidth + 8), SUM(ResolutionWidth + 9), SUM(ResolutionWidth + 10), SUM(ResolutionWidth + 11), SUM(ResolutionWidth + 12), SUM(ResolutionWidth + 13), SUM(ResolutionWidth + 14), SUM(ResolutionWidth + 15), SUM(ResolutionWidth + 16), SUM(ResolutionWidth + 17), SUM(ResolutionWidth + 18), SUM(ResolutionWidth + 19), SUM(ResolutionWidth + 20), SUM(ResolutionWidth + 21), SUM(ResolutionWidth + 22), SUM(ResolutionWidth + 23), SUM(ResolutionWidth + 24), SUM(ResolutionWidth + 25), SUM(ResolutionWidth + 26), SUM(ResolutionWidth + 27), SUM(ResolutionWidth + 28), SUM(ResolutionWidth + 29), SUM(ResolutionWidth + 30), SUM(ResolutionWidth + 31), SUM(ResolutionWidth + 32), SUM(ResolutionWidth + 33), SUM(ResolutionWidth + 34), SUM(ResolutionWidth + 35), SUM(ResolutionWidth + 36), SUM(ResolutionWidth + 37), SUM(ResolutionWidth + 38), SUM(ResolutionWidth + 39), SUM(ResolutionWidth + 40), SUM(ResolutionWidth + 41), SUM(ResolutionWidth + 42), SUM(ResolutionWidth + 43), SUM(ResolutionWidth + 44), SUM(ResolutionWidth + 45), SUM(ResolutionWidth + 46), SUM(ResolutionWidth + 47), SUM(ResolutionWidth + 48), SUM(ResolutionWidth + 49), SUM(ResolutionWidth + 50), SUM(ResolutionWidth + 51), SUM(ResolutionWidth + 52), SUM(ResolutionWidth + 53), SUM(ResolutionWidth + 54), SUM(ResolutionWidth + 55), SUM(ResolutionWidth + 56), SUM(ResolutionWidth + 57), SUM(ResolutionWidth + 58), SUM(ResolutionWidth + 59), SUM(ResolutionWidth + 60), SUM(ResolutionWidth + 61), SUM(ResolutionWidth + 62), SUM(ResolutionWidth + 63), SUM(ResolutionWidth + 64), SUM(ResolutionWidth + 65), SUM(ResolutionWidth + 66), SUM(ResolutionWidth + 67), SUM(ResolutionWidth + 68), SUM(ResolutionWidth + 69), SUM(ResolutionWidth + 70), SUM(ResolutionWidth + 71), SUM(ResolutionWidth + 72), SUM(ResolutionWidth + 73), SUM(ResolutionWidth + 74), SUM(ResolutionWidth + 75), SUM(ResolutionWidth + 76), SUM(ResolutionWidth + 77), SUM(ResolutionWidth + 78), SUM(ResolutionWidth + 79), SUM(ResolutionWidth + 80), SUM(ResolutionWidth + 81), SUM(ResolutionWidth + 82), SUM(ResolutionWidth + 83), SUM(ResolutionWidth + 84), SUM(ResolutionWidth + 85), SUM(ResolutionWidth + 86), SUM(ResolutionWidth + 87), SUM(ResolutionWidth + 88), SUM(ResolutionWidth + 89) FROM hits;
Time: 11.105 ms
Time: 8.922 ms
Time: 9.185 ms
SELECT SearchEngineID, ClientIP, COUNT(*) AS c, SUM(IsRefresh), AVG(ResolutionWidth) FROM hits WHERE SearchPhrase <> '' GROUP BY SearchEngineID, ClientIP ORDER BY c DESC LIMIT 10;
Time: 6.071 ms
Time: 2.539 ms
Time: 2.641 ms
SELECT WatchID, ClientIP, COUNT(*) AS c, SUM(IsRefresh), AVG(ResolutionWidth) FROM hits WHERE SearchPhrase <> '' GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10;
Time: 12.118 ms
Time: 2.991 ms
Time: 2.812 ms
SELECT WatchID, ClientIP, COUNT(*) AS c, SUM(IsRefresh), AVG(ResolutionWidth) FROM hits GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10;
Time: 8.664 ms
Time: 4.891 ms
Time: 4.826 ms
SELECT URL, COUNT(*) AS c FROM hits GROUP BY URL ORDER BY c DESC LIMIT 10;
Time: 5.407 ms
Time: 3.179 ms
Time: 4.233 ms
SELECT 1, URL, COUNT(*) AS c FROM hits GROUP BY 1, URL ORDER BY c DESC LIMIT 10;
Time: 6.185 ms
Time: 4.393 ms
Time: 3.317 ms
SELECT ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3, COUNT(*) AS c FROM hits GROUP BY ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3 ORDER BY c DESC LIMIT 10;
Time: 10.050 ms
Time: 5.916 ms
Time: 6.056 ms
SELECT URL, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND DontCountHits = 0 AND IsRefresh = 0 AND URL <> '' GROUP BY URL ORDER BY PageViews DESC LIMIT 10;
Time: 6.211 ms
Time: 2.643 ms
Time: 2.461 ms
SELECT Title, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND DontCountHits = 0 AND IsRefresh = 0 AND Title <> '' GROUP BY Title ORDER BY PageViews DESC LIMIT 10;
Time: 5.747 ms
Time: 2.652 ms
Time: 2.736 ms
SELECT URL, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND IsRefresh = 0 AND IsLink <> 0 AND IsDownload = 0 GROUP BY URL ORDER BY PageViews DESC LIMIT 10 OFFSET 1000;
Time: 5.280 ms
Time: 2.803 ms
Time: 2.467 ms
SELECT TraficSourceID, SearchEngineID, AdvEngineID, CASE WHEN (SearchEngineID = 0 AND AdvEngineID = 0) THEN Referer ELSE '' END AS Src, URL AS Dst, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND IsRefresh = 0 GROUP BY TraficSourceID, SearchEngineID, AdvEngineID, Src, Dst ORDER BY PageViews DESC LIMIT 10 OFFSET 1000;
Time: 6.184 ms
Time: 3.323 ms
Time: 2.938 ms
SELECT URLHash, EventDate, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND IsRefresh = 0 AND TraficSourceID IN (-1, 6) AND RefererHash = 3594120000172545465 GROUP BY URLHash, EventDate ORDER BY PageViews DESC LIMIT 10 OFFSET 100;
Time: 5.267 ms
Time: 2.694 ms
Time: 2.455 ms
SELECT WindowClientWidth, WindowClientHeight, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND IsRefresh = 0 AND DontCountHits = 0 AND URLHash = 2868770270353813622 GROUP BY WindowClientWidth, WindowClientHeight ORDER BY PageViews DESC LIMIT 10 OFFSET 10000;
Time: 5.209 ms
Time: 2.482 ms
Time: 2.531 ms
SELECT DATE_TRUNC('minute', EventTime) AS M, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-14' AND EventDate <= '2013-07-15' AND IsRefresh = 0 AND DontCountHits = 0 GROUP BY DATE_TRUNC('minute', EventTime) ORDER BY DATE_TRUNC('minute', EventTime) LIMIT 10 OFFSET 1000;
SELECT DATE_TRUNC('minute', EventTime) AS M, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-14' AND EventDate <= '2013-07-15' AND IsRefresh = 0 AND DontCountHits = 0 GROUP BY DATE_TRUNC('minute', EventTime) ORDER BY DATE_TRUNC('minute', EventTime) LIMIT 10 OFFSET 1000;
Time: 5.567 ms
SELECT DATE_TRUNC('minute', EventTime) AS M, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-14' AND EventDate <= '2013-07-15' AND IsRefresh = 0 AND DontCountHits = 0 GROUP BY DATE_TRUNC('minute', EventTime) ORDER BY DATE_TRUNC('minute', EventTime) LIMIT 10 OFFSET 1000;
Time: 2.887 ms
SELECT DATE_TRUNC('minute', EventTime) AS M, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-14' AND EventDate <= '2013-07-15' AND IsRefresh = 0 AND DontCountHits = 0 GROUP BY DATE_TRUNC('minute', EventTime) ORDER BY DATE_TRUNC('minute', EventTime) LIMIT 10 OFFSET 1000;
Time: 2.538 ms
Data size: 75648
[0.006359,0.001031,0.000927],
[0.007311,0.002159,0.001711],
[0.008554,0.003772,0.003413],
[0.008894,0.007014,0.004308],
[0.005971,0.003423,0.002587],
[0.007407,0.00247,0.002089],
[0.010218,0.00461,0.00487],
[0.004711,0.001768,0.002433],
[0.006917,0.002965,0.003116],
[0.021958,0.004241,0.003989],
[0.006864,0.002405,0.002021],
[0.004708,0.002197,0.002204],
[0.005586,0.00251,0.002261],
[0.004628,0.002099,0.002113],
[0.00506,0.002199,0.00222],
[0.005957,0.002955,0.002669],
[0.012301,0.006051,0.007826],
[0.007472,0.00886,0.011859],
[0.008453,0.006062,0.006511],
[0.004108,0.001361,0.001219],
[0.006865,0.002345,0.002053],
[0.006156,0.002535,0.002064],
[0.007954,0.002471,0.00233],
[0.006646,0.004062,0.00408],
[0.003481,0.001017,0.000954],
[0.003899,0.001819,0.001326],
[0.003769,0.001175,0.001163],
[0.006452,0.002718,0.002502],
[0.015984,0.002735,0.002504],
[0.011105,0.008922,0.009185],
[0.006071,0.002539,0.002641],
[0.012118,0.002991,0.002812],
[0.008664,0.004891,0.004826],
[0.005407,0.003179,0.004233],
[0.006185,0.004393,0.003317],
[0.01005,0.005916,0.006056],
[0.006211,0.002643,0.002461],
[0.005747,0.002652,0.002736],
[0.00528,0.002803,0.002467],
[0.006184,0.003323,0.002938],
[0.005267,0.002694,0.002455],
[0.005209,0.002482,0.002531],
[0.005567,0.002887,0.002538],

[rschu1ze]

Here are the measurements with c6a.4xlarge instances: https://pastila.clickhouse.com/?00012d7f/6abbadde357ce424eac9b0db7b1aa307#sOCA4BXIIk/CnS1ic4Kt8w== (runtimes are at the end)

I couldn't spot any error messages in the log output. What makes me suspicious is that the corresponding "pure" ClickHouse measurements on c6a.4xlarge systems return worse runtimes: https://github.com/ClickHouse/ClickBench/blob/main/clickhouse/results/c6a.4xlarge.json (--> see e.g. the cold runs = the first values in each row).

Do you have an explanation how that may be?

Maybe we need to add more checks that the data was successfully imported and the queries return non-empty results?

[theory]

I'm pretty sure it exits with an error on load failure; I saw that a lot while I was working out how to make it work. Same for query failures; they mess with the resulting output.

I ran the test in Docker without greping-out query output and got this. Note that this is just one parquet file; I don't have capacity for 100.

For comparison, here's what I get in Docker for one parquet file and running the base ClickHouse benchmark . Looks fairly consistent at a glance.

[theory]

I'm working through the queries, comparing outputs. Found a that pg_clickhouse isn't pushing down MIN() and MAX(), and that the EventDate column is turning up empty. Going to work on that, then once fixed, go through the rest of the queries and fix any other issues I find.