Performance Benefits of KDB+ Attributes

Attributes in KDB+ are powerful performance optimisations that can dramatically accelerate queries on large datasets. In this blog, we’ll dive into the tangible performance benefits they offer - measuring real-world improvements in speed and efficiency. If you’re new to attributes or want a refresher on their mechanics and use cases, check out my previous blog post for a detailed introduction.

Note

• Results were generated using Q/KDB+ version 4.1 (2025.04.28).
• Times are shown in milliseconds and space in megabytes.
• Figures are shown to three significant figures.

Basic List Operations

To understand the impact of attributes on performance, we begin by analysing basic list operations.

Each test followed this methodology:

1. A sorted list of N unique elements was created using til N.

2. For binary operations, a random element from the list was selected as the second operand.

3. The operation was executed on the list without any attribute, then repeated with each attribute applied.

4. Steps 2–3 were repeated multiple times, and the average execution time was recorded for accuracy.

In addition to the primary list type, I also tested two other structures:

• A unique, unsorted list generated using neg[N]?N (list is sorted when applying sorted and parted attributes).
• A non-unique, unsorted list generated using N?N div 4 (unique attribute excluded for this case).

This allows us to determine if the underlying structure of the list affects performance trends. Unless otherwise noted, the trends were consistent across list types.

Comparison Operators

KDB+ provides several standard comparison operators, familiar to users of other languages:

How the Sorted Attribute Boosts Performance

The sorted attribute enables Q/KDB+ to use binary search instead of linear search, significantly improving performance on large lists. Consider the following example using the less than (<) operator:

List: 1 2 3 4 5 6 7 8 9 10, searching for < 7

1 2 3 4 5 6 7 8 9 10
        ^              // Midpoint: 5 < 7 → Search right half
1 1 1 1 1 . . . . .

6 7 8 9 10
    ^                  // Midpoint: 8 > 7 → Search left half
. . 0 0 0

6 7
^                      // Midpoint: 6 < 7 → Search right
1 .

7
^                      // 7 = 7 → Done
0

The full comparison is completed in just a few steps using binary search — far faster than checking each element sequentially.

Equal (=) and Not Equal (<>)

Surprisingly, the = and <> operators do not appear to benefit from any attribute — even on large lists. The operation time scales roughly linearly with list count. This behaviour contrasts sharply with other comparison operators that benefit from the sorted attribute.

Less Than (<) and Greater Than (>)

These results clearly demonstrate the benefit of the sorted attribute for range-based comparisons. The performance gain becomes increasingly significant as the data volume grows, thanks to binary search.

Less Than Or Equal (<=) and Greater Than Or Equal (>=)

These compound comparisons also benefit from the sorted attribute, although not as dramatically as their strict counterparts. This is likely because the equality portion of the operation introduces a slight overhead compared to strict inequality.

The Find Operator (?)

The find operator (?) searches for an item in a list. If found, it returns the index of the first occurrence. If not found, it returns the count of the list, which is one past the last index.

The find operation shows dramatic performance improvements when using any of the attributes. The unique, parted, and grouped attributes offer nearly 2× better performance than sorted for a list with 100 million elements.

However, these three attributes require additional overhead to store internal structures (such as hash maps), whereas the sorted attribute does not require any extra space — making it a more lightweight option when applicable.

Minimum (min) and Maximum (max)

The min and max functions return the smallest and largest elements in a list, respectively.

When a list is sorted, the minimum and maximum values are simply the first and last elements. This makes these operations constant time, regardless of list count, as reflected in the table above.

Other attributes (unique, parted, grouped) do not show meaningful benefit.

Special Case: Non-Unique, Unsorted Lists

In the case of a non-unique, unsorted list, we observed a performance gain with the parted attribute:

This improvement likely stems from how the parted attribute internally manages buckets of unique values. In a non-unique list, the underlying hash map only needs to examine the keys (i.e., unique values) when computing the minimum or maximum. This subset is often much smaller than the original list, making the operation faster in practice.

Distinct (distinct)

The distinct function returns a list of the unique elements from the input list, preserving their first occurrence.

With the unique attribute, the distinct operation becomes essentially free — the list is already composed of unique elements, so the function simply returns the list as-is. This results in constant-time performance, regardless of the list count.

The parted attribute also provides a significant performance boost. This is because a parted list internally maintains a hash map of unique values, so distinct can efficiently return the map’s keys, bypassing a full scan of the list.

The sorted attribute, by contrast, offers no substantial benefit. While sorted lists could potentially allow for early exit strategies or linear scanning, this doesn’t appear to be leveraged in the current implementation.

Surprisingly, the grouped attribute — despite also maintaining a hash map — performs worse than having no attribute at all. The reason for this is unclear. It may be due to internal overheads in the way grouped lists are structured or accessed.

Special Case: Non-Unique, Unsorted Lists

When applied to a non-unique, unsorted list, the parted attribute once again offers a meaningful advantage:

In this case, the parted attribute delivers a ~5× improvement compared to the normal case (unique, sorted list). This confirms that parted is particularly effective for lists with high duplication, where the internal hash structure enables fast access to the unique keys.

Disk Read Operations

Up to this point, we’ve explored how attributes affect in-memory lists. While useful for illustration, the more common real-world application of attributes is in on-disk (splayed) tables — particularly to optimise disk reads, which are far more costly than memory access.

In this section, we evaluate a few basic queries to see how disk read performance is affected by applying attributes to specific columns.

For the tests, I created several splayed tables with the following schema, each containing a different number of rows:

// Schema
([] time:`timestamp$(); sym:`symbol$(); price:`float$(); size:`long$(); side:`char$())

Query: time > randTime

In time-series data, the time column typically arrives in order, making it a good candidate for the sorted attribute. This query selects sym and price for rows where the time is greater than a randomly chosen timestamp:

randTime:exec rand time from table
select sym, price from table where time>randTime

Both execution time and memory usage (space) was measured for two cases: with and without the sorted attribute on the time column.

The sorted attribute yields substantial improvements in both speed and memory efficiency. This is due to the underlying binary search strategy enabled by sorted data, which minimises disk reads.

Query: time = randTime

In earlier in-memory tests, we observed that the = operator did not benefit from the sorted attribute. Here, we test whether that holds true for on-disk tables:

select sym, price from table where time=randTime

Interestingly, when querying on-disk data, = benefits greatly from the sorted attribute — both in speed and memory usage. This is a contrast to the in-memory case, and it highlights how Q’s columnar storage and I/O behaviour differ between memory and disk.

Query: sym = randSym

The sym column is one of the most commonly queried columns in KDB+, often representing stock tickers or instrument identifiers. Due to its high selectivity and frequent use in filters, it is a natural candidate for attribute optimisation.

In this test, we query for all rows where sym equals a randomly selected symbol:

randSym:exec rand sym from table
select time, side from table where sym=randSym

We evaluate four cases based on the attribute applied to the sym column: No Attribute, Sorted, Parted, and Grouped.

Time

All three attributes — sorted, parted, and grouped — offer significant speedups over the unoptimised case. Of these, the parted attribute delivers the best performance at scale. This aligns with best practices in KDB+, where the sym column in on-disk tables is often parted to optimise equality-based filters.

Space

Using attributes also leads to drastic reductions in memory usage — by orders of magnitude. Notably, sorted and parted produce identical memory footprints, suggesting similar internal mechanisms in how the filter is executed. Grouped shows more memory use but still vastly outperforms the baseline.

Query: max price by sym

A common and often costly query pattern in KDB+ involves performing aggregations by a column — typically sym. These select ... by ... queries group data based on distinct values of a column, then apply an aggregation such as max, min, or avg.

For this test, we perform a simple aggregation: finding the maximum price for each symbol.

select max price by sym from table

We again compare performance across four attribute configurations for the sym column: No Attribute, Sorted, Parted, and Grouped.

Time

The parted attribute yields the most significant performance improvement. This is because a parted column is already physically grouped by its unique values, allowing the aggregation to operate directly on these pre-formed groups without needing to reorganise the data.

In contrast, sorted and grouped offer no meaningful performance advantage over the unoptimised case in this scenario.

Space

When using the parted attribute, memory usage remains constant regardless of dataset size. This suggests that no additional memory is consumed during grouping — likely because the data is already partitioned, and aggregation can proceed without intermediate allocation.

Conclusion

KDB+ attributes offer a powerful way to optimise query performance by influencing how data is accessed and retrieved. As we’ve seen, each attribute — sorted, unique, parted, and grouped — serves a specific purpose and can dramatically reduce query times when applied appropriately. Whether it’s enabling binary search through a sorted list, ensuring fast membership checks via unique, or accelerating group-based operations with parted and grouped, attributes are key to writing high-performance Q code.