RediSearch is a Redis module that provides querying, secondary indexing, and full-text search for Redis.

Last update: Jan 5, 2023

Overview

RediSearch

Querying, secondary indexing, and full-text search for Redis

Overview

RediSearch is a Redis module that provides querying, secondary indexing, and full-text search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch query language to query that data.

RediSearch uses compressed, inverted indexes for fast indexing with a low memory footprint.

RediSearch indexes enhance Redis by providing exact-phrase matching, fuzzy search, and numeric filtering, among many other features.

Getting started

If you're just getting started with RediSearch, check out the official RediSearch tutorial. Also, consider viewing our RediSearch video explainer.

The fastest way to get up and running with RediSearch is by using the RediSearch Docker image.

Trying RediSearch

To try RediSearch, either use the RediSearch Docker image, or create a free Redis Cloud Essentials account to get a RediSearch instance in the cloud.

Docker image

The RediSearch Docker image makes it easy to try RediSearch.

To create a local RediSearch container, run:

$ docker run -p 6379:6379 redislabs/redisearch:latest

To connect to this instance, run:

$ redis-cli

Documentation

The RediSearch documentation provides a complete overview of RediSearch. Helpful sections include:

The RediSearch quick start
The RediSearch command reference
References on features such as aggregations, higlights, stemming, and spelling correction.

Mailing list and forum

Got questions? Join us in #redisearch on the Redis Discord server.

If you have a more detailed question, drop us a line on the RediSearch Discussion Forum.

Client libraries

You can use any standard Redis client library to run RediSearch commands, but it's easiest to use a library that wraps the RediSearch API.

Language	Library	Author	License
Python	redisearch-py	Redis Labs	BSD
Java (Jedis client library)	JRediSearch	Redis Labs	BSD
Java (Lettuce client library)	lettusearch	Redis Labs	Apache-2.0
Java	spring-redisearch	Redis Labs	Apache-2.0
Java	redis-modules-java	dengliming	Apache-2.0
Go	redisearch-go	Redis Labs	BSD
JavaScript	RedRediSearch	Kyle J. Davis	MIT
JavaScript	redis-redisearch	Kyle J. Davis	MIT
TypeScript	redis-modules-sdk	Dani Tseitlin	BSD-3-Clause
C#	NRediSearch	Marc Gravell	MIT
PHP	redisearch-php	Ethan Hann	MIT
PHP	php-redisearch	MacFJA	MIT
Rust	redisearch-api-rs	Redis Labs	BSD
Ruby on Rails	redi_search_rails	Dmitry Polyakovsky	MIT
Ruby	redisearch-rb	Victor Ruiz	MIT
Ruby	redi_search	Nick Pezza	MIT

Other available Libraries

Language	Library	Author	License	Stars	Comments
Rust	redisearch-api-rs	Redis Labs	BSD		API for Redis Modules written in Rust

RediSearch features

Full-Text indexing of multiple fields in Redis hashes
Incremental indexing without performance loss
Document ranking (using tf-idf, with optional user-provided weights)
Field weighting
Complex boolean queries with AND, OR, and NOT operators
Prefix matching, fuzzy matching, and exact-phrase queries
Support for double-metaphone phonetic matching
Auto-complete suggestions (with fuzzy prefix suggestions)
Stemming-based query expansion in many languages (using Snowball)
Support for Chinese-language tokenization and querying (using Friso)
Numeric filters and ranges
Geospatial searches using Redis geospatial indexing
A powerful aggregations engine
Supports for all utf-8 encoded text
Retrieve full documents, selected fields, or only the document IDs
Sorting results (for example, by creation date)

Cluster support

RediSearch has a distributed cluster version that scales to billions of documents across hundreds of servers. At the moment, distributed RediSearch is available as part of Redis Enterprise Cloud and Redis Enterprise Software.

See RediSearch on Redis Enterprise for more information.

License

RediSearch is licensed under the Redis Source Available License Agreement.

Comments

Faulty replication; some changes slaved, some not; mem usage increasing but "on hold"
We had an outage on our live slave systems last week which seems to have been caused by RediSearch (we use a pre-release of RediSearch against Redis 4.0.2, and have protected ourselves enough to minimize enduser impact, and we were able to contain the situation).

Our live system uses RediSearch branch aof_fixes on master as well as slaves.

So not the latest version of RediSearch, and we fully understand we have to upgrade, but we will have to wait a few days until we can update the master (planning, free timeslot, dependencies).

For now, all we are interested in is: "Is it probable that this is fixed in RC 0.99.0?"

What I did just now, is create a new redis slave on a test system, with RediSearch 0.99.0, with slaveof to a live master Redis instance, which still uses version aof_fixes.

The result is similar to what we saw in live slaves a few days back. Here are the symptoms:

A simple set hello world does replicate (Note: we did not test this at crisis time on live)

A complex update from a Lua script (but not hitting any RediSearch commands) does not replicate

The master is 100% okay (redis contents are daily diffed to origin), also RediSearch data seems accurate (but not daily diffed yet)

We disabled snapshotting, and rdb_changes_since_last_save only rises when doing set hello world examples

The used_memory_dataset does reflect all changes coming in (so data is recieved from the master, and is put somewhere in memory, but the Lua initiated changes never make it to the keyspace as far as we can tell)

We consistently get master_link_status:up, master_last_io_seconds_ago:0 and master_sync_in_progress:0

The logs do not show anything out of the ordinary

On the day of the live issue, there was a partial sync at around 02:00AM, which went fine. We scanned the AOF of the master (not rewritten), and the first command of that day (!!!!!) that was able to trigger a RediSearch index write (from a Lua script, SAFEADD) was at 11:43AM. Our two slaves stopped replicating at 11:43AM, both at the same time. After that, memory usage kept climbing on both of the slaves, but the data was stale, and rdb_changes_since_last_save stopped rising (and snapshotting is disabled in the config, so no rdb saves are ever done, except for service stop or client request).

Here are some Graphs of the live issue. From: 2017-11-16 11:00:00 To: 2017-11-16 12:00:00

Master (shows derivative of rdb_changes_since_last_save):

Slave 1 (shows derivative of rdb_changes_since_last_save, notice the flatliner after 11:43):

Slave 2 (shows derivative of rdb_changes_since_last_save, notice the flatliner after 11:43):

Graphs below are of Slave 1, notice the mem usage after 11:43:

At this point, could you provide hints on more data to gather, which could be useful at a later stage (after upgrading RediSearch for example)? I still have the 0.99.0 slave running on a test system with the faulty behaviour, and I'll try to keep it running today.

Hope you can help, cheers, TW
opened by tw-bert 48

RediSearch crash with BGSAVE

RediSearch 1.0.5 , Redis 4.0.7 (also happens on RediSearch 1.0.4, Redis 4.0.6, but upgraded and repro'd to avoid any fuzz over a fixed issue)

After issuing a BGSAVE:

25396:C 24 Jan 14:43:34.387 # Redis 4.0.7 crashed by signal: 11
25396:C 24 Jan 14:43:34.387 # Crashed running the instruction at: 0x7f3eea498085
25396:C 24 Jan 14:43:34.387 # Accessing address: (nil)
25396:C 24 Jan 14:43:34.387 # Failed assertion: <no assertion failed> (<no file>:0)

------ STACK TRACE ------
EIP:
/usr/local/bin/RediSearch/1.0.5/redisearch.so(SortingVector_RdbSave+0x15)[0x7f3eea498085]

Backtrace:
redis-rdb-bgsave *:14213(logStackTrace+0x3c)[0x46b81c]
redis-rdb-bgsave *:14213(sigsegvHandler+0xa3)[0x46ca73]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7f3eecd6f8d0]
/usr/local/bin/RediSearch/1.0.5/redisearch.so(SortingVector_RdbSave+0x15)[0x7f3eea498085]
/usr/local/bin/RediSearch/1.0.5/redisearch.so(DocTable_RdbSave+0x1ec)[0x7f3eea48d3ac]
/usr/local/bin/RediSearch/1.0.5/redisearch.so(IndexSpec_RdbSave+0x9b)[0x7f3eea481e1b]
redis-rdb-bgsave *:14213(rdbSaveObject+0x408)[0x44d0e8]
redis-rdb-bgsave *:14213(rdbSaveKeyValuePair+0x9c)[0x44d2ec]
redis-rdb-bgsave *:14213(rdbSaveRio+0x2b6)[0x44d5b6]
redis-rdb-bgsave *:14213(rdbSave+0x95)[0x44d785]
redis-rdb-bgsave *:14213(rdbSaveBackground+0xa0)[0x44da50]
redis-rdb-bgsave *:14213(bgsaveCommand+0xdf)[0x44dc9f]
redis-rdb-bgsave *:14213(call+0x87)[0x42dc97]
redis-rdb-bgsave *:14213(processCommand+0x36d)[0x42e30d]
redis-rdb-bgsave *:14213(processInputBuffer+0x10d)[0x43e5bd]
redis-rdb-bgsave *:14213(aeProcessEvents+0x15d)[0x42945d]
redis-rdb-bgsave *:14213(aeMain+0x2b)[0x42977b]
redis-rdb-bgsave *:14213(main+0x512)[0x4323d2]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f3eec9d9b45]
--snip--
------ CURRENT CLIENT INFO ------
id=142179 addr=127.0.0.1:51334 fd=9 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=bgsave
argv[0]: 'bgsave'
6428:M 24 Jan 14:43:34.553 # Background saving terminated by signal 11

The BGSAVE failed, but the instance keeps running. I can do a BGREWRITEAOF, and use it to restore to another redis instance. A BGSAVE on that redis instance crashes as well.

Sidenote: AOF is around 5GB and NDA.

opened by tw-bert 33

Crash while inserting

While testing using node/redisearchclient by inserting wikipedia abstracts, redis server crashed.

31654:C 03 Sep 2019 12:45:09.987 * RDB: 389 MB of memory used by copy-on-write
27981:M 03 Sep 2019 12:45:10.092 * Background saving terminated with success
warning: got a bad length when writing to pipe.
27981:M 03 Sep 2019 12:45:35.019 # Out Of Memory allocating 13614287927004626944 bytes!


=== REDIS BUG REPORT START: Cut & paste starting from here ===
27981:M 03 Sep 2019 12:45:35.019 # ------------------------------------------------
27981:M 03 Sep 2019 12:45:35.019 # !!! Software Failure. Press left mouse button to continue
27981:M 03 Sep 2019 12:45:35.019 # Guru Meditation: Redis aborting for OUT OF MEMORY #server.c:3893
27981:M 03 Sep 2019 12:45:35.019 # (forcing SIGSEGV in order to print the stack trace)
27981:M 03 Sep 2019 12:45:35.019 # ------------------------------------------------
27981:M 03 Sep 2019 12:45:35.019 # Redis 5.0.5 crashed by signal: 11
27981:M 03 Sep 2019 12:45:35.019 # Crashed running the instruction at: 0x46f427
27981:M 03 Sep 2019 12:45:35.019 # Accessing address: 0xffffffffffffffff
27981:M 03 Sep 2019 12:45:35.019 # Failed assertion: <no assertion failed> (<no file>:0)

------ STACK TRACE ------
EIP:
redis-server 127.0.0.1:6379(_serverPanic+0x117)[0x46f427]

Backtrace:
redis-server 127.0.0.1:6379(logStackTrace+0x29)[0x4710f9]
redis-server 127.0.0.1:6379(sigsegvHandler+0xac)[0x47179c]
/lib64/libpthread.so.0(+0xf100)[0x7f103d140100]
redis-server 127.0.0.1:6379(_serverPanic+0x117)[0x46f427]
redis-server 127.0.0.1:6379(redisOutOfMemoryHandler+0x2e)[0x42e6ee]
redis-server 127.0.0.1:6379(zmalloc+0x49)[0x437a29]
./redisearch.so(+0xbfbea)[0x7f1034247bea]
./redisearch.so(+0xc02de)[0x7f10342482de]
./redisearch.so(+0xc11d6)[0x7f10342491d6]
./redisearch.so(+0xc1307)[0x7f1034249307]
./redisearch.so(+0xc2260)[0x7f103424a260]
./redisearch.so(FGC_parentHandleFromChild+0x84)[0x7f103424a4f3]
./redisearch.so(+0xc2844)[0x7f103424a844]
./redisearch.so(+0xc86db)[0x7f10342506db]
./redisearch.so(+0x1314ef)[0x7f10342b94ef]
/lib64/libpthread.so.0(+0x7dc5)[0x7f103d138dc5]
/lib64/libc.so.6(clone+0x6d)[0x7f103ce6621d]

------ INFO OUTPUT ------
# Server
redis_version:5.0.5
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:24fc166a0b660279 
redis_mode:standalone
os:Linux 3.10.0-957.10.1.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:4.8.5
process_id:27981
run_id:83d1972f93e1bcf5920d6202b80a018bfc270627
tcp_port:6379
uptime_in_seconds:1178
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:7258846
executable:/usr3/dbtest/redis/redis-server
config_file:/usr3/dbtest/redis/./redis.conf

# Clients
connected_clients:1
client_recent_max_input_buffer:374618
client_recent_max_output_buffer:0
blocked_clients:1

# Memory
used_memory:2061747840
used_memory_human:1.92G
used_memory_rss:1768161280
used_memory_rss_human:1.65G
used_memory_peak:2061747840
used_memory_peak_human:1.92G
used_memory_peak_perc:148.30%   
used_memory_overhead:136151160  
used_memory_startup:791456
used_memory_dataset:1925596680  
used_memory_dataset_perc:93.43% 
allocator_allocated:1394146056  
allocator_active:1397882880
allocator_resident:1435181056   
total_system_memory:50465402880 
total_system_memory_human:47.00G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:38654705664
maxmemory_human:36.00G
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:3736824
allocator_rss_ratio:1.03
allocator_rss_bytes:37298176
rss_overhead_ratio:1.23
rss_overhead_bytes:332980224
mem_fragmentation_ratio:1.27
mem_fragmentation_bytes:377867880
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:387280
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:149373
rdb_bgsave_in_progress:0
rdb_last_save_time:1567539910   
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:22
rdb_current_bgsave_time_sec:-1  
rdb_last_cow_size:407920640
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1 
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0

# Stats
total_connections_received:9
total_commands_processed:2377791
instantaneous_ops_per_sec:3271  
total_net_input_bytes:292353044 
total_net_output_bytes:5135789  
instantaneous_input_kbps:966.14 
instantaneous_output_kbps:15.97 
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:35924
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

# Replication
role:master
connected_slaves:0
master_replid:1920d458e45ff7b3fb9d574c8bdc38049f174e2b
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:130.684637
used_cpu_user:342.585239
used_cpu_sys_children:66.336626 
used_cpu_user_children:126.113013

# Commandstats
cmdstat_FT.CREATE:calls=9,usec=53995,usec_per_call=5999.44
cmdstat_FT.ADD:calls=1007939,usec=110731446,usec_per_call=109.86
cmdstat_FT.DROP:calls=9,usec=13324347,usec_per_call=1480483.00
cmdstat_del:calls=1369806,usec=2325602,usec_per_call=1.70
cmdstat_info:calls=28,usec=3026,usec_per_call=108.07

# Cluster
cluster_enabled:0

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=2535451,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------
id=1369867 addr=127.0.0.1:52188 fd=8 name= age=326 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=271435 qbuf-free=98915 obl=5 oll=0 omem=0 events=r cmd=FT.ADD

------ CURRENT CLIENT INFO ------
id=1369867 addr=127.0.0.1:52188 fd=8 name= age=326 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=271435 qbuf-free=98915 obl=5 oll=0 omem=0 events=r cmd=FT.ADD
argv[0]: 'ft.add'
argv[1]: 'wp'
argv[2]: '1007921'
argv[3]: '1'
argv[4]: 'FIELDS'
argv[5]: 'url'
argv[6]: 'https://en.wikipedia.org/wiki/Never_a_Dull_Moment_(1968_film)'
argv[7]: 'title'
argv[8]: 'Wikipedia: Never a Dull Moment (1968 film)'
argv[9]: 'abstract'
argv[10]: '| runtime        = 99 minutes'

------ REGISTERS ------
27981:M 03 Sep 2019 12:45:35.021 #
RAX:0000000000000000 RBX:000000000052d518
RCX:00007f103ce57a4d RDX:0000000000000000
RDI:0000000000000000 RSI:00007f103d12ca10
RBP:0000000000000f35 RSP:00007f0fc37f41d0
R8 :00007f0fc37f6700 R9 :0000000000000053
R10:0000000000000030 R11:0000000000000000
R12:0000000000000000 R13:00007f0fc37f69c0
R14:00007f0fc37f6700 R15:000000000000000f
RIP:000000000046f427 EFL:0000000000010206
CSGSFS:0000000000000033
27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41df) -> 00000000004fbfcd
27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41de) -> 0000000000000000
27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41dd) -> 00007f0fc0000900
27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41dc) -> 00007f0fc37f4340
27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41db) -> 00007f0fc0004840
27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41da) -> 0000000000000000
27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d9) -> 00007f0fc37f4eb0
27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d8) -> 00007f0fc37f4300
27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d7) -> 59524f4d454d2046
27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d6) -> 4f2054554f20726f
27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d5) -> 6620676e6974726f
27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d4) -> 6261207369646552
27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d3) -> 00007f0fc37f42f0
27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d2) -> 00007f0fc37f43c0
27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d1) -> 0000003000000018
27981:M 03 Sep 2019 12:45:35.021 # (00007f0fc37f41d0) -> 00007f0fc37f4330

c:bug

opened by aflin 32

Support for Chinese tokenization
Following the recent refactor of the tokenizer API, it's now much easier to add additional tokenizers.

Before we'll add it to the extnesion API, let's first add a Chinese one.

A few notes on that:

The candidates I'm seeing as valid options are:

a. https://github.com/yanyiwu/cjieba
b. https://github.com/lionsoul2014/friso

Another option would be to take their dictionaries, and implement something based on our own triemap, with the advantage of scanning the dictionary letter by letter and not doing a full lookup for each term. That can be left to a later stage or if the libraries are not fast enough.

Currently we do not support external data files, and all the algorithms rely on a very big dictionary (a few megabytes). There are two options here: include it as static data in the module and make it very big, or rely on an external file that will need to be distributed alongside redisearch (not a problem in the docker image, more so in Enterprise Cluster).

If we're going with bundling it into the source, we might want to consider making the whole Chinese support a compile time option, as many people will not need it.

Let's start by making this work, and then discuss the packaging options. Another would be a script that downloads the dictionary and saves it into Redis itself :) This might actually a good distribution option if we're going with a non bundled version.

Please make sure that whatever we have works well with mixed Chinese / English documents.

We will not do automatic language detection, and for now we will not add anything to the API. FT.ADD already supports the LANGUAGE modifier, we can use LANGUAGE chinese to trigger the special tokenizer. So when creating the add document context, based on the language you select the tokenizer.

PS Make sure whatever we end up doing is thread safe as tokenization is done in the thread pool.
c:enhancement
opened by dvirsky 32
RediSearch - FT.SEARCH with Field Modifiers or INFIELDS option

Trying to perform FT.SEARCH using Field Modifier and INFIELDS option but getting wrong results. Please see below. I mentioned to search within Title but got result even if it contains in description. Please suggest the right syntax if I am doing wrong. Thanks in advance.

opened by nreddipalle 31
Unused FT -> high CPU -> production issues, possibly GC related
Hi,

We've fairly recently upscaled our usage of RediSearch, including some larger datasets than the last two years. This caused issues on unrelated Redis traffic. In my absence (holidays) our IT had already disabled populating AND using the larger FT index (this was possible, since the new functionality is still in beta), but it didn't solve the production issues. I just solved the issue by doing a simple FT.DROP . Here are some CPU graphs (larger FT dataset was introduced on April 1st 2019):

I'm showing the CPU usage of a Redis slave here (running on Azure, replicated from our LAN through a VPN tunnel). CPU usage doesn't seem like much, but one core is almost constantly at 85% CPU. It is directly related to the live issues, above a certain percentage. Live performance became unacceptable after/around April 23rd.

Here's some stats, from just before I dropped the index (about 4M documents):

index_name ~ftidx.lo.pklstat index_options fields cusno_public_shanos type TAG SEPARATOR ; SORTABLE pkkno type TAG SEPARATOR ; SORTABLE artno type TAG SEPARATOR ; SORTABLE unmcd type TAG SEPARATOR ; SORTABLE pklnobox type TAG SEPARATOR ; SORTABLE iShipped type NUMERIC SORTABLE iReadyToShip type NUMERIC SORTABLE iOnShelve type NUMERIC SORTABLE iIDLiPklStat type NUMERIC SORTABLE NOINDEX dtShipped type NUMERIC SORTABLE fMechanicalsVV type NUMERIC NOINDEX num_docs 3949195 max_doc_id 4254756 num_terms 0 num_records 1.8446744073707577e+19 inverted_sz_mb 17592186044410.168 offset_vectors_sz_mb 0 doc_table_size_mb 511.26409149169922 sortable_values_size_mb 1079.3353042602539 key_table_size_mb 97.942524909973145 records_per_doc_avg 4671013731585.6904 bytes_per_record_avg 1 offsets_per_term_avg 0 offset_bits_per_record_avg -nan gc_stats current_hz 1 bytes_collected 6114654 effectiv_cycles_rate 0.018745357995123502 cursor_stats global_idle 0 global_total 0 index_capacity 128 index_total 0

My uneducated guess would be: The GC. We also noticed a very high number:

num_records 1.8446744073707577e+19

But we don't know exactly what the num_records metric means - we only know that the index contains (almost) 4M rows, like shown in num_docs .

What do you guys think? Can I tune this somehow?

I'll be happy to give you an rdb, but it's a couple of G's. Just let me know.
opened by tw-bert 29
Add info and info_len field for suggestion trie node

The original trie node can only have 'str' field for matching. Some application needs more fields to save attached information for the terminal trie node. So we supply the info_len and dynamic info fields.

opened by xushenkun 29

nil values returned for non-existent fields when sorting

RS returns a nil value when I am sorting and the document doesn't have that field. I am investigating further to understand if that happens in all cases or there are special circumstances that lead to this.

The document p:734802 does NOT have a field "price". Here I dump the entire document, add it to search, and then search for it with sorting.

127.0.0.1:6379> HGETALL p:734802
 1) "imgurlsmall"
 2) "https://storage.googleapis.com/lwimg/mh/l/20190615/5bc8cded36741bf5e2bce5fad41fe6d4.png"
 3) "feed"
 4) "f:themotorcafenew"
 5) "_t"
 6) "1560589933"
 7) "title"
 8) "2019 MULTISTRADA 1260S TOURING - Ducati"
 9) "tags"
10) "vm:77,w:couchride:t:europe,vm:146,w:couchride:t:adventure423,vm:162,vm:191,w:couchride:t:dealer,w:couchride:t:new,t:image,w:couchride:t:hasvin,w:couchride:t:touring,w:couchride:t:ducati,w:couchride:t:multistrada,w:couchride:t:multistrada1260,w:couchride:t:multistrada1260s,w:couchride,f:themotorcafenew,cf:d:729,cf:m:ari,CA"
11) "location"
12) "Sunnyvale, CA"
13) "geo"
14) "-122.0363496 37.36883"
15) "created"
16) "2019-06-15 09:12:13"
17) "ctitle"
18) "2019 MULTISTRADA 1260S TOURING Ducati"
19) "mpid"
20) "cf:m:ari"
21) "ct"
22) "l"
23) "year"
24) "2019"
25) "imgurl"
26) "https://cdnmedia.endeavorsuite.com/images/ThumbGenerator/Thumb.aspx?img=https%3a%2f%2fcdnmedia.endeavorsuite.com%2fimages%2fassets%2fimg%2fno-image-if.png&mw=350&mh=200&f=1&fz=1"
27) "vin"
28) "zdmaaekw1kb007485"
29) "psd"
30) "ari"
31) "is"
32) "\n\x0f\[email protected]\x93%\x12\x17[\xda#\xdf\x97F\a\x05"
33) "mileage"
34) "000003"
35) "ssc"
36) "2272"
37) "st"
38) "CA"
39) "did"
40) "cf:d:729"
41) "ss"
42) "88329"
43) "url"
44) "https://www.themotorcafe.com/inventory/2019-ducati-multistrada-1260s-touring-sunnyvale-ca-94087-2643036i"
45) "imgurllarge"
46) "https://storage.googleapis.com/lwimg/mh/l/20190615/105ec8b4ff94206ae43fd3fd9619aa8f.png"
47) "ssf"
48) "9399"


127.0.0.1:6379> FT.ADDHASH il1 p:734802 1.0 REPLACE
OK

127.0.0.1:6379> "FT.SEARCH" "il1" "@ctitle:multistrada* 1260s* @tags:{w\\:couchride\\:t\\:adventure423} @year:[2019 inf] @geo:[-122.0363496 37.36883 10 mi]" "LIMIT" "0" "20" "SORTBY" "price" "DESC"
1) (integer) 1
2) "p:734802"
3)  1) price                           \<----------- HERE
    2) (nil)                                \<----------- HERE
    3) imgurlsmall
    4) "https://storage.googleapis.com/lwimg/mh/l/20190615/5bc8cded36741bf5e2bce5fad41fe6d4.png"
    5) feed
    ... remaining fields .....

Let's try to explicitly "delete" this field:

127.0.0.1:6379> HDEL p:734802 price
(integer) 0
127.0.0.1:6379> FT.ADDHASH il1 p:734802 1.0 REPLACE
OK
127.0.0.1:6379> "FT.SEARCH" "il1" "@ctitle:multistrada* 1260s* @tags:{w\\:couchride\\:t\\:adventure423} @year:[2019 inf] @geo:[-122.0363496 37.36883 10 mi]" "LIMIT" "0" "20" "SORTBY" "price" "DESC"
1) (integer) 1
2) "p:734802"
3)  1) price
    2) (nil)
    3) imgurlsmall
   ..... more ....

"price" is still there in the results. If I ask for specific fields with RETURN, then it's ok.

This is causing the go library I use to parse the result to fail: https://github.com/gomodule/redigo/issues/437

opened by mmasouras 28

Patches for OSX compilation warnings
Fix compilation warnings caught on OS X:

NormalizeFunc was defined twice: remove the duplicate and fix up #includes

Avoid unsigned integer overflow bug vector in trie_type.c: slen-len isn't safe when the lengths are unsigned.

Also modified the main Makefile to use the -Werror flag in all environments, since OS X doesn't complain anymore.
opened by mheiber 26
Redis 4.0.11 crashed by signal: 11

=== REDIS BUG REPORT START: Cut & paste starting from here === 6215:M 11 Oct 19:07:34.617 # Redis 4.0.11 crashed by signal: 11 6215:M 11 Oct 19:07:34.617 # Crashed running the instruction at: 0x555589cc5015 6215:M 11 Oct 19:07:34.617 # Accessing address: (nil) 6215:M 11 Oct 19:07:34.617 # Failed assertion: (:0)

------ STACK TRACE ------ EIP: /usr/bin/redis-server 0.0.0.0:6600(je_arena_tcache_fill_small+0x1e5)[0x555589cc5015]

Backtrace: /usr/bin/redis-server 0.0.0.0:6600(logStackTrace+0x5a)[0x555589c5f65a] /usr/bin/redis-server 0.0.0.0:6600(sigsegvHandler+0xb1)[0x555589c5fe11] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fc393e47890] /usr/bin/redis-server 0.0.0.0:6600(je_arena_tcache_fill_small+0x1e5)[0x555589cc5015] /usr/bin/redis-server 0.0.0.0:6600(je_tcache_alloc_small_hard+0x14)[0x555589ce8544] /usr/bin/redis-server 0.0.0.0:6600(je_arena_ralloc+0x826)[0x555589cc7bf6] /usr/bin/redis-server 0.0.0.0:6600(je_realloc+0xf3)[0x555589cbaaf3] /usr/bin/redis-server 0.0.0.0:6600(zrealloc+0x22)[0x555589c24d02] /etc/redis/modules/redisearch-1.4.0.so(Buffer_Grow+0x54)[0x7fc391808eb4] /etc/redis/modules/redisearch-1.4.0.so(+0x5340b)[0x7fc39181c40b] /etc/redis/modules/redisearch-1.4.0.so(InvertedIndex_WriteEntryGeneric+0x7d)[0x7fc39181d1ed] /etc/redis/modules/redisearch-1.4.0.so(InvertedIndex_WriteForwardIndexEntry+0x7d)[0x7fc39181d2ed] /etc/redis/modules/redisearch-1.4.0.so(+0x5136a)[0x7fc39181a36a] /etc/redis/modules/redisearch-1.4.0.so(+0x519e6)[0x7fc39181a9e6] /etc/redis/modules/redisearch-1.4.0.so(+0x51ea8)[0x7fc39181aea8] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fc393e3c6db] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fc393b6588f]

------ INFO OUTPUT ------

Server

redis_version:4.0.11 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:4bc3aa215ca9cf78 redis_mode:standalone os:Linux 4.15.0-36-generic x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:atomic-builtin gcc_version:7.3.0 process_id:6215 run_id:23c9a2164d6237dd48dbf6aae6342f9f9df2dda0 tcp_port:6600 uptime_in_seconds:4405 uptime_in_days:0 hz:10 lru_clock:12558198 executable:/usr/bin/redis-server config_file:/etc/redis/redis-hi-instant-stag.zooster.nl.conf

Clients

connected_clients:4 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:1

Memory

used_memory:103099856 used_memory_human:98.32M used_memory_rss:219738112 used_memory_rss_human:209.56M used_memory_peak:198129824 used_memory_peak_human:188.95M used_memory_peak_perc:52.04% used_memory_overhead:3044344 used_memory_startup:786800 used_memory_dataset:100055512 used_memory_dataset_perc:97.79% total_system_memory:2090156032 total_system_memory_human:1.95G used_memory_lua:37888 used_memory_lua_human:37.00K maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction mem_fragmentation_ratio:2.13 mem_allocator:jemalloc-4.0.3 active_defrag_running:0 lazyfree_pending_objects:0

Persistence

loading:0 rdb_changes_since_last_save:94615 rdb_bgsave_in_progress:1 rdb_last_save_time:1539284754 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:3 rdb_current_bgsave_time_sec:0 rdb_last_cow_size:17166336 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:0

Stats

total_connections_received:34 total_commands_processed:521727 instantaneous_ops_per_sec:4988 total_net_input_bytes:22978214 total_net_output_bytes:11222261 instantaneous_input_kbps:4.76 instantaneous_output_kbps:0.18 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 evicted_keys:0 keyspace_hits:633773 keyspace_misses:105 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:10180 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0

Replication

role:master connected_slaves:0 master_replid:546a55113d9284281b4a829c37a10b4370345895 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0

CPU

used_cpu_sys:28.42 used_cpu_user:203.27 used_cpu_sys_children:4.29 used_cpu_user_children:26.47

Commandstats

cmdstat_sadd:calls=4221,usec=25237,usec_per_call=5.98 cmdstat_del:calls=303450,usec=996872,usec_per_call=3.29 cmdstat_FT.SEARCH:calls=69,usec=4399,usec_per_call=63.75 cmdstat_hset:calls=4221,usec=201846,usec_per_call=47.82 cmdstat_FT.AGGREGATE:calls=232,usec=9559,usec_per_call=41.20 cmdstat_FT.ADDHASH:calls=66436,usec=26641655,usec_per_call=401.01 cmdstat_get:calls=112,usec=631,usec_per_call=5.63 cmdstat_mget:calls=2,usec=905,usec_per_call=452.50 cmdstat_exec:calls=67555,usec=411243,usec_per_call=6.09 cmdstat_keys:calls=5,usec=203513,usec_per_call=40702.60 cmdstat_scan:calls=5133,usec=702923,usec_per_call=136.94 cmdstat_FT.CREATE:calls=4,usec=20690,usec_per_call=5172.50 cmdstat_smembers:calls=6,usec=79187,usec_per_call=13197.83 cmdstat_hgetall:calls=68852,usec=2106792,usec_per_call=30.60 cmdstat_multi:calls=1322,usec=962,usec_per_call=0.73 cmdstat_info:calls=36,usec=3750,usec_per_call=104.17 cmdstat_FT.DROP:calls=4,usec=4481329,usec_per_call=1120332.25 cmdstat_ping:calls=63,usec=100,usec_per_call=1.59 cmdstat_FT.INFO:calls=4,usec=8629,usec_per_call=2157.25

Cluster

cluster_enabled:0

Keyspace

db0:keys=26899,expires=0,avg_ttl=0

opened by vlradstake 25
ReplyError: Could not load document

Works perfectly with latest dockerimage but fails with RedisLabs ePack.

I am using FT.ADDHASH after a hmset.

Any hint on how to get a more descriptive error?

opened by kilianc 25
Enabled Pull Request CI Performance info generation and publishing
This PR enables Automated performance analysis summaries to be generated and published to PRs that are labeled with action:run-benchmark label.

Here's a sample output of a PR comment: https://github.com/RedisGraph/RedisGraph/pull/2789#issuecomment-1371139517

Notice that multiples pushes to the PR will always update the previous comment to avoid bloating the PR.

action:run-benchmark
opened by filipecosta90 1
Inconsistent behavior of sorting functionality
Hi

Sorting documentation mentioned:

If an index includes sortable fields, you can add the SORTBY parameter to the search request (outside the query body), and order the results by it. https://redis.io/docs/stack/search/reference/sorting/

However I could created index without SORTABLE keyword and still can use SORTBY option. Is it error in the documentation/implementation?

FT.CREATE personIdx ON JSON SCHEMA $.name as name TEXT $.age as age NUMERIC JSON.SET person2 $ '{"name": "Peter", "age": 45}' FT.SEARCH personIdx "@name:(John|Peter)" SORTBY name

Redis Stack: 7.0.6 RC3
opened by sergey-morenets 0
Sorting documentation still uses deprecated FT.ADD command
Hi

It seems that mentioned page still refers to FT.ADD command although it's deprecated:

# Add some users > FT.ADD users user1 1.0 FIELDS first_name "alice" last_name "jones" age 35 > FT.ADD users user2 1.0 FIELDS first_name "bob" last_name "jones" age 36

https://redis.io/docs/stack/search/reference/sorting/
x:docs
opened by sergey-morenets 0
Make refcount for doctable metadata atomic - [MOD-4571]

This PR makes the increment and decrement of the document metadata ref count atomic and handles the potential freeing of a dmd.

For safer handling, all the functions that return a reference to a dmd also increase its ref count, and it's the caller's responsibility to decrease the ref count when it no longer needs it.

opened by GuyAv46 1

Releases(v2.6.4)

v2.6.4(Dec 29, 2022)
This is a maintenance release for RediSearch 2.6

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent.

Details

Bug fixes:

#3289 Potential crash when querying multiple fields (MOD-4639)

#3279 Potential crash when querying using wildcard * on TAG field (MOD-4653)

Improvements:

#3256 Support IPv6 on cluster set command

#3194 Add the query dialects that are in use to FT.INFO and INFO MODULE commands (MOD-4232)

#3258 Add the module version and Redis version to INFO MODULE

Source code(tar.gz)
Source code(zip)
v2.6.3(Nov 30, 2022)
This is the General Availability release of RediSearch 2.6.

Highlights

This new major version introduces the ability to search using wildcard queries for TEXT and TAG fields. This enables the frequently asked feature of suffix search (*vatore and ant?rez are now supported). In addition, the 2.6 release is all about multi-value indexing and querying of attributes for any attribute type ( Text, Tag, Numeric, Geo and Vector) defined by a JSONPath leading to an array or to multiple scalar values. Lastly, this version adds support for indexing double-precision floating-point vectors and range queries from a given vector.

Details

Improvements:

#2886 Support for wildcard queries for TEXT and TAG fields, where

? matches any single character

* matches zero or more characters

use ' and \ for escaping, other special characters are ignored

#2932 Optimized wildcard query support (i.e., suffix trie)

Multi-value indexing and querying

#2819, #2947 Multi-value text search - perform full-text search on array of string or on a JSONPath leading to multiple strings

#3131 Geo #3118 Vector #2985 Numeric #3180 Tag

#3060 Return JSON rather than scalars from multi-value attributes. This is enabled via Dialect 3 in order not to break existing applications.

Support indexing and querying of multi-value JSONPath attributes and/or arrays (requires JSON >2.4.1)

#3182 Support for SORTABLE fields on JSON in an implicit un-normalized form (UNF)

#3156 Vector similarity 0.5.1:

Better space optimization selection (#175)

Aligning index capacity with block size (#177)

#3129 Support FLOAT64 as vector data type

#3176 Range query support

#3105 Support query attributes for vector queries

Bugs (since 2.6-RC1 / v2.6.1):

#3197 Failure to create temporary indices

#3098 Wrong return value in Geo query

#3230 Use the correct total number of matching records

Notes:

The version inside Redis will be 2.6.3 in semantic versioning. Since the version of a module in Redis is numeric, we could not add a GA flag.

Minimal Redis version: 6.0

With this release, we stop supporting direct upgrades from RediSearch v1.4 and v1.6 that are End-of-Life. Such rdb files can still be upgraded to RediSearch 2.0 first

If indexing and querying RedisJSON data structures, this version is best combined with RedisJSON 2.4 GA (v2.4.1 onwards)

License update: see LICENSE.txt

Source code(tar.gz)
Source code(zip)
v2.4.16(Nov 7, 2022)
This is a maintenance release for RediSearch 2.4

Update urgency: HIGH: There is a critical bug that may affect a subset of users. Upgrade!

Details

Bug fixes:

#2962 Crash upon AOF preload - Use local GC context in periodic callback (MOD-3951)

#2863 High CPU Utilization - Change MAX_LEV_DISTANCE to 4 (MOD-3563)

#3041, #3063, #3051, #3143 Several memory leaks (MOD-4121, MOD-4252)

#3119 Crash on intersect iterator GetCriteriaTester (MOD-4200)

#3128 Filter rules must be reevaluated per index per document (MOD-4207)

#3127 Fix assertion failure on wrong result counting, which leads to a crash (MOD-4214)

#3171 Missing implementation of NumericRangeIterator_OnReopen which lead to crash (MOD-4255)

#3191 Wrong query iterator casting which resulted in 100% CPU utilization (MOD-4290)

#3197 Release failed to create temporary indices on the main thread (MOD-4388)

#2981 Double freeing in iterators of hybrid queries resulting in crash (MOD-4411)

#3161 Latency used to increase over time when combining INKEYS and wildcard query (MOD-4343)

Source code(tar.gz)
Source code(zip)
v2.6.1(Nov 1, 2022)
This is the first Release Candidate of RediSearch 2.6.

Highlights

This new major version introduces the ability to search using wildcard queries for TEXT and TAG fields. This enables the frequently asked feature of suffix search (*vatore and ant?rez are now supported). In addition, the 2.6 release is all about multi-value indexing and querying of attributes for any attribute type ( Text, Tag, Numeric, Geo and Vector) defined by a JSONPath leading to an array. Lastly, this version adds support for indexing double-precision floating-point vectors and range queries from a given vector.

Details

Improvements:

#2886 Support for wildcard queries for TEXT and TAG fields, where

? matches any single character

* matches zero or more characters

use ’ and \ for escaping, other special characters are ignored

#2932 Optimized wildcard query support (i.e., suffix trie)

Multi-value indexing and querying

#2819, #2947 Multi-value text search - perform full-text search on array of string or on a JSONPath leading to multiple strings

#3131 Geo #3118 Vector #2985 Numeric #3180 Tag

#3060 Return JSON rather than scalars from multi-value attributes. This is enabled via Dialect 3 in order not to break existing applications.

#3182 Support for SORTABLE fields on JSON in an implicit un-normalized form (UNF)

#3156 Vector similarity 0.5.1:

Better space optimization selection (#175)

Aligning index capacity with block size (#177)

#3129 Support FLOAT64 as vector data type

#3176 Range query support

#3105 Support query attributes for vector queries

Notes:

The version inside Redis will be 2.6.1 in semantic versioning. Since the version of a module in Redis is numeric, we could not add a Release Candidate flag. The GA version will follow soon.

Minimal Redis version: 6.0

Source code(tar.gz)
Source code(zip)
v2.4.15(Oct 3, 2022)
This is a maintenance release for RediSearch 2.4

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent.

Details

Bug fixes:

#3095 Replace order of parsing the parameters and query in the coordinator (MOD-4205)

#3012 Improved efficiency of LLAPI findInfo which could reduce stability during upgrade in Redis Enterprise (MOD-4197, MOD-4052)

#3040, #3049 fix for SORTBY numeric field for non-SORTABLE fields on the coordinator (MOD-4115)

#3050 Results from fields from a missing value should come last when combined with SORTBY. (MOD-4120 )

Improvements:

#3047 Add strlen string function to FT.AGGREGATE (MOD-4141)

#3038 Add number_of_uses to FT.INFO for the number of times the index was queried

Source code(tar.gz)
Source code(zip)
v2.6.0(Sep 11, 2022)
This is the first Milestone of RediSearch 2.6.

Highlights

This first milestone release introduces the ability to search using wildcard queries for TEXT and TAG fields. This enables the frequently asked feature of suffix search. In addition, the 2.6 release is all about multi-value indexing and querying of attributes defined by a JSONPath leading to an array. In this first milestone release, we add support for full-text search indexing and querying multi-value string attributes. In upcoming milestone releases, we'll add multi-value indexing support for numeric and geo attribute types.

Details

Improvements:

#2886 Support for wildcard queries for TEXT and TAG fields, where

? matches any single character

* matches zero or more characters

use ’ and \ for escaping, other special characters are ignored

#2932 Optimized wildcard query support (i.e., suffix trie)

#2819, #2947 Multi-value text search - perform full text search on array of string or on a JSONPath leading to multiple strings

#2953 Vector similarity 0.4.2:

Better space optimization selection (#175)

Aligning index capacity with block size (#177)

Performance improvements:

#2958 Only scan modified indexes

#2906 Optimization for sorting an intersect iterator's children iterators

Bugs:

#3017 Redis crashes on a new doc Numeric index [MOD-4057]

#2950, #2952, #2965 Avoid terms dictionary rehashing on multiple readers

#2865 Endless loop on negative radius search

Source code(tar.gz)
Source code(zip)
v2.4.14(Aug 23, 2022)
This is a maintenance release for RediSearch 2.4

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent.

Details

Improvements:

Vector similarity v0.3.2

#2955 Add timeout during prefix query (MOD-3949)

#2957 Efficient removal from prefix-list for cases with many indices

#2760, #2916 Expose Index statistics information in INFO MODULES section [MOD-2611]

Bugs:

#2937 Returning NULL response after encountering an expired document (MOD-3515)

#2962 Crash upon AOF preload (MOD-3951)

#2986 Memory leak related to schema prefixes

#2967 Bug in the intersect iterator

#2900 Disable SORTBY + MAX in FT.SEARCH which could cause crash (but was not supported) [MOD-3540]

Source code(tar.gz)
Source code(zip)
v2.4.11(Jul 27, 2022)
This is a maintenance release for RediSearch 2.4

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent.

Details

Bug fixes:

#2892 Combining SORTBY with MAX on FT.SEARCH (which is not supported) caused an inconsistent response and out-of-memory (MOD-3540, MOD-3644)

VecSim v0.3.1

HNSW indices: reclaim memory upon deletion - HNSW index's data structures now reclaim memory and shrink upon deletion

Improvements:

VecSim v0.3.1

HNSW indices: delete procedure is up to 40% faster

More accurate memory consumption reporting for HNSW indices

Source code(tar.gz)
Source code(zip)
v2.4.10(Jul 27, 2022)
This is a maintenance release for RediSearch 2.4

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent.

Details

Bug fixes:

#2863 Crash due to too high (Levenstein) DISTANCE in FT.SPELLCHECK. This fix limits the DISTANCE to 4. (MOD-3563)

#2875 Not all documents with Vector fields were indexed with Redis-on-Flash (MOD-3584)

#2846 Enforce Redis Enterprise memory limit for vector indices.

Source code(tar.gz)
Source code(zip)
v2.4.9(Jun 30, 2022)
This is a maintenance release for RediSearch 2.4

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent.

Details

Bug fixes:

#2837, #2836 crash on FT.AGGREGATE "... APPLY '-INF % -1'..."

#2814 FT.EXPLAIN w/o parameters causes a crash

#2790 Incorrect num_terms value in FT.INFO after a term is deleted from all the docs (Garbage Collection)

#2804 Freeze when OFFSET+LIMIT was greater than maxSearchResults (config)

#2791 Add BlockedClientMeasureTime to coordinator for more correct performance stats

#2802 Tagged parts of keys (curly brackets {}) are now returned by FT.SEARCH.

Improvements:

#2806 Do not load JSON API when RediSearch inits as a library

Minor breaking change

As pointed out above, #2802 is a bug fix. However, if your application was relying on RediSearch wrongly trimming the tagged part of a key (using {}), this could break your application. This only applies to users who are using RedisSearch in clustered databases.

Source code(tar.gz)
Source code(zip)
v2.4.8(May 25, 2022)
This is a maintenance release for RediSearch 2.4

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent. However, if you're using Vector Similarity (introduced in RediSearch 2.4), there are some critical bugs that may affect a subset of users. Please Upgrade!

Details

Bugfixes:

#2739 Memory leak in coordinator related to Vector Similarity (MOD-3023)

#2736, #2782 Memory allocation restrictions for Vector Similarity indices (causing OOM) (MOD-3195)

#2755 Compare the entire vector field name instead of a prefix when creating a new vector index

#2780 Init all vars in EvalContext (which might have led to crashes in clustered databases)

Improvements:

#2740 Performance optimisation for hybrid vector queries

Source code(tar.gz)
Source code(zip)
v2.4.6(May 25, 2022)
This is a maintenance release for RediSearch 2.4

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent.

Details

Bugfixes:

#2716 Removed assert that could cause crashes with replica-of (MOD-3008, MOD-3012)

#2734 ON_TIMEOUT RETURN policy fix: return results obtained until timeout rather than discarding them

#2714 Memory leak on non-TLS setup in Coordinator

Source code(tar.gz)
Source code(zip)
v2.4.5(Apr 27, 2022)
This is a maintenance release for RediSearch 2.4

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent.

Details

Bug fixes:

#2702: INKEYS combined with Vector Similarity caused server unresponsiveness (MOD-2952)

#2705: Incorrect results when deleting a document that was skipped at index time

#2698: Synonyms in Chinese

Improvements:

#2694: Performance: In a TEXT field, skip term iterator if term does not appear in requested field

Source code(tar.gz)
Source code(zip)
v2.4.3(Mar 29, 2022)
This is the General Availability release of RediSearch 2.4

Headlines:

RediSearch 2.4 introduces a new capability, Vector Similarity Search (VSS), which allows indexing and querying vector data stored (as BLOBs) in Redis hashes.

It also introduces a new query syntax to address query parser inconsistencies found in previous versions of RediSearch. Users can now choose between Dialect version 1 (to keep existing query parser behavior) or Dialect version 2 (to switch to the updated behavior).

All VSS queries or any query using the PARAMS option must use Dialect version 2

What's new in 2.4?

FT.CREATE is extended to support the creation of 2 popular types of vector indexes:

FLAT Index - This type of index is used when the recall is more important than the speed of query execution. A query vector will be compared to all vectors in a flat index. The search results will return the exact Top K Nearest Neighbors to the query vector

Hierarchical Navigable Small World (HNSW) Index. A modified implementation of the library written by the author of the influential academic paper. An HNSW vector index is used when the speed of query execution is preferred over recall. The results returned are Approximate Nearest Neighbors (ANNs). You can try out different HNSW index parameters (M, EFCONSTRUCTION, EFRUNTIME) to improve the “recall vs speed” balance

Use FT.SEARCH to retrieve the Top K hashes with the most similar vectors to a given query vector

Hybrid queries - Discover the power of hybrid queries in FT.SEARCH Use hybrid queries to retrieve Redis hashes matching a combination of vector and non-vector search criteria. Non-vector search criteria can include expressions combining NUMERIC, TEXT, TAG, GEO fields. Hybrid queries are commonly used in modern e-commerce search applications featuring “visual” similarity + metadata similarity. For example, find products that are visually similar to a given image within a price range and/or geo-location. All in one query.

Use FT.CONFIG to set DEFAULT_DIALECT at the module level. By default, DEFAULT_DIALECT is set to 1

Override DIALECT. It is possible to override the module-level dialect for a specific command run. You can specify dialect when executing any of the following commands: FT.SEARCH, FT.AGGREGATE, FT.EXPLAIN, FT.EXPLAINCLI,FT.SPELLCHECK If you do not specify dialect when running any of these commands, the default module-level dialect value will be used

Details

Features and improvements:

#2671 Add Dialect support

#2645 Ignore NULL values in JSON documents on indexing. (prior behaviour would ignore the entire document)

Performance enhancements (since 2.4-RC1):

#2647 normalize vector once for ad-hoc flat search

#2638 optimized hybrid query when no scores are required

#2653 updating specific field load optimization rule

#2670 Use REDISMODULE_EVENT_SHUTDOWN to clear resources

Security and privacy (since 2.4-RC1):

#2584 Added support for tls-key-file-pass capability (MOD-2086)

Bug fixes (since 2.4-RC1):

#2651 Client freeze on docs expire during query

#2641 Memory leak in Coordinator

#2654 VecSim hybrid query - return empty iterator for invalid intersection child iterator

Introducing DIALECT

RediSearch 2.4.3 introduces a new query syntax to address query parser inconsistencies found in previous versions of RediSearch. Users can now choose between:

Dialect version 1 (to keep query dialect as in RediSearch 2.2) or

Dialect version 2 (to use updated dialect).

Existing RediSearch 2.2 users will not have to modify their queries as the default dialect is 1 HOWEVER, and over time, we encourage all RediSearch users to gradually update their queries to use dialect version 2

Background

We found that under certain conditions, some query parsing rules did not behave as originally intended Particularly, some queries containing the operators below could return unexpected results

1. AND 2. quotes, ~, -, and % (exact, optional, negation, fuzzy) 3. OR

To minimize the impact on existing RediSearch users not affected, it was decided that a DIALECT setting could be introduced to allow:

Existing queries to run without any modification (DIALECT 1)

New queries to benefit from the updated query parsing behavior (DIALECT 2)

Examples of queries impacted

Your existing queries may behave differently under different DIALECT versions, if they fall into any of the following categories:

1. Your query has a field modifier followed by multiple words.

Consider this simple query @name:James Brown.

Here the field modifier @name is followed by 2 words James and Brown

In DIALECT 1, this query would be interpreted as find James Brown in the @name field. In DIALECT 2, this query will be interpreted as find James in the @name field AND Brown in ANY text field. In other words, it would be interpreted as (@name:James) Brown In DIALECT 2, You could achieve the default behavior as in dialect 1 by updating your query to @name:(James Brown)

2. Your query uses quotes, ~, -, % (exact, optional, negation, fuzzy)

Consider a simple query with negation -hello world.

In DIALECT 1, this query is interpreted as "find values in any field that does not contain hello AND does not contain world". The equivalent of -(hello world) or -hello -world. In DIALECT 2, this query is interpreted as -hello AND world (only hello is negated) In DIALECT 2, You could achieve the default behavior as in dialect 1 by updating your query to -(hello world)

Another example illustrating the differences in parser behavior hello world | "goodbye" moon.

In DIALECT 1, this query is interpreted as searching for (hello world | "goodbye") moon. In DIALECT 2, this query is interpreted as searching for either hello world OR "goodbye" moon

Notes:

This is the first GA version of 2.4. The version inside Redis will be 2.4.3 in semantic versioning. Since the version of a module in Redis is numeric, we could not add a GA flag.

Minimal Redis version: 6.0.0

Source code(tar.gz)
Source code(zip)
v2.2.10(Mar 23, 2022)
This is a maintenance release for RediSearch 2.2

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent.

Details

Improvements:

Mac M1 support

#2645 Ignore NULL values in JSON documents on indexing. (prior behaviour would ignore the entire document)

#2623 Improved Multi Sortby error message for FT.SEARCH

Bug fixes:

#2641 Memory leak in Coordinator

#2651 Client freeze on docs expire during query

#2670 Memory leak in RediSearch found in Active-Active (MOD-2518)

Source code(tar.gz)
Source code(zip)
v2.4.2(Mar 11, 2022)
This is the first Release Candidate of RediSearch 2.4

Headlines:

RediSearch 2.4 introduces a new capability, Vector Similarity Search (VSS), which allows indexing and querying vector data stored (as BLOBs) in Redis hashes.

This release ships an important bug fix addressing operator precedence rules in the RediSearch query syntax. Please see the "Breaking behaviour" section below as you may need to update existing FT.SEARCH queries to assure same behaviour when upgrading.

What's new in 2.4

FT.CREATE is extended to allow creation of 2 types vector indexes:

FLAT Index - This type of index is used when recall is more important than speed of query execution. With a flat index, a query vector will be compared to all vectors in the index and the search results will content the exact Top K Nearest Neighbors.

Hierarchical Navigable Small World (HNSW) Index. A custom implementation of this influential academic paper. An HNSW vector index is used when speed of query execution is preferred over recall. The results returned are Approximate Nearest Neighbors (ANNs). You can try out different HNSW index parameters to improve the “recall vs speed” balance for your dataset

Use FT.SEARCH to retrieve the Top K hashes with most similar vectors to a given query vector

Discover the power of hybrid queries in FT.SEARCH . Use hybrid queries to retrieve redis hashes matching a combination of vector and non-vector search criteria data. Non-vector search criteria can include NUMERIC, TEXT, TAG, GEO fields. Hybrid queries are commonly used in modern e-commerce search applications featuring “visual” similarity + metadata similarity. For example, find products that are visually similar to a given image but also within a price range and available in a geo-location. All in one query.

Details

Features (since 2.4-M01):

#2590 Vecsim coordinator allows executing KNN and hybrid queries on a RediSearch cluster.

Improvements (since 2.4-M01):

#2619 Added and improved hybrid query execution optimisations. An on-demand flat index scan when a small subset of vectors satisfy the query filter. This PR also improves an existing hybrid query execution when a large number of vectors that satisfy query filter.

#2583 Release index specific information off the main thread (performance enhancement)

Bug fixes (since 2.4-M01):

#2641 Memory leak in the coordinator (MOD-2716)

#2623 Multi SORTBY error message for FT.SEARCH

#2596 Operator precedence rules in the RediSearch query syntax. This PR also changes modifier's behaviour to ensure consistency with other logical operators. Please read section "Breaking Behaviour".

Breaking behaviour

Part of this release is aligning the query syntax operator precedence. There were certain situations where the following rules were not respected in the query syntax of RediSearch:

Operator precedence (in order of appearance): 1. AND 2. quotes, ~, -, and % (exact, optional, negation, fuzzy) 3. OR Parentheses can override precedence

(AND in RediSearch is a space and OR is a |)

Your existing queries may be impacted if they fall into any of the following categories:

1. Your query has a field modifier followed by multiple words.

Consequence 1: If a field modifier precedes multiple words or expressions, it applies only to the adjacent expression

Consider this simple query @name:James Brown. Here the field modifier @name is followed by 2 words James and Brown Previously, the query would be interpreted as find James Brown in the @name field.

From this version on, this query will be interpreted as find James in the @name field AND Brown in ANY text field. In other words, it would be interpreted as (@name:James) Brown

In order to keep the previous behaviour, you should update your query to @name:(James Brown)

2. Your query uses quotes, ~, -, % (exact, optional, negation, fuzzy)

Consequence 2: The operators: quotes, `~`, `-`, and `%` (exact, optional, negation, and fuzzy) are not affecting AND'ing and OR'ing of expressions

Consider a simple query with negation -hello world. Previously, the query would be interpreted as find values in any field that do not contain hello AND do not contain world. The equivalent of -(hello world) or -hello -world.

From now on, the original query would be interpreted as -hello AND world (only hello is negated)

In order to keep the previous behaviour, you should update your query to -(hello world)

Another examples is this query hello world | "goodbye" moon. Previously, this query was interpreted as searching for (hello world | "goodbye") moon. This query is now interpreted as searching for either hello world OR "goodbye" moon

Notes:

The version inside Redis will be 2.4.2 in semantic versioning. Since the version of a module in Redis is numeric, we could not add a Release Candidate flag.

Minimal Redis version: 6.0

Source code(tar.gz)
Source code(zip)
v2.2.9(Mar 11, 2022)
This is a maintenance release for RediSearch 2.2

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent.

Details

Improvements:

#2605 Added support for tls-key-file-pass capability (MOD-2086)

#2583 Release index specific information off the main thread (performance enhancement)

Bug fixes:

#2436 When indexing JSON documents filters cause no documents to be indexed (MOD-2214)

#2507 QUANTILE aggregation function outputting wrong values (MOD-2432)

#2521 contains() with an empty string argument leaves Redis hanging at CPU 100% indefinitely (MOD-2428)

#2560 Free prefix and cursor efficiently for cases with many indices (MOD-2080)

#2541 Numeric types for FT.INFO on coordinator

#2553 Fix union high iterator

#2404 Update coordination strategy of FlatSearchCommandHandler

Source code(tar.gz)
Source code(zip)
v2.4.0(Feb 23, 2022)
This is the first Milestone of RediSearch 2.4

Highlights

This milestone release introduces a new Vector Similarity Search (VSS) capability, which allows indexing and querying vector data stored (as BLOBs) in Redis hashes.

Details

FT.CREATE is extended to allow creation of 2 types vector indexes:

FLAT Index This type of index is used when recall is more important than speed of query execution. With a flat index, a query vector will be compared to all vectors in the index and the search results will content the Top K Nearest Neighbors

Hierarchical Navigable Small World (HNSW) Index. A custom implementation of this influential academic paper. An HNSW vector index is used when query execution speed is more important than recall. While it returns approximate nearest neighbors, you can try out different HNSW index parameters to improve the “recall vs speed” balance for your dataset

Use FT.SEARCH to query vector data. FT.SEARCH allows you to retrieve the Top K hashes with most similar vectors to a given query vector

Discover the power of hybrid queries in FT.SEARCH . Use hybrid queries to retrieve redis hashes matching a combination of vector and non-vector data (NUMERIC, TEXT, TAG, GEO). Hybrid queries are commonly used in modern e-commerce search applications featuring “visual” similarity + metadata similarity. For example, find products that are visually similar to a given image but also within a price range and available in a geo-location. All in one query.

To learn more:

Check out this blog post Building Intelligent Apps with New Vector Similarity Search

Try out this hands on demo showcasing semantic similarity on Financial news headlines

Notes:

The version inside Redis will be 2.4.0 in semantic versioning. Since the version of a module in Redis is numeric, we could not add a Milestone flag.

Minimal Redis version: 6.2

Source code(tar.gz)
Source code(zip)
v2.2.7(Feb 6, 2022)
This is a maintenance release for RediSearch 2.2

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent.

Details

Bug fixes:

#2466 FT.PROFILE for FT.AGGREGATE on clustered databases

#2473 FT.PROFILE with no result processor present

#2490 Case sensitivity issue in searches of TAG field on JSON

Improvements:

#2469 Add API for TAG children query nodes

Source code(tar.gz)
Source code(zip)
v2.2.6(Jan 9, 2022)
This is a maintenance release for RediSearch 2.2.

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent.

Details:

Bug fixes:

#2362 Crash on empty field name

#2407 Inconsistency with FT.CREATE and ON: return error for spec w/o fields parameters

#2392 Remove limit on Geo field percision

#2440 Fix in NOT(-) iterator

#2414 Prevent server freeze when FT.SEARCH timeout while sorting

#2386 Memory leak in numeric field index

Improvements:

#2424, #2425 Performance improvements for numeric indices

#2408 API: Added RediSearch_IndexInfo

#2448 API: Added setLang and setScore

Source code(tar.gz)
Source code(zip)
v2.0.15(Dec 14, 2021)
This is a maintenance release for RediSearch 2.0.

Update urgency: MODERATE - Program an upgrade of the server, but it's not urgent.

Details:

Bug fixes:

#2388 Garbage Collection (GC) for empty ranges in numeric index

#2409 Introduction of FORK_GC_CLEAN_NUMERIC_EMPTY_NODES module argument to enable #2388 (off by default)

#325 Used Redis allocator in hiredis (RSCoordinator)

#2362 Crash on empty field name

Source code(tar.gz)
Source code(zip)
v2.2.5(Nov 16, 2021)
This is the General Availability release of RediSearch 2.2.

Headlines

Searching and indexing JSON documents

This release introduces the ability to index, query, and full-text search JSON documents using JSONPath queries.

On the schema creation FT.CREATE, it is now possible to map a JSONPath query with a field. When a JSON document is indexed, the value extracted by the JSONPath query is indexed in the given field.

This features require the module RedisJSON 2.0 to be installed.

Profiling queries

With the new FT.PROFILE command it is now possible to profile in detail the execution time of several internal steps involved in the execution of FT.SEARCH and FT.AGGREGATE. That way, it is possible to understand which part of the query is taking most of the resources.

Field aliasing

With the support of JSON document indexing, It is now possible to map a JSONPath query to an alias. Therefore, it is possible to index the same value in different fields with different indexing strategies.

Details

Enhancements

Add support for redis COPY command (#2337)

Add LOAD * for FT.AGGREGATE (#2243)

Add multi value recursive decent tag (#2207)

Add UNF flag for SORTABLE fields (#2188)

LLAPI getter functions for score, language and stopwords list (#2184)

JSON array can be stored in a TAG field (#2133)

Improve FT.INFO complexity to O(1) (#2153)

Add CASESENSITIVE to TAG fields (#2138)

FT.INFO has identifier and attribute for fields (#2137)

Bug fixes

Fix score field for JSON (#2341)

Fix escaping for tags (#2325)

Remove empty tag values (#2269)

Replace NULL with empty iterator for child of negative iterator (#2223)

Update field limit on tags (#2215)

Partial JSON documents are not indexed (#2143)

Field loaded with 'AS' cant be used by funcs (#2109)

Notes: This is the first GA version of 2.2. The version inside Redis will be 2.2.5 in semantic versioning. Since the version of a module in Redis is numeric, we could not add a GA flag.
Source code(tar.gz)
Source code(zip)
v2.0.13(Nov 8, 2021)
This is a maintenance release for RediSearch 2.0.

Update urgency: MODERATE: Program an upgrade of the server, but it's not urgent.

Details

Bug fixes:

#2269 #2291 Remove TAG values from trie with no entries on Garbage Collection.

#2287 Uninitialized read on FT.ADD

#2342 Check for NULL result on intersect iterator

#2350 Crash on FT.AGGREGATE with LIMIT 0 0

Improvements:

#2243 Add LOAD * for FT.AGGREGATE which will load all fields

Source code(tar.gz)
Source code(zip)
v2.0.12(Sep 8, 2021)
This is a maintenance release for version 2.0. Update urgency: MODERATE - Program an upgrade of the server, but it's not urgent.

Details:

Enhancements:

#2184 API: getter functions for score, language and stop words list

#2188 Introduced the UNF parameter to SORTABLE to disable normalisation on TAG/TEXT fields

#2218 API: added RediSearch_CreateDocument2

Bug fix:

#2153 Restore FT.INFO complexity to O(1)

#2203 FT.AGGREGATE returns inaccurate results when TAG field is not set in hash

Source code(tar.gz)
Source code(zip)
v2.0.11(Aug 9, 2021)
This is a maintenance release for version 2.0. Update urgency: LOW - No need to upgrade unless there are new features you want to use.

Headlines: This release improves overall stability and provides fixes for issues found after the previous release.

Details:

Enhancements:

#2156 TAG fields can now be case sensitive using the CASESENSITIVE parameter

#2113 An already existing document that can't be updated is removed from the index, (JIRA MOD-1266)

Enhancements on RSCoordinator:

#267 #287 Updated Hiredis to support Intershard TLS

Bug Fixes:

#2117 (#2115) Fix crash on coordinator on first value reducer

Source code(tar.gz)
Source code(zip)
v2.0.10(Jul 7, 2021)
This is a maintenance release for version 2.0. Update urgency: HIGH - There is a critical bug that may affect a subset of users. Upgrade!

Details:

Enhancements:

#2025 Performance improvement on numeric range search

#1958 #2033 Support of sortable on the GEO type

#2079 Update to Snowball 2.1.0, adds Armenian, Serbian and Yiddish stemming support

#2002 Add stopwords list support in the API

Bug fix:

#2045 Possible crash when loading an RDB file (silently ignore double load of alias)

#1994 Skip intersect iterator qsort if INORDER flag is used

Bug fix on RSCoordinator:

#257 Switch coordinator to send _FT.CURSOR instead FT.CURSOR to prevent data access without holding the lock

Source code(tar.gz)
Source code(zip)
v2.2.0(Jul 7, 2021)
This is the first release candidate for RediSearch 2.2

Highlights

This release adds the following main feature: Indexing, querying and full-text search of JSON documents

Until now, RediSearch has been providing indexing and search capabilities on hashes. Going forward, by exposing its capabilities to other modules, RedisJSON gives RedisSearch the possibility to index JSON documents.

Another feature part of this release is the ability to profile queries (FT.PROFILE). This feature will increase the developer experience by empowering them to understand the query execution of RediSearch queries and allow them to optimise them.

Lean more by reading the blog post: Indexing, querying and full-text search of JSON documents with Redis

Getting started:

The easiest way to discover these new features is by using this Docker image. The docker image is publically available and contains Redis together with the main Redis modules, including RediSearch and RedisJSON

docker run -p 6379:6379 redislabs/redismod:preview

For more details read the dedicated page about JSON indexing.

Notes:

Requires Redis v6 or above

Requires RediJSON 2.0-RC1 for JSON support

This is not the GA version of 2.2. The version inside Redis will be 20200 or 2.2.0 in semantic versioning. Since the version of a module in Redis is numeric, we could not add an RC1 flag.

For production usage, please use RediSearch 2.0.

Source code(tar.gz)
Source code(zip)
v1.6.16(Jun 21, 2021)
This is a maintenance release for version 1.6. Update urgency: MODERATE - Program an upgrade of the server, but it's not urgent.

Details:

Bug fix:

#2018: FT.ADD REPLACE leaves fields from the previous document that are not included in the new document #647 #1193

Source code(tar.gz)
Source code(zip)
v2.0.9(May 13, 2021)
This is a maintenance release for version 2.0. Update urgency: SKIP - There are new critical issues introduced by this release. Skip or Upgrade ASAP.

Known issue: This release contains a major bug when it is used with RSCoordinator #259 #275. In this case we recommend to skip this version and stick to v2.0.8. This issue will soon be fixed in v2.0.10.

Details:

Bug fix on RSCoordinator:

#259: Fix deadlock on cursor read by performing cursor command on background thread

Other changes:

Build fixes

Documentation fixes

Source code(tar.gz)
Source code(zip)
v2.0.7(May 5, 2021)
This is a maintenance release for version 2.0 Update urgency: Medium

Details:

Major enhancements:

#1864 Improve query time by predetermining reply array length.

#1879 Improve loading time by calling RM_ScanKey instead of RM_Call

Major bugfix:

#1866 Fix a linking issue causing incompatibility with Redis 6.2.0.

#1842 #1852 Fix macOS build.

Minor bugfixes:

#1850 Fix a race condition on drop temporary index.

#1855 Fix a binary payload corruption.

#1876 Fix crash if the depth of the reply array is larger than 7.

#1843 #1860 Fix low-level API issues.

Various documentation updates.

Various small tweaks under the hood.

Source code(tar.gz)
Source code(zip)