Exception in thread "main" java.lang.NoSuchMethodError: kotlinx.coroutines.AbstractCoroutine.(Lkotlin/coroutines/CoroutineContext;ZZ)V at kotlinx.coroutines.future.CompletableFutureCoroutine.(Future.kt:51) at kotlinx.coroutines.future.FutureKt.future(Future.kt:42) at kotlinx.coroutines.future.FutureKt.future$default(Future.kt:34) at io.hstream.impl.UtilsKt.futureForIO(Utils.kt:124) at io.hstream.impl.UtilsKt.futureForIO$default(Utils.kt:123) at io.hstream.impl.UtilsKt.unaryCallWithCurrentUrlsAsync(Utils.kt:115) at io.hstream.impl.UtilsKt.unaryCallWithCurrentUrls(Utils.kt:119) at io.hstream.impl.HStreamClientKtImpl.(HStreamClientKtImpl.kt:53) at io.hstream.impl.HStreamClientBuilderImpl.build(HStreamClientBuilderImpl.java:74) at MainClass.main(MainClass.java:6)

Process finished with exit code 1

CODE:

import io.hstream.*;

public class MainClass { public static void main(String[] args) throws Exception { final String serviceUrl = "127.0.0.1:6570"; HStreamClient client = HStreamClient.builder().serviceUrl(serviceUrl).build(); System.out.println("Connected"); } }

#quick-start.yaml

version: "3.5"

services: hserver0: image: hstreamdb/hstream:v0.9.3 depends_on: - zookeeper - hstore ports: - "127.0.0.1:6570:6570" expose: - 6570 networks: - hstream-quickstart volumes: - /var/run/docker.sock:/var/run/docker.sock - /tmp:/tmp - data_store:/data/store command: - bash - "-c" - | set -e /usr/local/script/wait-for-storage.sh hstore 6440 zookeeper 2181 600
/usr/local/bin/hstream-server
--host 0.0.0.0 --port 6570
--internal-port 6571
--server-id 100
--seed-nodes "$$(hostname -I | awk '{print $$1}'):6571,hserver1:6573"
--address $$(hostname -I | awk '{print $$1}')
--zkuri zookeeper:2181
--store-config /data/store/logdevice.conf
--store-admin-host hstore --store-admin-port 6440
--io-tasks-path /tmp/io/tasks
--io-tasks-network hstream-quickstart

hserver1: image: hstreamdb/hstream:v0.9.3 depends_on: - zookeeper - hstore ports: - "127.0.0.1:6572:6572" expose: - 6572 networks: - hstream-quickstart volumes: - /var/run/docker.sock:/var/run/docker.sock - /tmp:/tmp - data_store:/data/store command: - bash - "-c" - | set -e /usr/local/script/wait-for-storage.sh hstore 6440 zookeeper 2181 600
/usr/local/bin/hstream-server
--host 0.0.0.0 --port 6572
--internal-port 6573
--server-id 101
--seed-nodes "hserver0:6571,$$(hostname -I | awk '{print $$1}'):6573"
--address $$(hostname -I | awk '{print $$1}')
--zkuri zookeeper:2181
--store-config /data/store/logdevice.conf
--store-admin-host hstore --store-admin-port 6440
--io-tasks-path /tmp/io/tasks
--io-tasks-network hstream-quickstart

hserver-init: image: hstreamdb/hstream:v0.9.3 depends_on: - hserver0 - hserver1 networks: - hstream-quickstart command: - bash - "-c" - | timeout=60 until (
/usr/local/bin/hadmin server --host hserver0 --port 6570 status &&
/usr/local/bin/hadmin server --host hserver1 --port 6572 status
) >/dev/null 2>&1; do >&2 echo 'Waiting for servers ...' sleep 1 timeout=$$((timeout - 1)) [ $$timeout -le 0 ] && echo 'Timeout!' && exit 1; done;
/usr/local/bin/hadmin server --host hserver0 init

hstore: image: hstreamdb/hstream:v0.9.3 networks: - hstream-quickstart volumes: - data_store:/data/store command: - bash - "-c" - | set -ex /usr/local/bin/ld-dev-cluster --root /data/store
--use-tcp --tcp-host $$(hostname -I | awk '{print $$1}')
--user-admin-port 6440
--no-interactive

zookeeper: image: zookeeper expose: - 2181 networks: - hstream-quickstart volumes: - data_zk_data:/data - data_zk_datalog:/datalog

networks: hstream-quickstart: name: hstream-quickstart

volumes: data_store: name: quickstart_data_store data_zk_data: name: quickstart_data_zk_data data_zk_datalog: name: quickstart_data_zk_datalog

PR Description

Type of change

• [x] New feature
• [x] Documentation updates required

Summary of the change and which issue is fixed

Main changes: multi-line mode for sql, for edit and update

      __  _________________  _________    __  ___
     / / / / ___/_  __/ __ \/ ____/   |  /  |/  /
    / /_/ /\__ \ / / / /_/ / __/ / /| | / /|_/ /
   / __  /___/ // / / _, _/ /___/ ___ |/ /  / /
  /_/ /_//____//_/ /_/ |_/_____/_/  |_/_/  /_/
  

Command
  :h                           To show these help info
  :q                           To exit command line interface
  :help [sql_operation]        To show full usage of sql statement
  \{                           To enter multi-line mode

SQL STATEMENTS:
  To create a simplest stream:
    CREATE STREAM stream_name;

  To create a query select all fields from a stream:
    SELECT * FROM stream_name EMIT CHANGES;

  To insert values to a stream:
    INSERT INTO stream_name (field1, field2) VALUES (1, 2);
  
> \{
-- Entering multi-line mode. Press <Ctrl-D> to finish.
\ select
\ *
\ from
\ s
\ emit
\ changes
\ ;
\ 
^CTerminated
> 

Checklist

• I have run format.sh under script
• I have commented my code, particularly in hard-to-understand areas
• New and existing unit tests pass locally with my changes

[email protected]:/# hadmin server status --host 172.16.3.179 --port 6570 +---------+---------+-------------------+ | node_id | state | address | +---------+---------+-------------------+ | 1 | Running | 172.16.3.179:6570 | | 2 | Running | 172.16.3.181:6570 | | 3 | Running | 172.16.3.182:6570 | +---------+---------+-------------------+ [email protected]:/# hadmin store status +----+---------+----------+-------+-----------+----------+---------+-------------+---------------+------------+---------------+ | ID | NAME | PACKAGE | STATE | UPTIME | LOCATION | SEQ. | DATA HEALTH | STORAGE STATE | SHARD OP. | HEALTH STATUS | +----+---------+----------+-------+-----------+----------+---------+-------------+---------------+------------+---------------+ | 0 | store-0 | 99.99.99 | ALIVE | 8 min ago | | ENABLED | HEALTHY(1) | READ_WRITE(1) | ENABLED(1) | HEALTHY | | 1 | store-1 | 99.99.99 | ALIVE | 8 min ago | | ENABLED | HEALTHY(1) | READ_WRITE(1) | ENABLED(1) | HEALTHY | | 2 | store-2 | 99.99.99 | ALIVE | 8 min ago | | ENABLED | HEALTHY(1) | READ_WRITE(1) | ENABLED(1) | HEALTHY | +----+---------+----------+-------+-----------+----------+---------+-------------+---------------+------------+---------------+

show streams; Succeeded. No results. create stream demo; demo INSERT INTO demo (temperature, humidity) VALUES (22, 80); Failed to get any available server.

According to the manual docker deployment, the cluster seems to be normal, but an error is reported when inserting data? Why?

PR Description

Type of change

• [x] New feature

Summary of the change and which issue is fixed

Main changes: dev-deploy: add docker node-exporter

Checklist

• I have run format.sh under script
• I have commented my code, particularly in hard-to-understand areas
• New and existing unit tests pass locally with my changes

PR Description

Type of change

• [x] New feature

Main changes: add cfg for memory and cpus

Checklist

• I have run format.sh under script
• I have commented my code, particularly in hard-to-understand areas
• New and existing unit tests pass locally with my changes

PR Description

Type of change

• [x] New feature

Summary of the change and which issue is fixed

Main changes:

> select
| *
| from
| s
| emit
| changes
| ;
^CTerminated
> show streams
| ;
+----------------------+---------+----------------+-------------+
| Stream Name          | Replica | Retention Time | Shard Count |
+----------------------+---------+----------------+-------------+
| juqlTVOFCAvGfHHMIPqE | 1       | 0sec           | 1           |
+----------------------+---------+----------------+-------------+
| ZSHhwSTZAiACReNLwkfQ | 1       | 0sec           | 1           |
+----------------------+---------+----------------+-------------+
| aKyGZDCkdeUmFIBamDtW | 1       | 0sec           | 1           |
+----------------------+---------+----------------+-------------+
| TgDNyMLAxEsbkKVMEuGx | 1       | 0sec           | 1           |
+----------------------+---------+----------------+-------------+
| DPpClhNohpBXbYYTQYjZ | 1       | 0sec           | 1           |
+----------------------+---------+----------------+-------------+
| IuAYjTuvkjdwEURatdcj | 1       | 0sec           | 1           |
+----------------------+---------+----------------+-------------+
| WHvdvbOSNAYZNRGBqhSK | 1       | 0sec           | 1           |
+----------------------+---------+----------------+-------------+
| s                    | 3       | 0sec           | 1           |
+----------------------+---------+----------------+-------------+
> show stream
Parse exception at <line 1,column 6>: syntax error before `STREAM'.
> 

Checklist

• I have run format.sh under script
• I have commented my code, particularly in hard-to-understand areas
• New and existing unit tests pass locally with my changes

I manually deployed docker according to the document, and when I wrote multiple data tests through hstram-java, I found that the writing speed was relatively slow, only less than 1,000 lines per second. This speed is too slow. I don’t know if it is my test program method. Is it wrong or what is the reason, are there any instructions or examples for stress testing?

Test program, write 1 million pieces of data:

package com.evoc.hstream; import java.util.ArrayList; import java.util.Date; import java.util.List; import java.util.Random; import java.util.concurrent.CompletableFuture; import java.util.concurrent.ExecutionException;

import io.hstream.BufferedProducer; import io.hstream.HRecord; import io.hstream.HStreamClient; import io.hstream.Producer; import io.hstream.Record; import io.hstream.RecordId;

//@SpringBootApplication public class HstreamTestApplication {

public static void main(String[] args) {

// SpringApplication.run(HstreamTestApplication.class, args); try { final String serviceUrl = "172.16.3.179:6570,172.16.3.181:6570,172.16.3.182:6570"; // final String serviceUrl = "172.16.3.221:6570"; HStreamClient client = HStreamClient.builder().serviceUrl(serviceUrl).build(); //Producer producer = client.newProducer().stream("demo").build(); BufferedProducer producer = client.newBufferedProducer() .stream("demo2") .recordCountLimit(100) .flushIntervalMs(10) .maxBytesSize(819200) .build(); Random random = new Random(); List list = new ArrayList(); for (int i = 0; i < 1000000; i++) { long time = new Date().getTime(); int id= random.nextInt(1000000); HRecord hRecord = HRecord.newBuilder() .put("id",id) .put("tt",time) .put("test", 10) .build(); list.add(hRecord); } list.parallelStream().forEach(e->{ synchronized (HstreamTestApplication.class) { Record record = Record.newBuilder().hRecord(e).build(); // producer.write(record); CompletableFuture future = producer.write(record); try { future.get().getBatchId(); } catch (InterruptedException e1) { // TODO Auto-generated catch block e1.printStackTrace(); } catch (ExecutionException e1) { // TODO Auto-generated catch block e1.printStackTrace(); } } }); producer.close(); client.close(); } catch (Exception e1) { e1.printStackTrace(); } } }

Signed-off-by: Alex Chi [email protected]

Pull Request Template

Description

Not everyone has SSH key added to their local machine. For most cases, developers simply run git clone with only HTTPS credentials or no credentials. Therefore, this PR changes submodule to HTTPS upstream instead of the SSH ones.

Type of change

Please delete options that are not relevant.

• [ ] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [x] This change requires a documentation update

How Has This Been Tested?

~/Work/hstream/external/gRPC-haskell
$ git remote get-url origin
https://github.com/hstreamdb/gRPC-haskell.git

Checklist:

Must:

• [x] I have run format.sh under script
• [x] I have performed a self-review of my own code
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] New and existing unit tests pass locally with my changes

Semi-Must

• [x] I have added new tests that prove my fix is effective or that my feature works
• [x] I have checked my code and corrected any misspellings

Optional:

• [x] My code follows the style guidelines of this project
• [x] I have made corresponding changes to the documentation
• [x] My changes generate no new warnings
• [x] Any dependent changes have been merged and published in downstream modules

创建流，ctl 报错

CREATE STREAM demo WITH (FORMAT = "JSON");

HttpExceptionRequest Request { host = "localhost" port = 6570 secure = False requestHeaders = [("Content-Type","application/json; charset=utf-8")] path = "/create/query" queryString = "" method = "POST" proxy = Nothing rawBody = False redirectCount = 10 responseTimeout = ResponseTimeoutDefault requestVersion = HTTP/1.1 proxySecureMode = ProxySecureWithConnect } (ConnectionFailure Network.Socket.connect: <socket: 12>: does not exist (Connection refused))

HStream-Server 报错 hstream-server: LOGS_SECTION_MISSING {name:LOGS_SECTION_MISSING, description:LOGS_SECTION_MISSING: Configuration file misses logs section, callstack:CallStack (from HasCallStack): throwStreamErrorIfNotOK', called at ./HStream/Store/Exception.hs:324:28 in hstream-store-0.1.0.0-00fd332cb31e900c02128ac35e4d993df94b7e2480f7781981c70e854378ff8e:HStream.Store.Exception}

PR Description

Type of change

• [x] New feature

Summary of the change and which issue is fixed

Main changes: upload store configuration to remote host

Checklist

• I have run format.sh under script
• I have commented my code, particularly in hard-to-understand areas
• New and existing unit tests pass locally with my changes

hstreamdb/hstream:v0.6.0 deploy failure

CentOS7, kernel 5.4.163-1.el7.elrepo.x86_64, Docker 20.10

Deploy a LogDevice cluster: a ld-admin-server and three logdeviced

• zookeeper and ld-admin-server in 10.0.0.2
• the 1st logdeviced in 10.0.0.3
• the 2ed logdeviced in 10.0.0.4
• the 3rd logdeviced in 10.0.0.5

1. run a zookeeper in 10.0.0.2

    docker run
        -e 'ZOO_CONF_DIR=/conf'
        -e 'ZOO_DATA_DIR=/data'
        -e 'ZOO_DATA_LOG_DIR=/datalog'
        -e 'ZOO_LOG_DIR=/logs'
        -e 'ZOO_TICK_TIME=3000'
        -e 'ZOO_INIT_LIMIT=5'
        -e 'ZOO_SYNC_LIMIT=2'
        -e 'ZOO_AUTOPURGE_PURGEINTERVAL=0'
        -e 'ZOO_AUTOPURGE_SNAPRETAINCOUNT=3'
        -e 'ZOO_MAX_CLIENT_CNXNS=60'
        -e 'ZOO_STANDALONE_ENABLED=true'
        -e 'ZOO_ADMINSERVER_ENABLED=true'
        -v /etc/localtime:/etc/localtime:ro
        -p 2181:2181
        --restart always
        --name zookeeper
        -d zookeeper:3.5.6

2. run ld-admin-server in 10.0.0.2

    mkdir -p $HOME/logdevice_conf
    docker run -d --network host --name logdevice_admin
        -v $HOME/logdevice_conf:/etc/logdevice/
        hstreamdb/hstream:v0.6.0 ld-admin-server
        --config-path /etc/logdevice/logdevice.conf
        --admin-port 6440
        --enable-maintenance-manager
        --enable-safety-check-periodic-metadata-update
        --maintenance-log-snapshotting

3.run the 1st logdeviced in 10.0.0.3

    mkdir -p $HOME/logdeviced/{conf,store}
    mkdir -p $HOME/logdeviced/store/shard0
    echo 1 | tee $HOME/logdeviced/store/NSHARDS
    docker run -d --network host --name logdeviced
        -v $HOME/logdeviced/conf:/etc/logdevice/
        -v $HOME/logdeviced/store:/data/logdevice/
        hstreamdb/hstream:v0.6.0 logdeviced
        --config-path /etc/logdevice/logdevice.conf
        --name server-0
        --address 10.0.0.3
        --local-log-store-path /data/logdevice
        --roles storage,sequencer
        --port 4440
        --gossip-port 4441
        --admin-port 6440
        --num-shards 1

4.run the 2nd logdeviced in 10.0.0.4

    mkdir -p $HOME/logdeviced/{conf,store}
    mkdir -p $HOME/logdeviced/store/shard0
    echo 1 | tee $HOME/logdeviced/store/NSHARDS
    docker run -d --network host --name logdeviced
        -v $HOME/logdeviced/conf:/etc/logdevice/
        -v $HOME/logdeviced/store:/data/logdevice/
        hstreamdb/hstream:v0.6.0 logdeviced
        --config-path /etc/logdevice/logdevice.conf
        --name server-1
        --address 10.0.0.4
        --local-log-store-path /data/logdevice
        --roles storage,sequencer
        --port 4440
        --gossip-port 4441
        --admin-port 6440
        --num-shards 1

5.run the 3rd logdeviced in 10.0.0.5

    mkdir -p $HOME/logdeviced/{conf,store}
    mkdir -p $HOME/logdeviced/store/shard0
    echo 1 | tee $HOME/logdeviced/store/NSHARDS
    docker run -d --network host --name logdeviced
        -v $HOME/logdeviced/conf:/etc/logdevice/
        -v $HOME/logdeviced/store:/data/logdevice/
        hstreamdb/hstream:v0.6.0 logdeviced
        --config-path /etc/logdevice/logdevice.conf
        --name server-2
        --address 10.0.0.5
        --local-log-store-path /data/logdevice
        --roles storage,sequencer
        --port 4440
        --gossip-port 4441
        --admin-port 6440
        --num-shards 1

6. logdevice.conf

{
    "cluster": "logdevice-first",
    "server_settings": {
        "enable-node-self-registration": "true",
        "enable-nodes-configuration-manager": "true",
        "enable-cluster-maintenance-state-machine": "true",
        "use-nodes-configuration-manager-nodes-configuration": "true"
    },
    "client_settings": {
        "enable-nodes-configuration-manager": "true",
        "use-nodes-configuration-manager-nodes-configuration": "true",
        "admin-client-capabilities": "true"
    },
    "internal_logs": {
        "config_log_deltas": {
            "replicate_across": {
                "node": 3
            }
        },
        "config_log_snapshots": {
            "replicate_across": {
                "node": 3
            }
        },
        "event_log_deltas": {
            "replicate_across": {
                "node": 3
            }
        },
        "event_log_snapshots": {
            "replicate_across": {
                "node": 3
            }
        },
        "maintenance_log_deltas": {
            "replicate_across": {
                "node": 3
            }
        },
        "maintenance_log_snapshots": {
            "replicate_across": {
                "node": 3
            }
        }
    },
    "metadata_logs": {
        "nodeset": [
            0,1,2
        ],
        "replicate_across": {
          "node": 3
        }
    },
    "zookeeper": {
        "zookeeper_uri": "ip://10.0.0.2:2181",
        "timeout": "30s"
    }
}

7. run the bootstrapping the logdevice

docker run -it --rm \
    --name nodes-config \
    --network host \
    hstreamdb/hstream:v0.6.0 \
    hadmin --host 10.0.0.2 --port 6440 nodes-config bootstrap --metadata-replicate-across node:3 

Successfully bootstrapped the cluster, new nodes configuration version: 7
Took 0.019s

issue #664 and #663 are the same deployment method three logdeviceds cannot do node-self-registration the sequencer cannot be found Similar message: Could not send gossip to WF0:N1:2 (10.0.0.4:4441): CONNFAILED: connection failed. Trying another node. No available sequencer node for log 4611686018427387899. All sequencer nodes are unavailable. Error during sequencer lookup for log 4611686018427387899 (NOSEQUENCER), reporting NOSEQUENCER.