How to Configure Logging Server with Grafana and qryn

STEP 1: Install Clickhouse

STEP 2: Install Node.js

STEP 3: Install qryn

STEP 4: Install grafana-server

apt-get install -y apt-transport-https software-properties-common wget
mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Updates the list of available packages

sudo apt-get update

Installs the latest OSS release:

sudo apt-get install grafana
systemctl daemon-reload
systemctl enable grafana-server
systemctl start grafana-server

Access Grafana UI with the following URL

http://localhost:3000

Username: admin

Password: admin

Add Loki Plugin and configure it with qryn-server

Open Menu --> Admisinstration --> Plugins --> Search "Loki" --> Install

Now add Loki as data source

Open Menu --> Connections --> Data Source --> Click "Add New Data Source" --> Select "Loki"

URL: http://qryn-server:3100

Basic Auth

Username: default

Password: Clickpass

Click "Save and test"

Now to go Open Menu --> Explore and try Searching Logs

Make sure you have also configured grafana-agent / vector to send logs to qryn.

How to install qryn

STEP 1: Install Clickhouse

STEP 2: Install Node.js

STEP 3: Install qryn

go to the directory

cd /usr/local/
git clone https://github.com/metrico/qryn && cd qryn
npm install

Create a Unit file

vim /etc/systemd/system/qryn.service

copy the following in the qryn.service add ENVs as required.

[Unit]
Description=Captures packets from wire and sends them to Hepic
After=network.target

[Service]
WorkingDirectory=/usr/local/qryn/
Environment="CLICKHOUSE_SERVER=127.0.0.1"
Environment="CLICKHOUSE_AUTH=default:clickpass"
Environment="CLICKHOUSE_DB=qryn"
ExecStart=node /usr/local/qryn/qryn_node.js
ExecStop=/bin/kill ${MAINPID}
Restart=on-failure
RestartSec=10s
Type=simple

[Install]
WantedBy=multi-user.target

start qryn

systemctl start qryn

Check status

systemctl status qryn
● qryn.service - Captures packets from wire and sends them to Hepic
     Loaded: loaded (/etc/systemd/system/qryn.service; disabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-12-27 19:41:17 PKT; 4s ago
   Main PID: 1186 (node)
      Tasks: 12 (limit: 2323)
     Memory: 172.7M
        CPU: 1.976s
     CGroup: /system.slice/qryn.service
             └─1186 node /usr/local/qryn/qryn_node.js

Dec 27 19:41:20 qryn-server node[1186]: {"level":30,"time":1703688080795,"pid":1186,"hostname":"qryn-server","name":"qryn","msg":"Initializing DB... qryn"}
Dec 27 19:41:20 qryn-server node[1186]: {"level":30,"time":1703688080390,"pid":1186,"hostname":"qryn-server","name":"qryn","msg":"xxh ready"}
Dec 27 19:41:20 qryn-server node[1186]: {"level":30,"time":1703688080860,"pid":1186,"hostname":"qryn-server","name":"qryn","msg":"xxh ready"}
Dec 27 19:41:21 qryn-server node[1186]: {"level":30,"time":1703688081019,"pid":1186,"hostname":"qryn-server","name":"qryn","msg":"Checking clickhouse capabilities"}
Dec 27 19:41:21 qryn-server node[1186]: {"level":30,"time":1703688081039,"pid":1186,"hostname":"qryn-server","name":"qryn","msg":"LIVE VIEW: unsupported"}
Dec 27 19:41:21 qryn-server node[1186]: {"level":30,"time":1703688081044,"pid":1186,"hostname":"qryn-server","name":"qryn","msg":"checking old samples support: samples_v2"}
Dec 27 19:41:21 qryn-server node[1186]: {"level":30,"time":1703688081052,"pid":1186,"hostname":"qryn-server","name":"qryn","msg":"checking old samples support: samples"}
Dec 27 19:41:21 qryn-server node[1186]: {"level":30,"time":1703688081374,"pid":1186,"hostname":"qryn-server","name":"qryn","msg":"Server listening at http://0.0.0.0:3100"}
Dec 27 19:41:21 qryn-server node[1186]: {"level":30,"time":1703688081375,"pid":1186,"hostname":"qryn-server","name":"qryn","msg":"Qryn API up"}
Dec 27 19:41:21 qryn-server node[1186]: {"level":30,"time":1703688081375,"pid":1186,"hostname":"qryn-server","name":"qryn","msg":"Qryn API listening on http://0.0.0.0:3100"}

qryn will start listening on port 3100.

try accessing qryn-view from the browser

http://qryn.hbvoice.local:3100

Dont forget to configure grafana-agent / vector to send logs to qryn.

Enjoy ;)

How to install Node

installs nvm (Node Version Manager)
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.0/install.sh | bash

download and install Node.js (you may need to restart the terminal)
nvm install 22

verifies the right Node.js version is in the environment
node -v # should print `v22.12.0`

verifies the right npm version is in the environment
npm -v # should print `10.9.0`

How to install Clickhouse Server

This installation procedure is for Debian/Ubuntu

apt install curl
curl https://clickhouse.com/ | sh

this will download a clickhouse file, you need to run the following command to install

sudo ./clickhouse install

it will ask the following questions

Enter password for default user: 
Allow server to accept connections from the network (default is localhost only), [y/N]: 

and prompt with this

ClickHouse has been successfully installed. Start clickhouse-server with:

sudo clickhouse start

Start clickhouse-client with:

clickhouse-client --password

Now create a unit file

vim /etc/systemd/system/clickhouse-server.service

and add the following content

[Unit]
Description=ClickHouse Server (analytic DBMS for big data)
Requires=network-online.target
# NOTE: that After/Wants=time-sync.target is not enough, you need to ensure
# that the time was adjusted already, if you use systemd-timesyncd you are
# safe, but if you use ntp or some other daemon, you should configure it
# additionaly.
After=time-sync.target network-online.target
Wants=time-sync.target

[Service]
Type=notify

# NOTE: we leave clickhouse watchdog process enabled to be able to see OOM/SIGKILL traces in clickhouse-server.log files.
# If you wish to disable the watchdog and rely on systemd logs just add "Environment=CLICKHOUSE_WATCHDOG_ENABLE=0" line.
User=clickhouse
Group=clickhouse
Restart=always
RestartSec=30
# Since ClickHouse is systemd aware default 1m30sec may not be enough
TimeoutStartSec=0
# %p is resolved to the systemd unit name
RuntimeDirectory=%p
ExecStart=/usr/bin/clickhouse-server --config=/etc/clickhouse-server/config.xml --pid-file=%t/%p/%p.pid
# Minus means that this file is optional.
EnvironmentFile=-/etc/default/%p
# Bring back /etc/default/clickhouse for backward compatibility
EnvironmentFile=-/etc/default/clickhouse
LimitCORE=infinity
LimitNOFILE=500000
CapabilityBoundingSet=CAP_NET_ADMIN CAP_IPC_LOCK CAP_SYS_NICE CAP_NET_BIND_SERVICE

[Install]
# ClickHouse should not start from the rescue shell (rescue.target).
WantedBy=multi-user.target

Start clickhouse server

systemctl restart clickhouse-server

Check Status

$ systemctl status clickhouse-server

● clickhouse-server.service - ClickHouse Server (analytic DBMS for big data)
     Loaded: loaded (/etc/systemd/system/clickhouse-server.service; disabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-12-27 14:01:00 PKT; 11s ago
   Main PID: 4548 (clickhouse-serv)
      Tasks: 673 (limit: 2323)
     Memory: 225.2M
        CPU: 956ms
     CGroup: /system.slice/clickhouse-server.service
             ├─4547 clickhouse-watchdog        --config=/etc/clickhouse-server/config.xml --pid-file=/run/clickhouse-server/clickho>
             └─4548 /usr/bin/clickhouse-server --config=/etc/clickhouse-server/config.xml --pid-file=/run/clickhouse-server/clickho>

Dec 27 14:00:59 qryn-server clickhouse-server[4548]: Merging configuration file '/etc/clickhouse-server/config.d/data-paths.xml'.
Dec 27 14:00:59 qryn-server clickhouse-server[4548]: Merging configuration file '/etc/clickhouse-server/config.d/listen.xml'.
Dec 27 14:00:59 qryn-server clickhouse-server[4548]: Merging configuration file '/etc/clickhouse-server/config.d/logger.xml'.
Dec 27 14:00:59 qryn-server clickhouse-server[4548]: Merging configuration file '/etc/clickhouse-server/config.d/openssl.xml'.
Dec 27 14:00:59 qryn-server clickhouse-server[4548]: Merging configuration file '/etc/clickhouse-server/config.d/user-directories.x>
Dec 27 14:00:59 qryn-server clickhouse-server[4548]: Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/>
Dec 27 14:00:59 qryn-server clickhouse-server[4548]: Processing configuration file '/etc/clickhouse-server/users.xml'.
Dec 27 14:00:59 qryn-server clickhouse-server[4548]: Merging configuration file '/etc/clickhouse-server/users.d/default-password.xm>
Dec 27 14:00:59 qryn-server clickhouse-server[4548]: Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/>
Dec 27 14:01:00 qryn-server systemd[1]: Started ClickHouse Server (analytic DBMS for big data).

Connect to clickhouse-server with the following command

$ clickhouse-client -u default --password
ClickHouse client version 23.12.1.1214 (official build).
Password for user (default):
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 23.12.1.

Warnings:
 * Linux transparent hugepages are set to "always". Check /sys/kernel/mm/transparent_hugepage/enabled
 * Linux threads max count is too low. Check /proc/sys/kernel/threads-max
 * Available memory at server startup is too low (2GiB).
 * Maximum number of threads is lower than 30000. There could be problems with handling a lot of simultaneous queries.

qryn-server :)

Enjoy ;)

Make Your First API in Golang

Install Go

cd /usr/local/src/
wget https://go.dev/dl/go1.21.5.linux-amd64.tar.gz
rm -rf /usr/local/go && tar -C /usr/local -xzf go1.21.5.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin
cd 
go version

Start Writing API

mkdir /usr/local/hbapp/
cd /usr/local/hbapp/
vim main.go

copy the code

package main

import (
	"database/sql"
	"fmt"
	"log"
	"net/http"

	_ "github.com/go-sql-driver/mysql" // Import MySQL driver
	"github.com/gorilla/mux"
)

var db *sql.DB

func init() {
	var err error
	// Connect to your database
	db, err = sql.Open("mysql", "username:password@tcp(localhost:3306)/your_database")
	if err != nil {
		log.Fatal(err)
	}

	// Check if the connection is successful
	err = db.Ping()
	if err != nil {
		log.Fatal(err)
	}
}

func main() {
	r := mux.NewRouter()

	// Define your API endpoint
	r.HandleFunc("/check/{input}", CheckExistence).Methods("GET")

	// Start the HTTP server
	log.Fatal(http.ListenAndServe(":8080", r))
}

// CheckExistence is the handler for the API endpoint
func CheckExistence(w http.ResponseWriter, r *http.Request) {
	// Get the input parameter from the URL
	vars := mux.Vars(r)
	input := vars["input"]

	// Call a function to check if the input exists in the database
	exists, err := checkDatabase(input)
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	// Return the result as JSON
	if exists {
		fmt.Fprint(w, `{"exists": true}`)
	} else {
		fmt.Fprint(w, `{"exists": false}`)
	}
}

// checkDatabase is a function to check if the input exists in the database
func checkDatabase(input string) (bool, error) {
	// Perform a query to check existence in your database
	// Replace "your_table" and "your_column" with your actual table and column names
	query := "SELECT COUNT(*) FROM your_table WHERE your_column = ?"
	var count int
	err := db.QueryRow(query, input).Scan(&count)
	if err != nil {
		return false, err
	}

	// If count is greater than 0, the input exists in the database
	return count > 0, nil
}

Install go modules

go install github.com/gorilla/mux@latest
go install github.com/go-sql-driver/mysql@latest

Compile your code

go mod init hbapp
go mod tidy
go build

Run the service with the following command

./hbapp

use the following command to test your API

curl http://your-ip:8080/check/{input}

Enjoy ;)