The post Rust: JSON stringify and parse in Node.js appeared first on Justin Silver.
]]>I’m learning Rust for a project to build native modules for use with Node.js and it took me a bit of time to figure out how to parse and stringify JSON using serde_json and neon_serde. With Rust still being a relatively new language, the API of both the language and its crates have shifted over time making some of the older examples no longer useful. The solution I came up with using Rust 1.35 and Neon Bindings 0.2 follows.
[package] name = "rust-json" [lib] name = "rust_json" crate-type = ["dylib"] [build-dependencies] neon-build = "0.2.0" [dependencies] neon = "0.2.0" 'neon-serde' = "0.1.1" serde_json = "1.0.39"
#[macro_use]
extern crate neon;
extern crate neon_serde;
extern crate serde_json;
use neon::prelude::*;
fn stringify(mut cx: FunctionContext) -> JsResult<JsString> {
// get the argument as a JsValue (any json type)
let value = cx.argument::<JsValue>(0)?;
// convert to a serde Value
let object: serde_json::Value = neon_serde::from_value(&mut cx, value)?;
// convert to a String
let string = serde_json::to_string(&object).unwrap();
// return the JsString
Ok(cx.string(string))
}
fn parse(mut cx: FunctionContext) -> JsResult<JsValue> {
// get the argument as a string
let string = cx.argument::<JsString>(0)?;
// convert from serde Value to serde_json Value
let object: serde_json::Value = serde_json::from_str(&string.value()).unwrap();
// now convert to JsValue (any json type)
let value = neon_serde::to_value(&mut cx, &object)?;
/ return the JsValue
Ok(value)
}
register_module!(mut cx, {
cx.export_function("stringify", stringify)?;
cx.export_function("parse", parse)?;
Ok(())
});
Node.js
const native = require('../native');
// more javascript logic here
module.exports = native;
You can now import this module as you would any other node module.
const rust = require('rust-json'); // or however you reference your module
const stringified = rust.stringify({ test: 123 });
console.log('stringified', stringified);
const parsed = rust.parse(stringified);
console.log('parsed', parsed);
The post Rust: JSON stringify and parse in Node.js appeared first on Justin Silver.
]]>The post Syscoin + ZMQ + Node.js = Realtime Blockchain Updates! appeared first on Justin Silver.
]]>You can use the ZMQ topic message queues in Syscoin to receive realtime updates for your application. Using in conjunction with syscoin-core to blockchain-enable your applications in no time.
Make sure to enable the ZMQ listeners in your syscoin.conf file and restart syscoind or Syscoin Core Qt.
# server server=1 daemon=1 # indexes addressindex=1 txindex=1 litemode=0 # rpc rpcuser=u rpcpassword=p rpcport=8370 rpcallowip=127.0.0.1 # zmq listener config zmqpubaliasrecord=tcp://127.0.0.1:3030 zmqpubaliashistory=tcp://127.0.0.1:3030 zmqpubaliastxhistory=tcp://127.0.0.1:3030 zmqpubassetrecord=tcp://127.0.0.1:3030 zmqpubassetallocation=tcp://127.0.0.1:3030 zmqpubassethistory=tcp://127.0.0.1:3030 zmqpubcertrecord=tcp://127.0.0.1:3030 zmqpubcerthistory=tcp://127.0.0.1:3030 zmqpubescrowrecord=tcp://127.0.0.1:3030 zmqpubescrowbid=tcp://127.0.0.1:3030 zmqpubescrowfeedback=tcp://127.0.0.1:3030 zmqpubofferrecord=tcp://127.0.0.1:3030 zmqpubofferhistory=tcp://127.0.0.1:3030 zmqpubhashblock=tcp://127.0.0.1:3030 zmqpubhashtx=tcp://127.0.0.1:3030 zmqpubhashtxlock=tcp://127.0.0.1:3030 zmqpubrawblock=tcp://127.0.0.1:3030 zmqpubrawtx=tcp://127.0.0.1:3030 zmqpubrawtxlock=tcp://127.0.0.1:3030
You will need to npm install the module zeromq.
const zeromq = require('zeromq');
const subscriber = zeromq.socket('sub');
subscriber.on('message', async (topic, message) => {
topic = topic.toString('utf8');
message = message.toString('utf8')
const alias = JSON.parse(message);
console.log(JSON.stringify(alias, null, 2));
});
// connect to message producer
subscriber.connect('tcp://127.0.0.1:3030');
subscriber.subscribe('aliasrecord');
console.log('subscribed to syscoin topic aliasrecord');
Run your script with the following:
> node zmq-client.js
subscribed to syscoin topic aliasrecord
{
"_id": "gldm1",
"address": "SRxK2GjfzTrm8z5PgCtLKzheN5ebd5kN8f",
"expires_on": 1590601936,
"encryption_privatekey": "",
"encryption_publickey": ""
}
{
"_id": "elatte",
"address": "Sd8JMHxtuFVSVJN2V51M27S6MkBBMjgjHY",
"expires_on": 1559077278,
"encryption_privatekey": "",
"encryption_publickey": ""
}
{
"_id": "primitive7",
"address": "Sk7q3kZcttBNVkUwpMXU59yQf9Pco4sAAJ",
"expires_on": 1558656041,
"encryption_privatekey": "",
"encryption_publickey": ""
}
{
"_id": "primitive9",
"address": "SYKff6VzkrzmSn9tL3zZE7FmV2dGFSKfxs",
"expires_on": 1558656041,
"encryption_privatekey": "",
"encryption_publickey": ""
}
// ....
The post Syscoin + ZMQ + Node.js = Realtime Blockchain Updates! appeared first on Justin Silver.
]]>The post Syscoin + Node.js = Blockchain Apps! appeared first on Justin Silver.
]]>You will need to have syscoind or Syscoin Core Qt running on your system and have node/npm installed. If you don’t have Node.js I recommend installing it via nvm.
First make sure that syscoind is configured so that you can connect to the RPC server locally. Use the example below, choosing a secret rcpuser, rpcpassword, and optionally changing the rpcport. You will need to use these values later to connect to query the blockchain using Node.js later.
You don’t really need the ZMQ config for this example, but it shows how to tell Syscoin to listen on those ports should you want to implement a Node.js ZMQ client in your application :).
Before making changes be sure to stop Syscoin by running syscoin-cli stop, update the configuration, then run syscoind to start the process again. If you change the config file before stopping Syscoin it will prevent syscoin-cli from being able to communicate with the RPC server properly. Changing the index values may require you to restart with syscoind -reindex.
# server server=1 daemon=1 # indexes addressindex=1 txindex=1 litemode=0 # rpc rpcuser=u rpcpassword=p rpcport=8370 rpcallowip=127.0.0.1 # zmq listener config zmqpubaliasrecord=tcp://127.0.0.1:3030 zmqpubaliashistory=tcp://127.0.0.1:3030 zmqpubaliastxhistory=tcp://127.0.0.1:3030 zmqpubassetrecord=tcp://127.0.0.1:3030 zmqpubassetallocation=tcp://127.0.0.1:3030 zmqpubassethistory=tcp://127.0.0.1:3030 zmqpubcertrecord=tcp://127.0.0.1:3030 zmqpubcerthistory=tcp://127.0.0.1:3030 zmqpubescrowrecord=tcp://127.0.0.1:3030 zmqpubescrowbid=tcp://127.0.0.1:3030 zmqpubescrowfeedback=tcp://127.0.0.1:3030 zmqpubofferrecord=tcp://127.0.0.1:3030 zmqpubofferhistory=tcp://127.0.0.1:3030 zmqpubhashblock=tcp://127.0.0.1:3030 zmqpubhashtx=tcp://127.0.0.1:3030 zmqpubhashtxlock=tcp://127.0.0.1:3030 zmqpubrawblock=tcp://127.0.0.1:3030 zmqpubrawtx=tcp://127.0.0.1:3030 zmqpubrawtxlock=tcp://127.0.0.1:3030
Next create a new folder for your project and create a Node project by running npm init and answering the questions as you see fit. The only module that is required is syscoin-core, a cutting edge version can be found at my fork.
mkdir -p my-syscoin-app && cd my-syscoin-app npm init # answer init questions npm install -S https://github.com/doublesharp/syscoin-core.git#dev-3.0
Once syscoin-core is installed you can use the following example to get started. Make sure that the port, username, and password match the values set in your syscoin.conf file.
const SyscoinClient = require('@syscoin/syscoin-core');
const syscoin = new SyscoinClient({
host: process.env.SYSCOIND_HOST || 'localhost',
port: process.env.SYSCOIND_PORT || 8370,
username: process.env.SYSCOIND_USER || 'u',
password: process.env.SYSCOIND_PASS || 'p',
timeout: 30000,
});
async function run() {
// prune expired data!
const pruneStart = Date.now();
const prune = await syscoin.pruneSyscoinServices();
console.log('pruned', prune, 'in', Date.now()-pruneStart, 'ms');
// get all alias records
const aliasStart = Date.now();
const list = await syscoin.listAliases(0);
console.log('fetched', list.length, 'aliases in', Date.now()-aliasStart, 'ms');
// get just an array of alias names
const aliases = list.map((alias) => alias._id);
// ...etc
process.exit(0);
}
run();
> node index.js
pruned { services_cleaned: 0 } in 341 ms
fetched 15162 aliases in 1685 ms
You can take this same basic structure and turn it into an Express app, etc, depending on the needs of your application. It’s surprisingly easy to get started building blockchain applications with Syscoin!
The post Syscoin + Node.js = Blockchain Apps! appeared first on Justin Silver.
]]>The post Node.js + PM2 + NGINX + Redis on CentOS 7 appeared first on Justin Silver.
]]>This is a quick setup guide for the application stack I have been using recently. Most of my latest work has been a throwback to the first server-side programming I did in the 90’s – javascript – except this time it’s Node.js instead of Netscape Communication Server. In this setup PM2 is used to manage the Node process running as an arbitrary user, running on an unprivileged port. This means that the application can be restarted without root credentials. The front-end is served by NGINX and it does need to be started as root because it runs on the privileged ports 80 and 443 in this use case. It also gives us a lot of the built in features that Nginx gives you on the front end, like serving all your content over SSL – for free using Let’s Encrypt event. My caching needs are provided by Redis.
Here is the setup – run as root or use sudo.
Enable the firewalld service and only allow http/s traffic to the server – in addition to the default of just ssh.
#!/bin/bash # enable on book systemctl enable firewalld # (re)start the service (service firewalld status > /dev/null && service firewalld restart) || service firewalld start # add the http and http services and reload firewall-cmd --permanent --zone=public --add-service=http firewall-cmd --permanent --zone=public --add-service=https firewall-cmd --reload
Use Letsencrypt for free SSL certificates.
yum -y install letsencrypt openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048
#!/bin/bash
# make sure the YUM_CRON_EMAIL is set
if [[ -z $YUM_CRON_EMAIL ]]; then
echo "You must specify an email using \$YUM_CRON_EMAIL";
else
# install and enable, plus patch for bug fixing
yum -y install yum-cron patch
chkconfig yum-cron on
# configure via sed replacements
sed -i "s|^email_to = root|email_to = ${YUM_CRON_EMAIL}|" /etc/yum/yum-cron.conf
sed -i 's|^update_messages = no|update_messages = yes|' /etc/yum/yum-cron.conf
sed -i 's|^download_updates = no|download_updates = yes|' /etc/yum/yum-cron.conf
sed -i 's|^apply_updates = no|apply_updates = yes|' /etc/yum/yum-cron.conf
sed -i 's|^emit_via = stdio|emit_via = email|' /etc/yum/yum-cron.conf
sed -i "s|^email_to = root|email_to = ${YUM_CRON_EMAIL}|" /etc/yum/yum-cron-hourly.conf
sed -i 's|^update_cmd = default|update_cmd = security|' /etc/yum/yum-cron-hourly.conf
sed -i 's|^update_messages = no|update_messages = yes|' /etc/yum/yum-cron-hourly.conf
sed -i 's|^download_updates = no|download_updates = yes|' /etc/yum/yum-cron-hourly.conf
sed -i 's|^apply_updates = no|apply_updates = yes|' /etc/yum/yum-cron-hourly.conf
sed -i 's|^emit_via = stdio|emit_via = email|' /etc/yum/yum-cron-hourly.conf
egrep '^email_to|^update_messages|^download_updates|^apply_updates|^emit_via' /etc/yum/yum-cron.conf
egrep '^email_to|^update_cmd|^update_messages|^download_updates|^apply_updates|^emit_via' /etc/yum/yum-cron-hourly.conf
# fix bug in yum-cron nightly updates
if [[ $(grep -q "# success, dependencies resolved" /usr/sbin/yum-cron) -ne 0 ]]; then
patch /usr/sbin/yum-cron <<PATCHFILE
--- yum-cron.orig 2016-10-23 19:24:57.099859931 +0000
+++ yum-cron 2016-10-23 19:27:58.048784006 +0000
@@ -504,7 +504,13 @@
except yum.Errors.RepoError, e:
self.emitCheckFailed("%s" %(e,))
sys.exit()
- if res != 2:
+ if res == 0:
+ # success, empty transaction
+ sys.exit(0)
+ elif res == 2:
+ # success, dependencies resolved
+ pass
+ else:
self.emitCheckFailed("Failed to build transaction: %s" %(str.join("\n", resmsg),))
sys.exit(1)
PATCHFILE
fi
# (re)start the yum-cron service
(service yum-cron status > /dev/null && service yum-cron restart) || service yum-cron start
fi
We are going to use the “mainline” repo to get HTTP2 support. I like to create a conf file in “sites-available” that is linked to “sites-enabled” so I can disable things easily – this is enabled by adding a file under /etc/nginx/conf.d.
#!/bin/bash
# import src utility
if [[ -z $(type -t src) ]]; then
source <(curl -sL https://www.doublesharp.com/src)
fi
src osname
src osversion
cat <<REPO > /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
# default repo
#baseurl=http://nginx.org/packages/$(osname)/$(osversion)/\$basearch/
# mainline "dev" repo for http2 support
baseurl=http://nginx.org/packages/mainline/$(osname)/$(osversion)/\$basearch/
gpgcheck=0
enabled=1
REPO
#install nginx
yum install -y nginx
# turn on for reboots
systemctl enable nginx
mkdir -p /etc/nginx/includes
mkdir -p /etc/nginx/sites-enabled
mkdir -p /etc/nginx/sites-available
mkdir -p /etc/nginx/streams-enabled
mkdir -p /etc/nginx/streams-available
# use a conf file to include our sites-enabled conf files
cat <<SITESENABLED > /etc/nginx/includes/sites-enabled.conf
include /etc/nginx/sites-enabled/*.conf;
SITESENABLED
[[ -f "/etc/nginx/conf.d/_.sites-enabled.conf" ]] || ln -s /etc/nginx/includes/sites-enabled.conf /etc/nginx/conf.d/_.sites-enabled.conf
# enable httpd in selinux
semanage permissive -a httpd_t
cat <<NGINX_CONF > /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
worker_rlimit_nofile 100000;
events {
# determines how much clients will be served per worker
# max clients = worker_connections * worker_processes
# max clients is also limited by the number of socket connections available on the system (~64k)
worker_connections 100000;
# optmized to serve many clients with each thread, essential for linux
use epoll;
# accept as many connections as possible, may flood worker connections if set too low
multi_accept on;
}
# web servers / virtual hosts
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '\$remote_addr - \$remote_user [\$time_local] "\$request" '
'\$status \$body_bytes_sent "\$http_referer" '
'"\$http_user_agent" "\$http_x_forwarded_for"';
access_log /var/log/nginx/access.log combined flush=1m buffer=128k;
# cache informations about FDs, frequently accessed files
# can boost performance, but you need to test those values
open_file_cache max=200000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;
# send headers in one peace, its better then sending them one by one
tcp_nopush on;
# don't buffer data sent, good for small data bursts in real time
tcp_nodelay on;
# server will close connection after this time
keepalive_timeout 30;
# allow the server to close connection on non responding client, this will free up memory
reset_timedout_connection on;
# request timed out -- default 60
client_body_timeout 10;
# if client stop responding, free up memory -- default 60
send_timeout 2;
# reduce the data that needs to be sent over network
gzip on;
gzip_min_length 10240;
gzip_proxied expired no-cache no-store private auth;
gzip_types text/plain text/css text/xml text/javascript application/x-javascript application/xml;
gzip_disable "MSIE [1-6]\.";
proxy_buffer_size 128k;
proxy_buffers 64 256k;
proxy_busy_buffers_size 256k;
proxy_ignore_client_abort on;
include /etc/nginx/conf.d/*.conf;
}
# load balancer streams
stream {
include /etc/nginx/streams-enabled/*.conf;
}
NGINX_CONF
# create a virtual server conf file that is in sites-available
cat <<NGINX_HOST > /etc/nginx/sites-available/myapp.conf
upstream myapp {
# our app will be on localhost port 3000, but you can change this here
server 127.0.0.1:3000 fail_timeout=0;
}
server {
listen 80;
server_name myapp.example.com;
location / {
proxy_set_header Host \$host:\$server_port;
proxy_set_header X-Real-IP \$remote_addr;
proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto \$scheme;
proxy_pass http://myapp;
}
}
NGINX_HOST
# link this conf to sites-enabled. it's important to use the full path
#ln -s /etc/nginx/sites-available/myapp.conf /etc/nginx/sites-enabled/myapp.conf
nginx -t && (service nginx status > /dev/null && service nginx restart)
To install Redis with yum, first you need to install EPEL. Once the installed, you will have access to the repository containing the Redis install.
#!/bin/bash # install the EPEL repo to access Redis yum install -y epel-release yum install -y redis # fix redis background saves on low memory sysctl vm.overcommit_memory=1 && cat <<SYSCTL_MEM > /etc/sysctl.d/88-vm.overcommit_memory.conf vm.overcommit_memory = 1 SYSCTL_MEM # increase max connections sysctl -w net.core.somaxconn=65535 && cat <<SYSCTL_CONN > /etc/sysctl.d/88-net.core.somaxconn.conf net.core.somaxconn = 65535 SYSCTL_CONN sysctl -w fs.file-max=100000 && cat <<SYSCTL_FILEMAX > /etc/sysctl.d/88-fs.file-max.conf fs.file-max = 100000 SYSCTL_FILEMAX sed -i "s|^tcp-backlog [[:digit:]]\+|tcp-backlog 65535|" /etc/redis.conf # enable redis service on reboot systemctl enable redis # start service (service redis status > /dev/null && service redis restart) || service redis start
We want to install Node.js and then the PM2 package globally so that it can be accessed by other users.
#!/bin/bash # make sure the SRC_NODE_VERSION is set if [[ -z $SRC_NODE_VERSION ]]; then echo "You must specify a node version using \$SRC_NODE_VERSION"; else # Select node version to install curl --silent --location https://rpm.nodesource.com/setup_$SRC_NODE_VERSION.x | bash - # install via yum yum install -y git gcc-c++ make nodejs fi # PM2 - install as global npm install pm2@latest -g
As root, create a new user named “appuser”, or whatever you want your app user to be named. This could even be the default centos@/ec2-user@/etc that many hosts provide.
adduser appuser passwd appuser
Log in as the “appuser” user and create the Node app in your home directory. This directory should be owned by the “appuser”. In this case we assume the server is going to be listening on localhost port 3000, which means we can manage it with pm2 without having root permissions.
mkdir ~/apps cd /apps # create your app here, git clone, whatever # we assume the app is in ~/apps/myapp/server.js pm2 start ~/apps/myapp/server.js --name=myapp pm2 status myapp pm2 restart myapp
The post Node.js + PM2 + NGINX + Redis on CentOS 7 appeared first on Justin Silver.
]]>