Millistream Database Application (MDA) version 0.9.40 released

The Millistream MDA project team is pleased to announce the release of version 0.9.40 of our open source application for updating SQL servers with data from the Millistream Data Feed.

  • Fixed an error on PostgreSQL where escaped string containing forward slash “\” characters would be sent wrongly to the SQL server if “standard_conforming_strings” is enabled on the server (which it is per default in newer versions of PostgreSQL).
  • Support “connectionstring” setting in mda.config for the Microsoft SQL Server and PostreSQL servers for power users that want to set more options for the connection to the SQL server than the available settings in mda.config allow. When “connectionstring” is used then the other settings such as “username”, “password”, “dbname”, “server” and “port” are ignored by the MDA.
  • On Microsoft SQL Server and PostgreSQL a “mda –check-types” could erroneously report errors for “numeric” columns with a length between (10,0) and (19,0).
  • When connecting with MySQL/MariaDB, MDA will now read options from the “mda” group in the “my.cnf” file if such a group exists.
  • The SSL settings added in v0.9.39 for MySQL/MariaDB will now also work for PostgreSQL (even though they are perhaps best set there using the “connectionstring” setting anyway).
  • Added man pages for “mda” and “mda.config” on Linux.

As earlier announced, as of v0.9.33 the database schema is developed independently of the MDA and the latest schema including upgrade scripts is available on our ftp server under the documents section (MDA_schemas_[version].zip). As of this writing the latest database schema is v89 and thus the file is available from ftp://ftp.millistream.com/documents/MDA_schema_89.zip.

Do note however that the latest schemas are only needed if you need the new columns/tables, they are not mandatory to operate the MDA!

We consider MDA 0.9.40 to be the best version of MDA available and we recommend that users of older versions upgrade as soon as possible.

MDA 0.9.40 is available for download from the following locations:

Windows users upgrade by running the installer for the new version. Linux users who have added our apt or yum repositories can upgrade by simply issuing:

  • Debian/Ubuntu: “apt update && apt upgrade“
  • Red Hat/CentOS/Amazon Linux AMI: “yum upgrade“

Information on how to add these repositories are available at ftp://ftp.millistream.com/binaries/linux/.

And as usual, if you have any questions what so ever regarding this release or other things, don’t hesitate to contact us at tech@millistream.com.

/by

Extended News Coding Latency Decreased to below 100microseconds

Since day one, extended news coding for news messages in the Millistream Market Data Feed (MDF) have been handled asynchronously in order to keep the latency of the news contents low. The initial News Headline message contains the coding supplied by the source and the asynchronous process uses this initial coding in order to find extended coding (i.e more ISIN codes connected with the companies in the text message and/or possible company insref:s).

Over time this process have been slightly slower and slower to process each new News Headline message as it worked with a single queue and used a SQL server as the database for lookups. However during the weekend we have put a new process into production that uses an internal in-memory database with a O(1) guarantee for lookups which have lowered the latency down to below 100µs.

/by

Added support for Debian 9 “Stretch”

Even though the release date of Debian 9 “Stretch” is yet unknown, it went into full freeze at 2017-02-05 and consequently we have updated our DEB repository with builds of all our software products for Debian 9.

For fresh installs all you have to do is to add our repository and our signing key:

sudo wget "https://packages.millistream.com/apt/sources.list.d/stretch.list" -O /etc/apt/sources.list.d/millistream.list
wget -q "https://packages.millistream.com/D2FCCE35.gpg" -O- | sudo apt-key add -
sudo apt update

Or if you are upgrading from a previous version of Debian you can simply open /etc/apt/sources.list.d/millistream.list in a text editor of your choice and change the distribution name to “stretch” so that it matches this content:

deb https://packages.millistream.com/apt stretch main
deb-src https://packages.millistream.com/apt stretch main

After which you can fetch the new binaries with:

sudo apt update
sudo apt upgrade

/by

Millistream Data Feed API (MDF) version 1.0.20 released

Millistream is pleased to announce the release of version 1.0.20 of our open source Data Feed API.

  • updated mdf_fields.h to the latest version (7 new fields)
  • support for new HMAC digest algorithms: RIPMD160, SHA512 and Whirlpool
  • support OpenSSL v1.1.0+
  • new function mdf_message_move()
  • new function mdf_message_serialize()
  • new function mdf_message_deserialize()

As usual this version of MDF is binary compatible with previous versions, so there is no need to recompile client applications, so users (including users of the Java JNI wrapper) can simply upgrade by replacing the libmdf file itself.

MDF 1.0.20 is available for download from the following locations:

Linux users who have added our apt or yum repositories can upgrade by simply issuing:

  • Debian/Ubuntu: “apt-get update && apt-get upgrade“
  • Red Hat/CentOS/Amazon Linux AMI: “yum upgrade“

Information on how to add these repositories are available at
ftp://ftp.millistream.com/binaries/linux/

And as usual, if you have any questions what so ever regarding this release or other things, don’t hesitate to contact us at tech@millistream.com.

/by

Added support for Ubuntu 17.04

Preparing for the upcoming release of Ubuntu 17.04 (Zesty Zapus) due to release on 2017-04-13 we have updated our DEB repository with compatible versions of all our software products.

For fresh installs all you have to do is to add our repository and our signing key:

sudo wget "https://packages.millistream.com/apt/sources.list.d/zesty.list" -O /etc/apt/sources.list.d/millistream.list
wget -q "https://packages.millistream.com/D2FCCE35.gpg" -O- | sudo apt-key add -
sudo apt update

Or if you are upgrading from a previous version of Ubuntu you can simply re-enable the repository (the upgrade process in Ubuntu/Debian disables all external repositories) and fetch the new binaries:

sudo wget "https://packages.millistream.com/apt/sources.list.d/zesty.list" -O /etc/apt/sources.list.d/millistream.list
sudo apt update
sudo apt upgrade

/by

Millistream Database Application (MDA) version 0.9.39 released

The Millistream MDA project team is pleased to announce the release of
version 0.9.39 of our open source application for updating SQL servers
with data from the Millistream Data Feed.

* The maximum value reported in the “instrument-hash-size” in the output of “mda –status”
used the configured value and not the actual value (the configured value is scaled to the
next power of two for performance reasons).

* Rows from corporateactions could not be removed if the message did not contain “source”
(and “source” as a field for this table will be deprecated in the future).

* If the calculated change factor when updating a corporate action with a new adjustment factor
is “1” then the MDA will no longer adjust the prices in the “adjustedpricehistory” table.

* Added error logging for adjustments from a corporate action event.

* On Unix/Linux, write human readable errors to the error log instead of just the errno value.

* Notify systemd on status and main pid of MDA on systemd supported systems.

* Support the systemd watchdog (at the moment only the status and feed received thread are
covered).

* Changed the maximum timeout value for the queue semaphores to 5 minutes from INFINITY

* Updated the AppArmor profile to be compatible with changes introduced in Ubuntu 16.04LTS

As earlier announced, as of v0.9.33 the database schema is developed independently of
the MDA and the latest schema including upgrade scripts is available on our ftp server
under the documents section (MDA_schemas_[version].zip). As of this writing the latest
database schema is v85 and thus the file is available from:
ftp://ftp.millistream.com/documents/MDA_schema_88.zip

Do note however that the latest schemas are only needed if you need the new columns/tables,
they are not mandatory to operate the MDA!

We consider MDA 0.9.39 to be the best version of MDA available and we
recommend that users of older versions upgrade as soon as possible.

MDA 0.9.39 is available for download from the following locations:

Windows installer: ftp://ftp.millistream.com/binaries/windows/
Red Hat / CentOS: https://packages.millistream.com/rpm/
Debian / Ubuntu: https://packages.millistream.com/apt/
Source code: ftp://ftp.millistream.com/sources/
Documentation: ftp://ftp.millistream.com/documents/mda.pdf

Windows users upgrade by running the installer for the new version.

Linux users who have added our apt or yum repositories can upgrade by simply
issuing “apt-get update && apt-get upgrade” (Debian/Ubuntu) or
“yum upgrade” (Red Hat/CentOS).

Information on how to add these repositories are available at
ftp://ftp.millistream.com/binaries/linux/
And as usual, if you have any questions what so ever regarding this release
or other things, don’t hesitate to contact us at tech@millistream.com

/by

Millistream to reduce latency by 94% from FIX/FAST sources during 2017

During 2017, Millistream will put our new proprietary FIX/FAST engine into production, replacing our 6 year old ageing FIX/FAST engine. With this new version comes a massive increase in message throughput (+1606%) and decrease in per message decoding latency (-94%). Changes in how data is presented to the application makes for less complex feed-handlers yielding further performance improvements as well as shorter time to market for future FIX/FAST sources.

Performance numbers (measured on a i5-4670K @ 3.4Ghz):

  • decodes 8+ million messages per second per CPU core
  • decoding latency per message 115ns

Affected marketplaces/sources:

  • Nordic Growth Market (marketplace #29929)
  • Nordic Derivatives Exchange (marketplace #29933)
  • Nordic MTF (marketplace #29930)
  • Oslo Børs – Fixed Income (marketplace #39890)
  • Oslo ABM (marketplace #33668)
  • Oslo Børs – Indexes (marketplace #39890)
  • Oslo Axess – Indexes (marketplace #33187)
  • Oslo Børs – News (newsref #8199)

Due to structural differences the new engine is not a drop-in replacement for the old engine so all our FIX/FAST feed-handlers have to be rewritten which is the reason why the new version will not be put into production until some time next year.

/by

Millistream Database Application (MDA) version 0.9.38 released

The Millistream MDA project team is pleased to announce the release of
version 0.9.38 of our open source application for updating SQL servers
with data from the Millistream Data Feed.

* For instruments that receive both a LastPrice and a NAV the ‘diffX’ and ‘diffXprc’
values in the ‘performance’ table could be accidentally calculated using the NAV instead
of the LastPrice if the LastPrice was NULL when the NAV was set. We have now changed it
so that NAV is only used for the calculation of diffs if the InstrumentType is set to 8
(Mutual Fund), for every other instrument type the LastPrice value is used.

* It was discovered that MDA upon receiving a InstrumentDelete message would try and
delete data from a table even if that table where disabled in the config (for example
if ‘requestperformance’ was set to “0” in mda.config then the MDA would still try to delete
the insref from the ‘performance’ table) this could lead to unnecessary error logs if
the tables where disabled in the config due to them not existing.

* The MDA will now only calculate the diffs in the ‘performance’ table if both
‘requestquotes’ and ‘requestperformance’ is enabled in the config, since data from both
messages are needed in order to calculate the diffs.

* On Linux systems we now create a “mda” user and group since the previous use of “nobody”
was not fully secure. This user and group is also used by default in new installs.

* Failed updates to the ‘orderbooks’ table would log more than one error for each error.

* The AppArmor profile have been updated to work with MariaDB, the previous profile referenced
the MySQL profile that is normally installed with the MySQL server but the MySQL profile
incuded in MariaDB is empty so the MDA could not access the MySQL socket in /run/mysqld/mysqld.sock

* Write the current size and the configured maximum size of the internal hash table used for
the orderbooks to the output of “mda –status” as ‘instrument-hash-size’ in the format:
‘current_value/maximum_value’. The maximum_value can be configured in mda.config under the
‘MILLISTREAM’ tab with the ‘instrumenthashsize’ property, the default value is ‘100000’.

As earlier announced, as of v0.9.33 the database schema is developed independently of
the MDA and the latest schema including upgrade scripts is available on our ftp server
under the documents section (MDA_schemas_[version].zip). As of this writing the latest
database schema is v85 and thus the file is available from:
ftp://ftp.millistream.com/documents/MDA_schema_86.zip

Do note however that the latest schemas are only needed if you need the new columns/tables,
they are not mandatory to operate the MDA!

We consider MDA 0.9.38 to be the best version of MDA available and we
recommend that users of older versions upgrade as soon as possible.

MDA 0.9.38 is available for download from the following locations:

Windows installer: ftp://ftp.millistream.com/binaries/windows/
Red Hat / CentOS: https://packages.millistream.com/rpm/
Debian / Ubuntu: https://packages.millistream.com/apt/
Source code: ftp://ftp.millistream.com/sources/
Documentation: ftp://ftp.millistream.com/documents/mda.pdf

Windows users upgrade by running the installer for the new version.

Linux users who have added our apt or yum repositories can upgrade by simply
issuing “apt-get update && apt-get upgrade” (Debian/Ubuntu) or
“yum upgrade” (Red Hat/CentOS).

/by

Millistream Database Application (MDA) version 0.9.37 released

The Millistream MDA project team is pleased to announce the release of
version 0.9.37 of our open source application for updating SQL servers
with data from the Millistream Data Feed.

* It was discovered that the columns ‘trades.canceltime’ and ‘trades.agreementtime’
where not converted to local time even of the ‘localtime’ setting in mda.confg where
set to ‘1’.

* It was discovered that if the ‘localtime’ in mda.config where set to ‘1’ in order to
convert timestamps from UTC to the local time of the machine, the timestamps where
truncated to whole seconds, i.e the millisecond part (if available) where removed.

* The ‘ignoremillisec’ setting in mda.config has been renamed to ‘truncatetime’ (the MDA
will however still support ‘ignoremillisec’) and the setting has been extended so that
it now support the following values:
– 0 : Truncate to microseconds, i.e ‘HH:MM:SS.mmmuuu’ or time(6). This is the default.
– 1 : Truncate to whole seconds, i.e ‘HH:MM:SS’ or time(0)
– 2 : Truncate to milliseconds, i.e ‘HH:MM:SS.mmm’ or time(3)

* The CommandTimeout property for the Microsoft SQL Server are now set to ‘0’ to allow queries
to complete even if they take a very long time (if the SQL server is busy). If not set as
was the case in previous versions of MDA the default value in MSSQL is 30 seconds which could
lead to endless loops on very busy SQL servers where the MDA would re-issue the same query over
and over again if they took longer to execute than 30 seconds. The new value of ‘0’ means that
the timeout is disabled which matches how the other SQL servers work (MySQL/MariaDB/PostgreSQL).

* If the account credentials is invalid then MDA will now log so instead of just logging that
it has been disconnected due to the server shutting down.

* The sql script for MSSQL have been cleaned up so that the ‘time’ column for tables which
will never get timestamps with fractional seconds now are time(0) instead of time(7). And
the ‘time’ column for the ‘trades’ and ‘orderbooks’ table have been changed from time(7) to
time(6) since time(6) will be the highest time resolution that MDA supports. This will only
affect new installs and there is no need to update your sql server in order to incorporate
these changes since they are purely estetic.

* The sql script for MySQL (v5.6.4 and above) and MariaDB (any version) have been changed in
order to prepare for microsecond resolution in the ‘trades’ and ‘orderbooks’ tables where
time(3) have been changed into time(6). There is at the moment no reason to update existing
installs since such timestamps are not yet distributed, and is really only needed if you
really want to have microsecond resolution in your database once they will be available.

As earlier announced, as of v0.9.33 the database schema is developed independently of
the MDA and the latest schema including upgrade scripts is available on our ftp server
under the documents section (MDA_schemas_[version].zip). As of this writing the latest
database schema is v85 and thus the file is available from:
ftp://ftp.millistream.com/documents/MDA_schema_85.zip

Do note however that the latest schemas are only needed if you need the new columns/tables,
they are not mandatory to operate the MDA!

We consider MDA 0.9.37 to be the best version of MDA available and we
recommend that users of older versions upgrade as soon as possible.

MDA 0.9.37 is available for download from the following locations:

Windows installer: ftp://ftp.millistream.com/binaries/windows/
Red Hat / CentOS: https://packages.millistream.com/rpm/
Debian / Ubuntu: https://packages.millistream.com/apt/
Source code: ftp://ftp.millistream.com/sources/
Documentation: ftp://ftp.millistream.com/documents/mda.pdf

Windows users upgrade by running the installer for the new version.

Linux users who have added our apt or yum repositories can upgrade by simply
issuing “apt-get update && apt-get upgrade” (Debian/Ubuntu) or
“yum upgrade” (Red Hat/CentOS).

Information on how to add these repositories are available at
ftp://ftp.millistream.com/binaries/linux/

And as usual, if you have any questions what so ever regarding this release
or other things, don’t hesitate to contact us at tech@millistream.com

Yours,
The Millistream Development Team

/by

Millistream Data Feed API (MDF) version 1.0.19 released

Millistream is pleased to announce the release of version 1.0.19 of our open
source Data Feed API.

* updated mdf_fields.h to the latest version (5 new fields)
* the status callback function have got a new status code: MDF_STATUS_READYTOLOGON
this signals that the client<->server handshake are complete and it’s now
possible for the client to send the Logon message.
* the message callback function can now be exited before all messages have
been pulled from the queue, previously this would stall the API until there
where more data sent on the wire but now the message callback is called
immediately again if there are more messages in the queue.
* fix in mdf_message_add_time() where milliseconds could be encoded wrongly
if they where not exactly three digits long (i.e cases such as ‘.1’, ‘.01’
and ‘.001’ would all be handled as ‘.001’)
* new function mdf_message_add_time3() that works like mdf_message_add_time2()
except it encodes nanoseconds instead of milliseconds.
* the internal code for converting the on-the-wire numbers to strings for
mdf_get_next_field() was optimized for large numbers.

As usual this version of MDF is binary compatible with previous versions, so
there is no need to recompile client applications, so users (including users of
the Java JNI wrapper) can simply upgrade by replacing the libmdf file itself.

MDF 1.0.19 is available for download from the following locations:

Windows: ftp://ftp.millistream.com/binaries/windows/
Mac OS X: ftp://ftp.millistream.com/binaries/mac/
Red Hat / CentOS: https://packages.millistream.com/rpm/
Debian / Ubuntu: https://packages.millistream.com/apt/
Source code: ftp://ftp.millistream.com/sources/
Documentation: ftp://ftp.millistream.com/documents/

Linux users who have added our apt or yum repositories can upgrade by simply
issuing “apt-get update && apt-get upgrade” (Debian/Ubuntu) or
“yum upgrade” (Red Hat/CentOS/AmazonLinux).

Information on how to add these repositories are available at
ftp://ftp.millistream.com/binaries/linux/

And as usual, if you have any questions what so ever regarding this release
or other things, don’t hesitate to contact us at tech@millistream.com

Yours,
The Millistream Development Team

/by