Millistream to reduce latency by 94% from FIX/FAST sources during 2017

During 2017, Millistream will put our new proprietary FIX/FAST engine into production, replacing our 6 year old ageing FIX/FAST engine. With this new version comes a massive increase in message throughput (+1606%) and decrease in per message decoding latency (-94%). Changes in how data is presented to the application makes for less complex feed-handlers yielding further performance improvements as well as shorter time to market for future FIX/FAST sources.

Performance numbers (measured on a i5-4670K @ 3.4Ghz):

  • decodes 8+ million messages per second per CPU core
  • decoding latency per message 115ns

Affected marketplaces/sources:

  • Nordic Growth Market (marketplace #29929)
  • Nordic Derivatives Exchange (marketplace #29933)
  • Nordic MTF (marketplace #29930)
  • Oslo Børs – Fixed Income (marketplace #39890)
  • Oslo ABM (marketplace #33668)
  • Oslo Børs – Indexes (marketplace #39890)
  • Oslo Axess – Indexes (marketplace #33187)
  • Oslo Børs – News (newsref #8199)

Due to structural differences the new engine is not a drop-in replacement for the old engine so all our FIX/FAST feed-handlers have to be rewritten which is the reason why the new version will not be put into production until some time next year.

Millistream Database Application (MDA) version 0.9.38 released

The Millistream MDA project team is pleased to announce the release of
version 0.9.38 of our open source application for updating SQL servers
with data from the Millistream Data Feed.

* For instruments that receive both a LastPrice and a NAV the ‘diffX’ and ‘diffXprc’
values in the ‘performance’ table could be accidentally calculated using the NAV instead
of the LastPrice if the LastPrice was NULL when the NAV was set. We have now changed it
so that NAV is only used for the calculation of diffs if the InstrumentType is set to 8
(Mutual Fund), for every other instrument type the LastPrice value is used.

* It was discovered that MDA upon receiving a InstrumentDelete message would try and
delete data from a table even if that table where disabled in the config (for example
if ‘requestperformance’ was set to “0” in mda.config then the MDA would still try to delete
the insref from the ‘performance’ table) this could lead to unnecessary error logs if
the tables where disabled in the config due to them not existing.

* The MDA will now only calculate the diffs in the ‘performance’ table if both
‘requestquotes’ and ‘requestperformance’ is enabled in the config, since data from both
messages are needed in order to calculate the diffs.

* On Linux systems we now create a “mda” user and group since the previous use of “nobody”
was not fully secure. This user and group is also used by default in new installs.

* Failed updates to the ‘orderbooks’ table would log more than one error for each error.

* The AppArmor profile have been updated to work with MariaDB, the previous profile referenced
the MySQL profile that is normally installed with the MySQL server but the MySQL profile
incuded in MariaDB is empty so the MDA could not access the MySQL socket in /run/mysqld/mysqld.sock

* Write the current size and the configured maximum size of the internal hash table used for
the orderbooks to the output of “mda –status” as ‘instrument-hash-size’ in the format:
‘current_value/maximum_value’. The maximum_value can be configured in mda.config under the
‘MILLISTREAM’ tab with the ‘instrumenthashsize’ property, the default value is ‘100000’.

As earlier announced, as of v0.9.33 the database schema is developed independently of
the MDA and the latest schema including upgrade scripts is available on our ftp server
under the documents section (MDA_schemas_[version].zip). As of this writing the latest
database schema is v85 and thus the file is available from:
ftp://ftp.millistream.com/documents/MDA_schema_86.zip

Do note however that the latest schemas are only needed if you need the new columns/tables,
they are not mandatory to operate the MDA!

We consider MDA 0.9.38 to be the best version of MDA available and we
recommend that users of older versions upgrade as soon as possible.

MDA 0.9.38 is available for download from the following locations:

Windows installer: ftp://ftp.millistream.com/binaries/windows/
Red Hat / CentOS: https://packages.millistream.com/rpm/
Debian / Ubuntu: https://packages.millistream.com/apt/
Source code: ftp://ftp.millistream.com/sources/
Documentation: ftp://ftp.millistream.com/documents/mda.pdf

Windows users upgrade by running the installer for the new version.

Linux users who have added our apt or yum repositories can upgrade by simply
issuing “apt-get update && apt-get upgrade” (Debian/Ubuntu) or
“yum upgrade” (Red Hat/CentOS).

Millistream Database Application (MDA) version 0.9.37 released

The Millistream MDA project team is pleased to announce the release of
version 0.9.37 of our open source application for updating SQL servers
with data from the Millistream Data Feed.

* It was discovered that the columns ‘trades.canceltime’ and ‘trades.agreementtime’
where not converted to local time even of the ‘localtime’ setting in mda.confg where
set to ‘1’.

* It was discovered that if the ‘localtime’ in mda.config where set to ‘1’ in order to
convert timestamps from UTC to the local time of the machine, the timestamps where
truncated to whole seconds, i.e the millisecond part (if available) where removed.

* The ‘ignoremillisec’ setting in mda.config has been renamed to ‘truncatetime’ (the MDA
will however still support ‘ignoremillisec’) and the setting has been extended so that
it now support the following values:
– 0 : Truncate to microseconds, i.e ‘HH:MM:SS.mmmuuu’ or time(6). This is the default.
– 1 : Truncate to whole seconds, i.e ‘HH:MM:SS’ or time(0)
– 2 : Truncate to milliseconds, i.e ‘HH:MM:SS.mmm’ or time(3)

* The CommandTimeout property for the Microsoft SQL Server are now set to ‘0’ to allow queries
to complete even if they take a very long time (if the SQL server is busy). If not set as
was the case in previous versions of MDA the default value in MSSQL is 30 seconds which could
lead to endless loops on very busy SQL servers where the MDA would re-issue the same query over
and over again if they took longer to execute than 30 seconds. The new value of ‘0’ means that
the timeout is disabled which matches how the other SQL servers work (MySQL/MariaDB/PostgreSQL).

* If the account credentials is invalid then MDA will now log so instead of just logging that
it has been disconnected due to the server shutting down.

* The sql script for MSSQL have been cleaned up so that the ‘time’ column for tables which
will never get timestamps with fractional seconds now are time(0) instead of time(7). And
the ‘time’ column for the ‘trades’ and ‘orderbooks’ table have been changed from time(7) to
time(6) since time(6) will be the highest time resolution that MDA supports. This will only
affect new installs and there is no need to update your sql server in order to incorporate
these changes since they are purely estetic.

* The sql script for MySQL (v5.6.4 and above) and MariaDB (any version) have been changed in
order to prepare for microsecond resolution in the ‘trades’ and ‘orderbooks’ tables where
time(3) have been changed into time(6). There is at the moment no reason to update existing
installs since such timestamps are not yet distributed, and is really only needed if you
really want to have microsecond resolution in your database once they will be available.

As earlier announced, as of v0.9.33 the database schema is developed independently of
the MDA and the latest schema including upgrade scripts is available on our ftp server
under the documents section (MDA_schemas_[version].zip). As of this writing the latest
database schema is v85 and thus the file is available from:
ftp://ftp.millistream.com/documents/MDA_schema_85.zip

Do note however that the latest schemas are only needed if you need the new columns/tables,
they are not mandatory to operate the MDA!

We consider MDA 0.9.37 to be the best version of MDA available and we
recommend that users of older versions upgrade as soon as possible.

MDA 0.9.37 is available for download from the following locations:

Windows installer: ftp://ftp.millistream.com/binaries/windows/
Red Hat / CentOS: https://packages.millistream.com/rpm/
Debian / Ubuntu: https://packages.millistream.com/apt/
Source code: ftp://ftp.millistream.com/sources/
Documentation: ftp://ftp.millistream.com/documents/mda.pdf

Windows users upgrade by running the installer for the new version.

Linux users who have added our apt or yum repositories can upgrade by simply
issuing “apt-get update && apt-get upgrade” (Debian/Ubuntu) or
“yum upgrade” (Red Hat/CentOS).

Information on how to add these repositories are available at
ftp://ftp.millistream.com/binaries/linux/

And as usual, if you have any questions what so ever regarding this release
or other things, don’t hesitate to contact us at tech@millistream.com

Yours,
The Millistream Development Team

Millistream Data Feed API (MDF) version 1.0.19 released

Millistream is pleased to announce the release of version 1.0.19 of our open
source Data Feed API.

* updated mdf_fields.h to the latest version (5 new fields)
* the status callback function have got a new status code: MDF_STATUS_READYTOLOGON
this signals that the client<->server handshake are complete and it’s now
possible for the client to send the Logon message.
* the message callback function can now be exited before all messages have
been pulled from the queue, previously this would stall the API until there
where more data sent on the wire but now the message callback is called
immediately again if there are more messages in the queue.
* fix in mdf_message_add_time() where milliseconds could be encoded wrongly
if they where not exactly three digits long (i.e cases such as ‘.1’, ‘.01’
and ‘.001’ would all be handled as ‘.001’)
* new function mdf_message_add_time3() that works like mdf_message_add_time2()
except it encodes nanoseconds instead of milliseconds.
* the internal code for converting the on-the-wire numbers to strings for
mdf_get_next_field() was optimized for large numbers.

As usual this version of MDF is binary compatible with previous versions, so
there is no need to recompile client applications, so users (including users of
the Java JNI wrapper) can simply upgrade by replacing the libmdf file itself.

MDF 1.0.19 is available for download from the following locations:

Windows: ftp://ftp.millistream.com/binaries/windows/
Mac OS X: ftp://ftp.millistream.com/binaries/mac/
Red Hat / CentOS: https://packages.millistream.com/rpm/
Debian / Ubuntu: https://packages.millistream.com/apt/
Source code: ftp://ftp.millistream.com/sources/
Documentation: ftp://ftp.millistream.com/documents/

Linux users who have added our apt or yum repositories can upgrade by simply
issuing “apt-get update && apt-get upgrade” (Debian/Ubuntu) or
“yum upgrade” (Red Hat/CentOS/AmazonLinux).

Information on how to add these repositories are available at
ftp://ftp.millistream.com/binaries/linux/

And as usual, if you have any questions what so ever regarding this release
or other things, don’t hesitate to contact us at tech@millistream.com

Yours,
The Millistream Development Team

Millistream now available on Verizon Financial Network

The Verizon Financial Network (VFN) would like to welcome Millistream to its catalog of available market data products and services!

Millistream is a provider of real-time market data specializing in feed and web solutions. The company was founded in 2008 and is led by a team of experts in the market data industry.

Millistream’s data feed solutions are based on the Millistream Engine, a modern platform developed for extremely high scalability and performance.

To order connectivity to Millistream or other services, please contact vfn-feed-change@verizon.com or your Verizon Account Manager. For a complete list of services available on VFN, please send an inquiry to VFNSales@Verizon.com.

Introducing the Millistream Sandbox

The Millistream Sandbox is a production-like test environment. We use an actual feed log from a historical trading day as the source of the data and it’s sent in ”real time”, which basically means that the exchange data will have the same timing as on an actual trading day.

All Millistream products, like the Millistream Web Service (MWS), the Millistream streaming API and the Millistream Database, are possible to use with the Sandbox. An application developed using Sandbox data can simply switch to a production account/URL and be live in minutes.

The data is filtered and not all exchanges and other sources available in the Millistream feed are possible to test in the Sandbox.

Millistream Database Application (MDA) version 0.9.36 released

The Millistream MDA project team is pleased to announce the release of
version 0.9.36 of our open source application for updating SQL servers
with data from the Millistream Data Feed.

* Only reset the ‘quotes’, ‘noii’, ‘performance’ and ‘orderbooks’ tables when
receiving a InstrumentReset message if MDA is configured to request these
tables.

As earlier announced, as of v0.9.33 the database schema is developed independently of
the MDA and the latest schema including upgrade scripts is available on our ftp server
under the documents section (MDA_schemas_[version].zip). As of this writing the latest
database schema is v83 and thus the file is available from:
ftp://ftp.millistream.com/documents/MDA_schema_83.zip

Do note however that the latest schemas are only needed if you need the new columns/tables,
they are not mandatory to operate the MDA!

We consider MDA 0.9.36 to be the best version of MDA available and we
recommend that users of older versions upgrade as soon as possible.

MDA 0.9.36 is available for download from the following locations:

Windows installer: ftp://ftp.millistream.com/binaries/windows/
Red Hat / CentOS: https://packages.millistream.com/rpm/
Debian / Ubuntu: https://packages.millistream.com/apt/
Source code: ftp://ftp.millistream.com/sources/
Documentation: ftp://ftp.millistream.com/documents/mda.pdf

Windows users upgrade by running the installer for the new version.

Linux users who have added our apt or yum repositories can upgrade by simply
issuing “apt-get update && apt-get upgrade” (Debian/Ubuntu) or
“yum upgrade” (Red Hat/CentOS).

Information on how to add these repositories are available at
ftp://ftp.millistream.com/binaries/linux/

And as usual, if you have any questions what so ever regarding this release
or other things, don’t hesitate to contact us at tech@millistream.com

Millistream Data Feed API (MDF) version 1.0.18 is released

Millistream is pleased to announce the release of version 1.0.18 of our open
source Data Feed API.

* updated mdf_fields.h to the latest version (44 new fields and 2 new messages,
also note that the MDF_F_FREETEXTCOMMENTx fields have been renamed to MDF_F_Sx)
* fixed the code so that it builds correctly on a big endian system
* fixed various compiler warnings
* fixed wrong encoding of the values “10741.80” and “10741.90”, decoding
was not afflicted so this only applies if you use the MDF to send data
into the system.
* new function mdf_message_add_time2() added that allows adding of time
without having to go by strings.
* new function mdf_message_add_date2() added that allows adding of date
without having to go by strings.

As usual this version of MDF is binary compatible with previous versions, so
there is no need to recompile client applications, so users (including users of
the Java JNI wrapper) can simply upgrade by replacing the libmdf file itself.

MDF 1.0.18 is available for download from the following locations:

Windows: ftp://ftp.millistream.com/binaries/windows/
Mac OS X: ftp://ftp.millistream.com/binaries/mac/
Red Hat / CentOS: https://packages.millistream.com/rpm/
Debian / Ubuntu: https://packages.millistream.com/apt/
Source code: ftp://ftp.millistream.com/sources/
Documentation: ftp://ftp.millistream.com/documents/

Linux users who have added our apt or yum repositories can upgrade by simply
issuing “apt-get update && apt-get upgrade” (Debian/Ubuntu) or
“yum upgrade” (Red Hat/CentOS/AmazonLinux).

Information on how to add these repositories are available at
ftp://ftp.millistream.com/binaries/linux/

And as usual, if you have any questions what so ever regarding this release
or other things, don’t hesitate to contact us at tech@millistream.com

Millistream Database Application (MDA) version 0.9.35 released

The Millistream MDA project team is pleased to announce the release of
version 0.9.35 of our open source application for updating SQL servers
with data from the Millistream Data Feed.

* If MDA discovers that it’s connected to a different server than before
after a disconnect, it will now issue a complete snapshot request of all
data and not the standard incremental one. This ensures that all data is
100% in sync even if there is a time discrepancy/latency between servers.

* Fixed a problem where deletions of rows in pricehistory, adjustedpricehistory,
corporateactions, fundamentals, estimates, estimateshistory, ci and cihistory
would be ignored if there was an update for that particular row in the current
queue which had as of yet not been written to the SQL server.

* Fixed a problem with the command line arguments where the “next” argument
would be ignored if a previous argument used the “=” sign to give the argument
a value.

List of fixes from v0.9.34 (this release was never publicly announced):

* adjustedhistory could get the unadjusted value if the source column
was not present (the source column was removed from new installs as of v0.9.33)

* Fixed a problem where the aggregated statistics in the “stats” table, from the
“mda –status” command and reported to the Windows Performance Monitor could
experience a race condition and thus not always display the correct value.

* creation of the pid file is now optional (since it’s no longer used by either
the SysVinit scripts or by the Upstart or systemd init systems).

As earlier announced, as of v0.9.33 the database schema is developed independently of
the MDA and the latest schema including upgrade scripts is available on our ftp server
under the documents section (MDA_schemas_[version].zip). As of this writing the latest
database schema is v77 and thus the file is available from:
ftp://ftp.millistream.com/documents/MDA_schema_77.zip

Do note however that the latest schemas are only needed if you need the new columns/tables,
they are not mandatory to operate the MDA!

We consider MDA 0.9.35 to be the best version of MDA available and we
recommend that users of older versions upgrade as soon as possible.

MDA 0.9.35 is available for download from the following locations:

Windows installer: ftp://millistream.com/binaries/windows/
Red Hat / CentOS: https://packages.millistream.com/rpm/
Debian / Ubuntu: https://packages.millistream.com/apt/
Source code: ftp://millistream.com/sources/
Documentation: ftp://millistream.com/documents/mda.pdf

Windows users upgrade by running the installer for the new version.

Linux users who have added our apt or yum repositories can upgrade by simply
issuing “apt-get update && apt-get upgrade” (Debian/Ubuntu) or
“yum upgrade” (Red Hat/CentOS).

Information on how to add these repositories are available at
ftp://millistream.com/binaries/linux/

And as usual, if you have any questions what so ever regarding this release
or other things, don’t hesitate to contact us at tech@millistream.com

Millistream adds a new 1 Gbit/s access in Stockholm

In our data center in Lunda, we now offer access via a new IP-Only 1 Gbit/s Internet connection. This is run in parallel with the existing Verizon connection and is available over IPv4 and IPv6 as all other connections (except Amazon who only provides IPv4 access).

Valid access points are thus:

Address 	      Provider       Location
sth.millistream.com 	Verizon 	Stockholm, Sweden
sth2.millistream.com 	IP-Only 	Stockholm, Sweden
lon.millistream.com 	C4L 	        London, UK
aws.millistream.com 	Amazon EC2 	Dublin, Ireland