Added Support for Debian 10

Preparing for the upcoming release of Debian 10 (Buster) due to release on soon in mid 2019 we have updated our DEB repository with compatible versions of all our software products.

For fresh installs all you have to do is to add our repository and our signing key:

sudo wget "https://packages.millistream.com/apt/sources.list.d/buster.list" -O /etc/apt/sources.list.d/millistream.list
wget -q "https://packages.millistream.com/D2FCCE35.gpg" -O- | sudo apt-key add -
sudo apt update

Or if you are upgrading from a previous version of Debian you can simply re-enable the repository (the upgrade process in Ubuntu/Debian disables all external repositories) and fetch the new binaries:

sudo wget "https://packages.millistream.com/apt/sources.list.d/buster.list" -O /etc/apt/sources.list.d/millistream.list
sudo apt update
sudo apt upgrade

/by

Additional feed access point in Stockholm

In our data center in Lunda, we have now added an additional feed access point on our IP-Only Internet connection.

Valid feed access points for the Millistream Data Feed (libmdf) and Millistream Database Application (MDA):

Address                 Provider        Location
sth3.millistream.com    IP-Only         Stockholm, Sweden
sth2.millistream.com    IP-Only         Stockholm, Sweden 
 sth.millistream.com    Verizon         Stockholm, Sweden
 lon.millistream.com    IDE Group       London, UK
 aws.millistream.com    Amazon EC2      Dublin, Ireland
/by

Added Support for Ubuntu 19.04

Preparing for the upcoming release of Ubuntu 19.04 (Disco Dingo) due to release on 2019-04-18 we have updated our DEB repository with compatible versions of all our software products.

For fresh installs all you have to do is to add our repository and our signing key:

sudo wget "https://packages.millistream.com/apt/sources.list.d/disco.list" -O /etc/apt/sources.list.d/millistream.list
wget -q "https://packages.millistream.com/D2FCCE35.gpg" -O- | sudo apt-key add -
sudo apt update

Or if you are upgrading from a previous version of Ubuntu you can simply re-enable the repository (the upgrade process in Ubuntu/Debian disables all external repositories) and fetch the new binaries:

sudo wget "https://packages.millistream.com/apt/sources.list.d/disco.list" -O /etc/apt/sources.list.d/millistream.list
sudo apt update
sudo apt upgrade

/by

Millistream Database Application (MDA) version 0.9.43 released

The Millistream MDA project team is pleased to announce the release of version 0.9.43 of our open source application for updating SQL servers with data from the Millistream Data Feed.

  • New mod_odbc module for ODBC connectivity to sql servers. On Windows this more or less replaces the old OLE DB module (mod_mssql) since Microsoft seams to premier ODBC over OLE DB these days. On Linux it allows connectivity to MSSQL servers running either on Linux, macOS, Windows or Azure.While both MySQL, MariaDB and PostgreSQL have been fully tested and is supported we would recommend to use our native modules there instead for both performance and stability (ODBC is native for MSSQL but not for other sql servers).This should also allow connectivity to other ODBC compliant sql servers such as IBM D2 and Oracle but this has not been tested and most likely “–check-types” will not work since every sql server seams to define their data types differently even on a unified interface such as ODBC.
  • INSERT first tables will now only retry with an UPDATE if the error from the sql server indicated that it was a key violation.
  • Updates to the trades, pricehistory and adjustedpricehistory tables did not log an error if the INSERT failed and the following UPDATE returned with zero affected rows.
  • String values are now truncated to the length defined in the tabledef so that strings that for whatever reason do not comply with our defines will be accepted by the sql server instead of failing the entire query.
  • News items that have lingered for longer than 3600 seconds without receiving a last block of text and that are kicked from the queue will now be logged with their newsid and date.
  • Added connectionstring support for mod_mysql, the connection string is parsed by mda since the MySQL native driver does not support connection strings.
  • Added support for “sslcrl”, “sslcrlpath”, “connect_timeout” and “write_timeout” in the connectionstring for mod_mysql.
  • In order to maintain a uniform and simple interface we have decied to deprecate the “username”, “password”, “server”, “dbname”, “ssl_xxx” arguments from mda.config and instead only support “connectionstring”. However since these settings are in use by existing users MDA will continue to read and parse them as before, we have just removed them from the documentation and the default mda.config in a new install.
  • The status thread could sometimes be confused during shutdown and log that the sql server disconnected mda.
  • Table/column name comparisons are now case insensitive on “–check” and “–check-types”.
  • On “–upgrade-db” mda would enclose table/column literals with [], “ and ” on mod_mssql, mod_mysql and mod_postgresql respectively but since mda does not do this when performing INSERT/UPDATE and since some sql servers (such as PostgreSQL) will handle literals case sensitive if they are created while being enclosed we have now removed this behaviour from “–upgrade-db”.

As earlier announced, as of v0.9.33 the database schema is developed independently of the MDA and the latest schema including upgrade scripts is available on our ftp server under the documents section  (MDA_schemas_[version].zip). As of this writing the latest database schema is v94 and is available from ftp://ftp.millistream.com/documents/MDA_schema_94.zip

Do note however that the latest schemas are only needed if you need the new columns/tables, they are not mandatory to operate the MDA!

We consider MDA 0.9.43 to be the best version of MDA available and we recommend that users of older versions upgrade as soon as possible.

MDA 0.9.43 is available for download from the following locations:

Windows users upgrade by running the installer for the new version.

Linux users who have added our apt or yum repositories can upgrade by simply issuing:

  • Debian/Ubuntu: “apt update && apt upgrade“
  • Red Hat/CentOS/Amazon Linux AMI: “yum upgrade“

Information on how to add these repositories are available at ftp://millistream.com/binaries/linux/

And as usual, if you have any questions what so ever regarding this release or other things, don’t hesitate to contact us at tech@millistream.com

/by

Millistream Database Application (MDA) version 0.9.42 released

The Millistream MDA project team is pleased to announce the release of version 0.9.42 of our open source application for updating SQL servers with data from the Millistream Data Feed.

  • Support for new table “newstags” from SQL Schema v94. Some News Agencies can provide a wealth of tags for each news item and in the future these tags will replace many of the current values in “news.newstype”, “news.topic” and “news.regulatory”.
  • Changed the umask of the semaphore used for “mda –status” on Linux so that other members of the “mda” group can call “mda –status”.
  • When calculating the diffX values in the “performance” table, MDA will now use MDF_F_PRICETYPE instead of MDF_F_INSTRUMENTTYPE when determining whether to calculate using NAV or LastPrice.
  • Log all failed sql statements to the sql log even if it’s an insert on insert-first or an update on update-first, also we always log the sql server error string to the sql log.
  • Append the queue number to the sql log so it’s possible to see which queue that ran the query.
  • Sql statements that resulted in AffectedRows==0 could be logged twice to the sql log.
  • Sets XACT_ABORT to “OFF” when connecting to a MSSQL Server to prevent duplicate key errors from rolling back a transaction if the server have been configured to do that otherwise.
  • Use the MSOLEDBSQL provider for MSSQL Server if found since this is the new OLE DB provider after SQLNCLI was deprecated (SQLNCLI13 was the last version from Microsoft).

As earlier announced, as of v0.9.33 the database schema is developed independently of the MDA and the latest schema including upgrade scripts is available on our ftp server under the documents section  (MDA_schemas_[version].zip). As of this writing the latest database schema is v94 and is available from ftp://ftp.millistream.com/documents/MDA_schema_94.zip

Do note however that the latest schemas are only needed if you need the new columns/tables, they are not mandatory to operate the MDA!

We consider MDA 0.9.42 to be the best version of MDA available and we recommend that users of older versions upgrade as soon as possible.

MDA 0.9.42 is available for download from the following locations:

Windows users upgrade by running the installer for the new version.

Linux users who have added our apt or yum repositories can upgrade by simply issuing:

  • Debian/Ubuntu: “apt update && apt upgrade“
  • Red Hat/CentOS/Amazon Linux AMI: “yum upgrade“

Information on how to add these repositories are available at ftp://millistream.com/binaries/linux/

And as usual, if you have any questions what so ever regarding this release or other things, don’t hesitate to contact us at tech@millistream.com

/by

Added Support for Ubuntu 18.10

Preparing for the upcoming release of Ubuntu 18.10 (Cosmic Cuddlefish) due to release on 2018-10-18 we have updated our DEB repository with compatible versions of all our software products.

For fresh installs all you have to do is to add our repository and our signing key:

sudo wget "https://packages.millistream.com/apt/sources.list.d/cosmic.list" -O /etc/apt/sources.list.d/millistream.list
wget -q "https://packages.millistream.com/D2FCCE35.gpg" -O- | sudo apt-key add -
sudo apt update

Or if you are upgrading from a previous version of Ubuntu you can simply re-enable the repository (the upgrade process in Ubuntu/Debian disables all external repositories) and fetch the new binaries:

sudo wget "https://packages.millistream.com/apt/sources.list.d/cosmic.list" -O /etc/apt/sources.list.d/millistream.list
sudo apt update
sudo apt upgrade

/by

Added Support for Ubuntu 18.04LTS

Preparing for the upcoming release of Ubuntu 18.04LTS (Bionic Beaver) due to release on 2018-04-26 we have updated our DEB repository with compatible versions of all our software products.

For fresh installs all you have to do is to add our repository and our signing key:

sudo wget "https://packages.millistream.com/apt/sources.list.d/bionic.list" -O /etc/apt/sources.list.d/millistream.list
wget -q "https://packages.millistream.com/D2FCCE35.gpg" -O- | sudo apt-key add -
sudo apt update

Or if you are upgrading from a previous version of Ubuntu you can simply re-enable the repository (the upgrade process in Ubuntu/Debian disables all external repositories) and fetch the new binaries:

sudo wget "https://packages.millistream.com/apt/sources.list.d/bionic.list" -O /etc/apt/sources.list.d/millistream.list
sudo apt update
sudo apt upgrade

/by

Upgraded feed access point in Amazon EC2

When Amazon applied patches to their EC2 infrastructure to secure it against the Meltdown vulnerability in their Intel processors on December 2017 we experienced a large performance degradation of our feed access point in AWS EC2 Ireland.

Unfortunately it turns out that network heavy distribution applications such as our feed access point is the type of application that are the most affected by the Meltdown mitigations, experiencing up to 50% performance degradation.

To mitigate this we have upgraded our feed access point in AWS EC2 to the instance type that have the highest performance available. Unfortunately upgrading to this instance type means that the old IP address cannot be retained (unfortunately the IP address of the old access point was not an Elastic address, but the new one is) and thus all users of the aws.millistream.com access point must open for port 9000/9100 (depending on if receiving delayed and/or realtime data) to IPv4 address 52.51.190.39.

The CNAME in our DNS servers for aws.millistream.com will change from the current value of 54.195.186.72 to 52.51.190.39 at 2018-02-03.

Valid feed access points for the Millistream Data Feed (MDF):

Address 	        Provider        Location
sth2.millistream.com    IP-Only         Stockholm, Sweden 
sth.millistream.com 	Verizon 	Stockholm, Sweden
lon.millistream.com 	C4L 	        London, UK
aws.millistream.com 	Amazon EC2 	Dublin, Ireland
/by

Millistream Database Application (MDA) version 0.9.41 released

The Millistream MDA project team is pleased to announce the release of version 0.9.41 of our open source application for updating SQL servers with data from the Millistream Data Feed.

  • Support for new table “underlyings” from SQL Schema v90. Instruments have started to use multiply underlying id:s so the current model with a single column in “instruments” does no longer suffice. To remedy this we have added a new table “underlyings” which will have one row per insref and underlying (as ISIN as usual). When we will start to send out multiple underlyings the “instruments.underlyingid” can contain multiple ISINs separated by a pipe “|” character which is hard to deocde using SQL so this new table is much better to use.
  • Support for new table “priip” from SQL Schema v90. This is the “European PRIIPs Template (EPT) v1.1” table. Please not that access to this data requires additional entitlements.
  • Support for new table “mifid” from SQL Schema v90. This is the “European MiFID Template (EMT) v1.0” table. Please not that access to this data requires additional entitlements.
  • Log the SQL Server error code on sql errors for easier troubleshooting
  • Made the source possible to compile under MSYS2 on Windows
  • The Windows Installer includes the latest version of libmdf (v1.0.21) needed to deflate the MMT column in the “trades” table (a new column from SQL Schema v90)

As earlier announced, as of v0.9.33 the database schema is developed independently of the MDA and the latest schema including upgrade scripts is available on our ftp server under the documents section (MDA_schemas_[version].zip). As of this writing the latest database schema is v90 and is available from ftp://ftp.millistream.com/documents/MDA_schema_90.zip.

Do note however that the latest schemas are only needed if you need the new columns/tables, they are not mandatory to operate the MDA!

We consider MDA 0.9.41 to be the best version of MDA available and we recommend that users of older versions upgrade as soon as possible.

MDA 0.9.41 is available for download from the following locations:

Windows users upgrade by running the installer for the new version. Linux users who have added our apt or yum repositories can upgrade by simply issuing:

  • Debian/Ubuntu: “apt update && apt upgrade“
  • Red Hat/CentOS/Amazon Linux AMI: “yum upgrade“

Information on how to add these repositories are available at ftp://ftp.millistream.com/binaries/linux/.

And as usual, if you have any questions what so ever regarding this release or other things, don’t hesitate to contact us at tech@millistream.com.

/by

Millistream Data Feed API (MDF) version 1.0.21 released

Millistream is pleased to announce the release of version 1.0.21 of our open source Data Feed API.

  • updated mdf_fields.h to the latest version (82 new fields)
  • two new MiFID2 messages MDF_M_PRIIP (EPT) and MDF_M_MIFID (EMT)
  • performance improvement of numeric field decoding (up to 113% increase)
  • automatically deflate the MDF_F_MMT string

As usual this version of MDF is binary compatible with previous versions, so there is no need to recompile client applications, so users (including users of the Java JNI wrapper) can simply upgrade by replacing the libmdf file itself.

MDF 1.0.21 is available for download from the following locations:

Linux users who have added our apt or yum repositories can upgrade by simply issuing:

  • Debian/Ubuntu: “apt-get update && apt-get upgrade“
  • Red Hat/CentOS/Amazon Linux AMI: “yum upgrade“

Information on how to add these repositories are available at
ftp://ftp.millistream.com/binaries/linux/

And as usual, if you have any questions what so ever regarding this release or other things, don’t hesitate to contact us at tech@millistream.com.

/by