recon -- Web and network friendly simulation data formats

Michael Tiller
Xogeny Inc., USA

Peter Harman
CyDesign Ltd., UK

Ladda ner artikelhttp://dx.doi.org/10.3384/ecp140961081

Ingår i: Proceedings of the 10th International Modelica Conference; March 10-12; 2014; Lund; Sweden

Linköping Electronic Conference Proceedings 96:113, s. 1081-1093

Visa mer +

Publicerad: 2014-03-10

ISBN: 978-91-7519-380-9

ISSN: 1650-3686 (tryckt), 1650-3740 (online)


There are many different commonly used file formats for storing time series data. Most of these file formats are designed with the assumption that the file itself will be locally available to the software that will be reading or writing the data stored in them. While this assumption is an excellent one for desktop based tools with direct access to disk drives capable of moving virtually instantaneously around from sector to sector; there are a growing number of applications for which local access is not necessarily available. For these applications; we’ve initiated the recon project to develop more suitable formats.

With the emergence of web and cloud based modeling and simulation technologies; the time has come to explore file formats specifically optimized for nondesktop applications. In this paper; we present a new set of file formats that are specifically designed for web and cloud based approaches. This paper reviews the key requirements for web and cloud enabled applications and then presents a specification for two file formats that address those requirements.

When considering the various use cases that drove our requirements; we recognized that two different file formats were actually required. The first format; the wall format; is optimized for writing. The other format; the meld format; is optimized for reading over a network (i.e.; minimizing the number of reads and bytes read). We will discuss the layout of each of these formats and describe the use cases for which they are most appropriate.

In the open tradition of the Modelica Association; the authors have made specifications and implementations for these formats available as open source libraries with the hope that they will benefit the community as a whole.


Modelica; FMI; simulation results; cloud; web; open source


[1] A. Pfeiffer, I. Bausch-Gall, and M. Otter. “Proposal for a Standard Time Series File Format in HDF5”. In: Proceedings of the 9th International Modelica Conference (2012). URL: http://www.bausch-gall.de/ecp12076495_ PfeifferBausch-GallOtter.pdf.

[2] Jörg Rädler. “DyMat - HDF5 export and other Modelica/Python projects”. In: (2013). URL: http://www.j-raedler.de/2013/01/dymat-hdf5-export-and-comparableprojects/.

[3] The HDF Group. HDF-Java 2.10 release. 2013. URL: http://www.hdfgroup.org/products/java/hdf-java-html/.

[4] Google. Google Protocol Buffers. 2012. URL: https://developers.google.com/protocol-buffers/.

[5] Apache Avro Project Members. Apache Avro 1.7.6. 2013. URL: https://avro.apache.org/.

[6] Apache Thrift Project Members. Apache Thrift 0.9.1. 2013. URL: https://thrift.apache.org/.

[7] Apache Hadoop Project Members. Apache Hadoop 2.2.0. 2013. URL: https://hadoop.apache.org/.

[8] Storm Development Team. STORM: Distributed and fault-tolerant realtime computation. 2013. URL: http://storm-project.net/.

[9] Sadayuki Furuhash. MessagePack - It’s like JSON but fast and small. 2013. URL: http://msgpack.org/.

[10] R. Fielding et al. Hypertext Transfer Protocol - HTTP/1.1. 1999. URL: http://www.w3.org/Protocols/rfc2616/rfc2616.html.

[11] JSON. JavaScript Object Notation. 2013. URL: http://www.json.org/.

[12] MongoDB Development Team. BSON Specification. 2013. URL: http://bsonspec.org.

[13] Inc. MongoDB. MongoDB Manual, Version 2.4. 2014. URL: http://docs.mongodb.org/manual/.

[14] Michael Tiller. recon – Web and network friendly simulation data formats, Python Interface. 2013. URL: https://github.com/xogeny/recon.

[15] Jackson JSON processor project team. Smile Format Specification. 2013. URL: http://wiki.fasterxml.com/SmileFormatSpec.

[16] Pietrzak Roman. Binary-JSON (BJSON) format specification. 2013. URL: http://bjson.org/.

[17] Riyad Kalla. Universal Binary JSON Specification. 2013. URL: http://ubjson.org.

[18] Julian Seward. Bzip2 Homepage. 2010. URL: http://www.bzip.org/.

[19] Peter Harman. recon – Web and network friendly simulation data formats, C Interface. 2014. URL: https://github.com/harmanpa/crecon.

[20] Peter Harman. jrecon – Web and network friendly simulation data formats, Java Interface. 2014. URL: https://github.com/harmanpa/jrecon.

Citeringar i Crossref