Msgcore — the structured message store

One heap.
Every field addressed.
Nothing to serialise.

Msgcore is a hierarchical store for structured data, addressable by offset or by field name — like files in a micro file system. No serialisation pass required. All data elements live in one contiguous memory image: send it over the network, save it to a file or to S3. Msgcore has been doing this since 2000.

The difference from a serialised format is what happens after the write. A serialiser emits a finished byte stream: adding a field means changing the schema, regenerating the writer and rewriting the message. Here the image is the live structure, so a store already on disk takes new fields at any point, vectors grow when you insert past their end, and the heap reallocates its base image to fit. And what will not fit need not be resident: a manager can hand a subtree’s residency to its host, which pages it in before it is read and out when it is done — so a store can describe far more data than it holds.

Watch the heap grow → Component reference Libraries & downloads

One heap · offset addressing · the same image on disk and on the wire

2000Store in service since

0Serialisation passes to save

282Flat C entry points

1Heap per store, one image

Just a few lines, and the store is on disk

The whole idea

One store, several ways to fill and read it. If you know the fields up front, declare them once and write obj->window->font->size = 11 — static access that reads like PHP. If you don’t, reach fields at run time — dynamic access by name, by function call or by path. Or build the store straight from JSON or XML. Every tab makes the same tree.

Static struct

Dynamic

From text

Six nodes, one heap · each reachable by call, by path or by field

#include "P2PmsgMgr.h"
#include "MsgFieldRef.hpp"

struct Font     : MsgView { MSG_FIELD ( size, int ); };
struct Window   : MsgView { MSG_FIELD ( x, int );
                            MSG_FIELD ( y, int );
                            MSG_FIELD_NODE ( font, std::wstring, Font ); };
struct Settings : MsgView { MSG_FIELD_NODE ( window, std::wstring, Window ); };

P2PmsgMgr mgr;                       // 1
mgr.r_name() = L"Settings";          // 2
MsgViewOf<Settings> obj ( mgr );
obj->window = L"";                   // 3
obj->window->x = 1240;               // 4
obj->window->y = 820;                // 5
obj->window->font = L"Consolas";     // 6  font holds a value
obj->window->font->size = 11;        // 7  and has children too
mgr.Save ( L"settings.p2p" );        // 8

P2PmsgMgr load ( L"settings.p2p" );  // 9
MsgViewOf<Settings> cfg ( load );
int nX = cfg->window->x;             // 10 -> 1240
int nPt = cfg->window->font->size;   //    -> 11

// MSG_FIELD is a leaf. MSG_NODE is a branch, and MSG_FIELD_NODE is a branch
// that also holds a value, so obj->f1->f2->f3 goes as deep as the structs
// do, with a type of its own at every level. Each -> is one more step in a
// path walked fresh on every access: a write creates what is missing, and a
// read creates nothing. A member name and its L"x" come from one identifier,
// x = L"wide" does not compile, and a name past 63 characters fails at
// compile time. Header-only: no exported class changed shape.

#include "P2PmsgMgr.h"
#include "MsgFieldRef.hpp"                                      // header-only

P2PmsgMgr mgr;                                                  // 1
mgr.r_name() = L"Settings";                                     // 2
Field ( mgr, L"window" ) = L"";                                 // 3
Field ( mgr, L"window" )[L"x"] = 1240;                          // 4  stored INT32
Field ( mgr, L"window" )[L"y"] = 820;                           // 5
Field ( mgr, L"window" )[L"font"] = L"Consolas";                // 6
Field ( mgr, L"window" )[L"font"][L"size"] = 11;                // 7
mgr.Save ( L"settings.p2p" );                                   // 8

P2PmsgMgr load ( L"settings.p2p" );                             // 9
int nX  = Field ( load, L"window" )[L"x"].AsInt();              // 10 -> 1240
int nPt = Field ( load, L"window" )[L"font"][L"size"].AsInt();  //    -> 11

// Each write is DeclareItem(..., TRUE): create or update, so line 7 alone
// would have made "window" and "font" on the way. One operator= per type,
// never a template, and a narrow "text" is taken as UTF-8 and stored
// UTF-16. A read creates nothing, and throws on a missing field or a
// different type.

#include "P2PmsgMgr.h"

P2PmsgMgr mgr;                                                    // 1
mgr.r_name() = L"Settings";                                       // 2
P3PmsgField& win = mgr.DeclareItem ( L"window",
                                   P3PmsgData ( L"" ) );          // 3
win.DeclareItem ( L"x", P3PmsgData ( (int)1240 ) );               // 4
win.DeclareItem ( L"y", P3PmsgData ( (int)820 ) );                // 5
P3PmsgField& font = win.DeclareItem ( L"font",
                                   P3PmsgData ( L"Consolas" ) );  // 6  a value,
font.DeclareItem ( L"size", P3PmsgData ( (int)11 ) );             // 7  and a child
mgr.Save ( L"settings.p2p" );                                     // 8

P2PmsgMgr load ( L"settings.p2p" );                               // 9
int nX = load.SelectItem ( L"window" )                            // 10 step,
            .SelectItem ( L"x" ).c_int();                         //    then step  -> 1240
int nPt = load.SelectItem ( L"window" ).SelectItem ( L"font" )
              .SelectItem ( L"size" ).c_int();                    //    -> 11

#include "P2PmsgMgr.h"

P2PmsgMgr mgr;                                            // 1
mgr.r_name() = L"Settings";                               // 2
mgr.DeclareItem ( L"window", P3PmsgData ( L"" ) );        // 3
P3PmsgField win ( mgr.RootPath2Object
                      ( L".Settings.window" ) );          //    reached by path
win.DeclareItem ( L"x", P3PmsgData ( (int)1240 ) );       // 4
win.DeclareItem ( L"y", P3PmsgData ( (int)820 ) );        // 5
win.DeclareItem ( L"font", P3PmsgData ( L"Consolas" ) );  // 6
P3PmsgField font ( mgr.RootPath2Object
                       ( L".Settings.window.font" ) );    //    one level deeper
font.DeclareItem ( L"size", P3PmsgData ( (int)11 ) );     // 7
mgr.Save ( L"settings.p2p" );                             // 8

P2PmsgMgr load ( L"settings.p2p" );                       // 9
P3PmsgField x ( load.RootPath2Object                      // 10 one string,
                    ( L".Settings.window.x" ) );          //    whole descent
int nX = x.c_int();                                       //    -> 1240
P3PmsgField pt ( load.RootPath2Object
                     ( L".Settings.window.font.size" ) );
int nPt = pt.c_int();                                     //    -> 11

// A path SELECTS an existing node; DeclareItem is what creates one, which
// is why lines 3-7 still call it. A missing dotted descendant throws
// "Path to object does not exist" rather than answering empty.

// The same store as a JSON document. MsgcoreUtils writes it with
// oMgr >> oJson and reads it back with oMgr << oJson -- the arrow
// always points at the thing being written.

// A plain field renders as its value; a field that also carries children
// renders as an object, with its own value under ".value". An item name
// can hold neither '.' nor '@', so the sentinel keys cannot collide with
// a descendant -- that is a property of the store, not of the renderer.

// The constructor seeds the renderer with the DOCUMENT, the way SetText
// does. It is explicit, because Load() takes a FILENAME and both take an
// LPCTSTR -- an implicit one would parse a path as JSON.

#include "MsgcoreUtils/PrintJson.h"

PrintJson oJson ( LR"({
  "Settings": {
    "window": {
      ".value": "",
      "x": 1240,
      "y": 820,
      "font": {
        ".value": "Consolas",
        "size": 11
      }
    }
  }
})" );

P2PmsgMgr mgr;
mgr << oJson;                         // value, attributes and descendants are
                                      // replaced -- the NAME is not. "Settings"
                                      // names the DOCUMENT, not the node, so the
                                      // wrapper key is dropped and the store keeps
                                      // the root name it already had. Address it
                                      // relatively, as below, or name the store
                                      // yourself if you want rooted paths.
if ( *oJson.GetError() )
    printf ( "line %d: %s\n", oJson.GetErrorLine(), oJson.GetError() );

P3PmsgField x ( mgr.Path2Object ( L"window.x" ) );
x.c_int64 ();                          // 1240 -- int64, NOT int32. JSON has one
                                      // number type, so c_int() would throw.
P3PmsgField pt ( mgr.Path2Object ( L"window.font.size" ) );
pt.c_int64 ();                         // 11, and "font" itself reads "Consolas"

// Nothing is written until the whole document has parsed: a failure
// leaves the target exactly as it was and says where it stopped.

// The same store as XML. MsgcoreUtils writes it with oMgr >> oXML and
// reads it back with oMgr << oXML -- the arrow always points at the
// thing being written. The node's own value rides in <p2p:value>, the
// sentinel the JSON tab spells ".value".

// This is the dialect that keeps the WIDTH. p2p:type travels on every
// cell that has one, so an int32 comes back an int32 -- where JSON has
// one number type and hands back an int64. SetTypes(false) drops the
// attributes for a foreign consumer and buys back exactly that bargain.

// An element name is folded to what XML allows, so a name with a space
// in it travels in p2p:name beside it. The real name is IN the
// document rather than guessed at from the element, which is what
// makes the read direction exact instead of a best effort.

#include "MsgcoreUtils/PrintXML.h"

PrintXML oXML ( LR"(<?xml version="1.0" encoding="UTF-8"?>
<Settings xmlns:p2p="urn:ivyware:msgcore:p2p">
  <window>
    <p2p:value p2p:type="WSTR16"></p2p:value>
    <x p2p:type="int32">1240</x>
    <y p2p:type="int32">820</y>
    <font>
      <p2p:value p2p:type="WSTR16">Consolas</p2p:value>
      <size p2p:type="int32">11</size>
    </font>
  </window>
</Settings>)" );

P2PmsgMgr mgr;
mgr << oXML;                          // value, attributes and descendants are
                                      // replaced -- the NAME is not. <Settings>
                                      // names the DOCUMENT, not the node, so the
                                      // store keeps the root name it already had.
if ( *oXML.GetError() )
    printf ( "line %d: %s\n", oXML.GetErrorLine(), oXML.GetError() );

P3PmsgField x ( mgr.Path2Object ( L"window.x" ) );
x.c_int ();                           // 1240 -- int32, as it was stored.
                                      // The declaration is optional on the
                                      // way in: a reader that believed the
                                      // flag would refuse valid documents.
P3PmsgField pt ( mgr.Path2Object ( L"window.font.size" ) );
pt.c_int ();                          // 11 -- int32 again, one level deeper.

Line by line · Msg tree
	Node	What it is
—	`.Settings`	The root field, named by `r_name()`. It has children and no data cell of its own, which is why its value column is a dash.
—	`.Settings.window`	A value and a parent at once: an empty WSTR16 string, with `x`, `y` and `font` under it. Nothing in the store declares a node as a branch or a leaf.
—	`.Settings.window.x / .y`	Two INT32 leaves. The tag is stored beside each value, so a reader asking for the wrong type is refused rather than handed reinterpreted bytes.
—	`.Settings.window.font`	The same shape as `window` one level down: the face name `L"Consolas"` as its own WSTR16 cell, and `size` under it.
—	`.Settings.window.font.size`	An INT32 leaf three levels below the root. Every tab reaches it in its own way; the path column is the one spelling that works everywhere.
—	`settings.p2p`	One contiguous image. Every node above is an offset inside it, never a pointer, so the same bytes are the file, the wire frame and the live store.

Line by line · Typed view
#	Command	What it does
—	`struct Window : MsgView { MSG_FIELD ( x, int ); … }`	Declares the names and their types once, at compile time. `MSG_FIELD` makes the member and its `L"x"` name from one identifier, so the two cannot drift, and a name past 63 characters fails to compile. Nothing is stored by declaring.
—	`MSG_FIELD_NODE ( font, std::wstring, Font )`	A member that holds a value and names a view of its children. `MSG_NODE ( id, View )` is the same without a value. Child views can declare nodes too, so `obj->f1->f2->f3` goes as deep as the structs do.
1	`P2PmsgMgr mgr;`	Constructs the manager: one heap, one root field, nothing on disk yet. Every tab starts here.
2	`mgr.r_name() = L"Settings";`	Names the root field — the first component of every rooted path into the store.
—	`MsgViewOf<Settings> obj ( mgr );`	Binds the view to the store. The view holds names, never item references, and walks the path afresh on every access — the price of the `->` syntax, and what keeps it safe when the tree moves.
3–7	`obj->window->font->size = 11;`	Each `->` adds one name to the path; the assignment is create-or-replace, so missing levels are made on the way. A member takes only its declared type: `obj->window->x = L"wide"` does not compile.
8	`mgr.Save ( L"settings.p2p" )`	Writes the heap as it stands: no serialisation pass, no pointer fix-up, and a temporary file renamed over the target so no reader sees half a store.
9	`P2PmsgMgr load ( L"settings.p2p" );`	Loads the image back as the bytes it was. Nothing about the reading side knows which tab wrote it — all four write the same tree.
10	`int nX = cfg->window->x;`	Reads by conversion to the declared type. A missing field or a different stored tag throws `P2Pevent*`, and a read creates nothing. The view `->` hands out is a temporary; keep one with `MsgViewOf<Window> w ( cfg->window.Anchor() )`.

Line by line · By field name
#	Command	What it does
—	`#include "MsgFieldRef.hpp"`	Header-only. No export, no class layout change, so nothing downstream rebuilds to use it.
1	`P2PmsgMgr mgr;`	Constructs the manager: one heap, one root field, nothing on disk yet. Every tab starts here.
2	`mgr.r_name() = L"Settings";`	Names the root field — the first component of every rooted path into the store.
3	`Field ( mgr, L"window" ) = L"";`	A reference to a field by name, written on `=`. There is one `operator=` per type and no template, so a `char*` cannot slip into the binary-blob overload; a narrow string is taken as UTF-8 and stored UTF-16.
4–7	`Field ( mgr, L"window" )[L"font"][L"size"] = 11;`	Each `[]` extends the path. The write is `DeclareItem ( …, TRUE )`: create or replace, type included, and missing levels are created on the way — line 7 alone would have made `window` and `font`.
8	`mgr.Save ( L"settings.p2p" )`	Writes the heap as it stands: no serialisation pass, no pointer fix-up, and a temporary file renamed over the target so no reader sees half a store.
9	`P2PmsgMgr load ( L"settings.p2p" );`	Loads the image back as the bytes it was. Nothing about the reading side knows which tab wrote it — all four write the same tree.
10	`Field ( load, L"window" )[L"x"].AsInt()`	One reader per type — `AsInt`, `AsInt64`, `AsReal`, `AsText` and the rest — none of which widens or narrows. A missing field or a different tag throws; a read creates nothing.

Line by line · By function
#	Command	What it does
1	`P2PmsgMgr mgr;`	Constructs the manager: one heap, one root field, nothing on disk yet. The `(uAddrNN, nSizeInitial, nSizeMax)` overload picks the addressing width and the growth bounds instead — `VBLock_Addr64` for anything that may get large, `Addr16` / `Addr32` to buy compactness at a size ceiling.
2	`mgr.r_name() = L"Settings";`	Names the root field. Every `r_` accessor hands back a reference, so one call both reads and writes. This name is the first component of every root path into the store.
3	`mgr.DeclareItem ( L"window", P3PmsgData ( L"" ) )`	Adds a child and returns a reference to it. The descendant collection is created on demand, so there is no separate “make a node” step and no declared shape to keep in sync. A third argument of `TRUE` re-declares in place instead of adding a second field under the same name.
4–5	`win.DeclareItem ( L"x", P3PmsgData ( (int)1240 ) )`	Two typed leaves under `window`. `P3PmsgData` takes its type tag from the C++ type of the argument, and every `c_` accessor checks that tag: `c_int()` on a `double` cell throws, it does not silently truncate.
6–7	`win.DeclareItem ( L"font", P3PmsgData ( L"Consolas" ) )`	A field can hold a value and have children at the same time: `font` carries the face name in its own WSTR16 cell and `size` under it, the same way `window` carries `L""` above `x` and `y`. Nothing is declared as a branch or a leaf; a node becomes a parent by being given a child.
8	`mgr.Save ( L"settings.p2p" )`	Writes the heap. It is already contiguous and already self-describing, so this is a copy of bytes — no serialisation pass, no pointer fix-up. It goes to a temporary file and renames over the target, so a reader never sees a half-written store. `Save()` with no filename saves over the file the store came from; `Save ( path, true )` defragments first.
9	`P2PmsgMgr load ( L"settings.p2p" );`	The constructor-from-filename is `Load()` — shared, header-validated. The image comes back as the bytes it was: `load.Sizeof()` equals what was saved.
10	`load.SelectItem ( L"window" ).SelectItem ( L"x" ).c_int()`	One step per call. `SelectItem` steps one level and hands back a reference into the tree, so a write through it lands in the tree. `RootPath2Object` takes the whole descent as one rooted string; `Path2Object` is the same thing relative to the manager, so `L"window.x"` reaches the same leaf. All of them end at one heap offset — `GetP2Pos()` on any of them answers the same `P2Pos`. Two things the parameter names hide: `RootPath2Object` requires the leading `.` and the root’s own name, while `Path2Object` takes the path relative to the manager; and of the three calls whose argument is spelt `…Name`, `SelectObject` and `Exists` take a whole path — only `SelectItem` takes a bare name.

Line by line · By path
#	Command	What it does
1	`P2PmsgMgr mgr;`	Constructs the manager: one heap, one root field, nothing on disk yet. Every tab starts here.
2	`mgr.r_name() = L"Settings";`	Names the root field — the first component of every rooted path into the store.
3	`mgr.DeclareItem ( L"window", P3PmsgData ( L"" ) )`	A path selects a node; it does not create one. So the nodes are still made with `DeclareItem`, exactly as in By function.
—	`mgr.RootPath2Object ( L".Settings.window" )`	Reaches a node by one rooted string. The leading `.` and the root’s own name are required; `Path2Object` takes the same path relative to the manager, as `L"window"`.
4–7	`font.DeclareItem ( L"size", P3PmsgData ( (int)11 ) )`	Children declared on the node the path reached. `P3PmsgData` takes its type tag from the C++ type of the argument.
8	`mgr.Save ( L"settings.p2p" )`	Writes the heap as it stands: no serialisation pass, no pointer fix-up, and a temporary file renamed over the target so no reader sees half a store.
9	`P2PmsgMgr load ( L"settings.p2p" );`	Loads the image back as the bytes it was. Nothing about the reading side knows which tab wrote it — all four write the same tree.
10	`load.RootPath2Object ( L".Settings.window.x" )`	The whole descent in one string, and the same heap offset as the stepwise `SelectItem` chain — `GetP2Pos()` answers the same `P2Pos`. A missing component throws “Path to object does not exist” rather than answering empty.

Line by line · From JSON
#	Command	What it does
—	`#include "MsgcoreUtils/PrintJson.h"`	The JSON renderer from MsgcoreUtils, a separate library over Msgcore. `oMgr >> oJson` writes a store out; `oMgr << oJson` reads one in — the arrow points at what is written.
—	`PrintJson oJson ( LR"({ … })" );`	Seeds the renderer with the document, as `SetText` does. The constructor is explicit because `Load()` takes a filename of the same type.
—	`"font": { ".value": "Consolas", "size": 11 }`	A field that also has children renders as an object, its own value under `".value"`. An item name cannot hold `.` or `@`, so the sentinel never collides with a child.
—	`mgr << oJson;`	Parses the whole document first, then replaces value, attributes and descendants. The root name is not replaced: `"Settings"` names the document, so the wrapper key is dropped.
—	`oJson.GetError()`	Empty on success. On failure nothing was written, and `GetErrorLine()` says where parsing stopped.
—	`mgr.Path2Object ( L"window.font.size" ).c_int64 ()`	JSON has one number type, so every number comes back INT64 and `c_int()` would throw. That is the width the XML tab keeps.

Line by line · From XML
#	Command	What it does
—	`#include "MsgcoreUtils/PrintXML.h"`	The XML renderer from MsgcoreUtils, with the same arrows as JSON: `oMgr >> oXML` writes, `oMgr << oXML` reads.
—	`<size p2p:type="int32">11</size>`	This dialect keeps the width: `p2p:type` travels on every cell, so an int32 comes back an int32. `SetTypes(false)` drops the attributes for a foreign consumer, and with them that guarantee.
—	`<p2p:value p2p:type="WSTR16">Consolas</p2p:value>`	A node’s own value, when it also has children — what the JSON tab spells `".value"`.
—	`p2p:name`	An element name is folded to what XML allows; a name XML cannot spell travels in `p2p:name` beside it, so reading back is exact rather than a guess.
—	`mgr << oXML;`	Parses the whole document, then replaces value, attributes and descendants. `<Settings>` names the document, so the store keeps its own root name. A failure writes nothing.
—	`mgr.Path2Object ( L"window.font.size" ).c_int ()`	An int32, as it was stored. The type declaration is optional on the way in: a reader that insisted on it would refuse valid documents.

Attributes and the stack

Two side-cars on every field

Besides its value and its children, a field carries two more collections, each with its own path delimiter — @ for attributes and ^ for the stack, alongside the . that descends into children. All three are declared together in P2Pmsg.h, and all three hold fields, so all three nest. The store is name-addressed — no node carries an id — so a path is how a node is named in text: in a log line, a config key, a FUSE filename, a COM argument.

Attributes — `@`

Metadata that must not turn up when something walks the content tree: a unit of measurement, a sensor id, provenance, a permission. r_Attr() is a second child collection on the same field, addressed with @ instead of ., so a scan for named children is never polluted by it. An attribute is an ordinary field, so it carries a typed value, its own children and its own attributes.

Through MsgFieldRef.hpp an attribute is reached the same two ways as a child. In a typed view, .Attrs() leads to the attributes, which are declared and checked at compile time. At run time, a name written with the _attr literal, as in [L"Unit"_attr], is the @ of a path.

Static struct

Dynamic

#include "P2PmsgMgr.h"
#include "MsgFieldRef.hpp"

struct UnitParts : MsgView { MSG_FIELD ( Symbol, std::wstring );
                             MSG_FIELD ( SI,     std::wstring );
                             MSG_FIELD ( Offset, double ); };
struct TempAttrs : MsgView { MSG_FIELD_NODE ( Unit, std::wstring, UnitParts );
                             MSG_FIELD ( Sensor, int ); };          // the @ side
struct TempDesc  : MsgView { MSG_FIELD ( Reading, double ); };     // the . side
struct Store     : MsgView { MSG_FIELD_NODE_ATTRS ( Temperature, double,
                                                    TempDesc, TempAttrs ); };

P2PmsgMgr mgr;
mgr.r_name() = L"Store";
MsgViewOf<Store> s ( mgr );
s->Temperature = 21.5;                             // .Store.Temperature
s->Temperature->Reading = 21.5;                    // .Store.Temperature.Reading
s->Temperature.Attrs()->Unit   = L"Celsius";       // .Store.Temperature@Unit
s->Temperature.Attrs()->Sensor = 7;                // .Store.Temperature@Sensor

// An attribute with structure of its own: a value, and children under it.
s->Temperature.Attrs()->Unit->Symbol = L"\u00B0C";  // @Unit.Symbol
s->Temperature.Attrs()->Unit->SI     = L"kelvin";
s->Temperature.Attrs()->Unit->Offset = 273.15;

int          nSensor = s->Temperature.Attrs()->Sensor;   // 7
std::wstring strUnit = s->Temperature.Attrs()->Unit;     // L"Celsius"
s->Temperature.Attrs()->Sensor.Erase ();
s->Temperature.Attrs()->Sensor.Exists ();              // false

// -> reaches the children and .Attrs()-> the attributes: two separate
// trees, the '.' and '@' of a path. Each _ATTRS macro is its plain twin
// plus Attrs(): MSG_FIELD_ATTRS (a value), MSG_NODE_ATTRS (children),
// MSG_FIELD_NODE_ATTRS (both). Attributes are as typed as any MSG_FIELD,
// so Attrs()->Sensor = L"x" does not compile. A read creates nothing; a
// write creates the field and its attribute collection.

#include "P2PmsgMgr.h"
#include "MsgFieldRef.hpp"
using namespace MsgLiterals;                                  // the _attr literal

P2PmsgMgr mgr;
mgr.r_name() = L"Store";
Field ( mgr, L"Temperature" )                 = 21.5;        // .Store.Temperature
Field ( mgr, L"Temperature" )[L"Reading"]     = 21.5;        // .Reading - a child
// L"Unit"_attr is a C++11 user-defined literal: MsgLiterals::operator""_attr
// makes the string a MsgAttrName, and [] given a MsgAttrName steps to '@'.
Field ( mgr, L"Temperature" )[L"Unit"_attr]   = L"Celsius";  // @Unit - an attribute
Field ( mgr, L"Temperature" )[L"Sensor"_attr] = 7;           // @Sensor, stored INT32

// An attribute with structure of its own: after _attr, [] goes on below it.
Field ( mgr, L"Temperature" )[L"Unit"_attr][L"Symbol"] = L"\u00B0C";   // @Unit.Symbol
Field ( mgr, L"Temperature" )[L"Unit"_attr][L"SI"]     = L"kelvin";
Field ( mgr, L"Temperature" )[L"Unit"_attr][L"Offset"] = 273.15;

int nSensor = Field ( mgr, L"Temperature" )[L"Sensor"_attr].AsInt ();  // 7
Field ( mgr, L"Temperature" )[L"Sensor"_attr] = 9;                     // update
Field ( mgr, L"Temperature" )[L"Sensor"_attr].Erase ();
Field ( mgr, L"Temperature" )[L"Nobody"_attr].Exists ();               // false

// An attribute of a field you hold, and a name chosen at run time:
P3PmsgField temp ( L"Temperature", P3PmsgData ( (double)21.5 ) );
Field ( temp, L"Unit"_attr ) = L"Celsius";                    // temp's own @Unit
std::wstring strName = L"Sensor";
Field ( temp, MsgAttrName ( strName.c_str() ) ) = 7;

// A plain L"x" in [] is a child and L"x"_attr is an attribute, so one chain
// walks both trees the way '.' and '@' do in a path; [L"x"_attr] is the
// same call as .Attr(L"x"). Each write is DeclareItem(..., TRUE) on the
// attribute collection, created on the first write: no AttrCMD_Create, no
// temporary field, no deep copy. A read creates nothing, and throws on a
// missing attribute or a different type.

P3PmsgField temp ( L"Temperature", P3PmsgData ( (double)21.5 ) );
temp.IsAttributed ();                           // false - no collection yet

temp.r_Attr ( P3PmsgField::AttrCMD_Create )
     += P3PmsgField ( L"Unit",   P3PmsgData ( L"Celsius" ) );
temp.r_Attr ()
     += P3PmsgField ( L"Sensor", P3PmsgData ( (int)7 ) );

temp.IsAttributed ();                           // true
temp.r_Attr().GetCount ();                      // 2
temp.r_Attr().Exists     ( L"Unit" );           // true
temp.r_Attr().SelectItem ( L"Unit" ).c_wstr();  // L"Celsius"

// The value is untouched, and the two collections are independent.
temp.c_double ();                               // 21.5
temp.DeclareItem ( L"Reading", P3PmsgData ( (double)21.5 ) );
temp.r_Desc().GetCount ();                      // 1
temp.r_Attr().GetCount ();                      // still 2 - separate trees

// (a) Declare the collection, then add to it. AttrCMD_Create is the house
//     style rather than a hard requirement - PushBack and DeclareItem both
//     create the block if it is absent.
temp.r_Attr ( P3PmsgField::AttrCMD_Create );
temp.r_Attr() += P3PmsgField ( L"Unit", P3PmsgData ( L"Celsius" ) );

// (b) DeclareItem on the collection - set-or-update in one call.
temp.r_Attr ( P3PmsgField::AttrCMD_Create )
     .DeclareItem ( L"Unit", P3PmsgData ( L"Celsius" ),  /*bUpdate*/ true );

// (c) Build the attribute as the field it is. A unit of measurement is a
//     good example of one that wants structure of its own.
P3PmsgField unit ( L"Unit", P3PmsgData ( L"Celsius" ) );
unit.DeclareItem ( L"Symbol", P3PmsgData ( L"\u00B0C" ) );
unit.DeclareItem ( L"SI",     P3PmsgData ( L"kelvin" ) );
unit.DeclareItem ( L"Offset", P3PmsgData ( (double)273.15 ) );
temp.r_Attr ( P3PmsgField::AttrCMD_Create ) += unit;     // deep copy

// (d) Hold the collection once, then work through it. Writes through a
//     reference land in the store.
P3PmsgAttr& attr = temp.r_Attr ();
attr += P3PmsgField ( L"Sensor", P3PmsgData ( (int)7 ) );
attr.SelectItem ( L"Sensor" ).c_int ( 9 );               // write, not read
attr.Delete ( L"Sensor" );
attr.Truncate ();                                        // empty the collection

P2PmsgMgr mgr;
mgr.r_name() = L"Store";
mgr.r_Desc() += temp;                                            // Temperature now lives in the store

mgr.RootPath2Object ( L".Store.Temperature"         );           // the field    -> 21.5
mgr.RootPath2Object ( L".Store.Temperature@Unit"    );           // the attribute -> "Celsius"
mgr.RootPath2Object ( L".Store.Temperature@Sensor"  );           // -> 7
mgr.RootPath2Object ( L".Store.Temperature.Reading" );           // a DESCENDANT, not an attribute

// A bare delimiter names the collection itself.
mgr.RootPath2Object ( L".Store.Temperature@" ).IsAttr ();        // true
mgr.RootPath2Object ( L".Store.Temperature." ).IsDesc ();        // true

// Relative to the manager, without naming the root:
mgr.Path2Object ( L"Temperature@Unit" );

// A miss on an '@' component answers void. A miss on a dotted descendant
// THROWS - "Path to object does not exist" - and so does a root path that
// does not start at the root.
mgr.RootPath2Object ( L".Store.Temperature@Nobody" ).IsVoid ();  // true

// Offset and path are the same address in two notations, and convert:
P3PmsgField& live = mgr.SelectItem ( L"Temperature" );
P2Pos   pos  = live.GetP2Pos ();                                 // an offset into the heap
CString path = live.GetPath   ();                                // the same node, as a string
mgr.P2Pos2Path ( pos ) == path;                                  // true

The stack — `^`

Every field carries its own stack. Push() copies the whole item — name, data, attributes and descendants — into a fresh block linked off the item; Pop() copies it back and drops the copy. Pushes nest. It is a scoped override: snapshot, mutate freely, then unwind, without holding a separate copy of your own. The snapshot keeps the name it was pushed under, so the name will not tell it from the live item — the handle will.

P3PmsgField quote ( L"Quote", DataBSTR08 ( L"100.25" ) );
quote.IsStacked ();                       // false

quote.r_Stck().Push ();                   // snapshot the WHOLE item
quote.IsStacked ();                       // true

quote = P3PmsgName ( L"Quote-Revised" );  // rename in place; the cell survives

if ( quote.IsStacked() )
    quote.r_Stck().Pop ();                // the override unwinds
quote == L"Quote";                        // true - and the copy is dropped

// (a) Pushes nest, and unwind last-in-first-out.
quote.r_Stck().Push ();
quote.r_Stck().Push ();
quote.r_Stck().Pop  ();                // back to the second snapshot
quote.r_Stck().Pop  ();                // back to the first
P3Pmsg_GetStckDepth ( &quote, 0 );     // how deep it is now

// (b) A snapshot is a whole item, not just its value: children and
//     attributes present at the push are inside it, later ones are not.
P3PmsgItem q ( L"Quote" );
q.r_Desc ( P3PmsgField::AttrCMD_Create );
q.r_Desc() += P3PmsgField ( L"Bid" );
q.r_Attr ( P3PmsgField::AttrCMD_Create ) += P3PmsgField ( L"Currency" );
q.r_Stck().Push ();                    // Bid and Currency go in with it
q.r_Desc() += P3PmsgField ( L"Ask" );  // added AFTER - live only

// (c) Read a pushed value back without popping - the stack keeps one
//     accessor per type, and each throws if asked for the wrong one.
q.r_Stck().r_name ();                  // the pushed NAME
q.r_Stck().r_data ();                  // the pushed VALUE
q.r_Stck().r_item ();                  // the pushed ITEM

// (d) += deep-copies into a collection, so a child you inserted is not the
//     live one. Fetch the live handle back out before pushing it.
mgr.r_Desc() += P3PmsgField ( L"BHP" );
P3PmsgField live = mgr.RootPath2Object ( L".Store.BHP" );
live.r_Stck().Push ();

// The item built in (b), addressed by path instead of by call.
P3Pmsg_SelectObject ( &q.r_Object(), L"^"          );                // the snapshot - name 'Quote'
P3Pmsg_SelectObject ( &q.r_Object(), L"^.Bid"      );                // Bid, as it stood
P3Pmsg_SelectObject ( &q.r_Object(), L"^Bid"       );                // same - the '.' is optional
P3Pmsg_SelectObject ( &q.r_Object(), L"^.Ask"      );                // void - added after the push
P3Pmsg_SelectObject ( &q.r_Object(), L"Ask"        );                // Ask - the live item has it
P3Pmsg_SelectObject ( &q.r_Object(), L"^^"         );                // the push before that
P3Pmsg_SelectObject ( &q.r_Object(), L"^@Currency" );                // == "@^Currency"

// The same shapes as a full root path. In this tree .Store.BHP was pushed
// after gaining a child 'Last', and .Store.RIO was never pushed at all.
mgr.RootPath2Object ( L".Store.BHP"          );                      // the live item
mgr.RootPath2Object ( L".Store.BHP@Currency" );                      // its attribute
mgr.RootPath2Object ( L".Store.BHP^"         );                      // its snapshot - a DIFFERENT object
mgr.RootPath2Object ( L".Store.BHP^.Last"    );                      // a child of the snapshot
mgr.RootPath2Object ( L".Store.RIO^"         );                      // void - RIO was never pushed

// Only the handle tells a snapshot from the live item: the two answer the
// same name, and IsVoid() is false for both.
P3PmsgField ( mgr.RootPath2Object ( L".Store.BHP"  ) ).GetP2Pos ();  // 259
P3PmsgField ( mgr.RootPath2Object ( L".Store.BHP^" ) ).GetP2Pos ();  // 1009

The path grammar
Path	Resolves to
`Item`	The child `Item` of the item you are standing on.
`Item.Child`	A descendant, one level down.
`Item@Tag`	The attribute `Tag`.
`Item/Child` · `Item\Child`	A descendant. `/` and `\` are accepted on input as synonyms for `.` — convenient when a path arrives from a file system.
`Item@` · `Item.`	The attribute collection, and the descendant collection, as objects in their own right.
`Item^`	`Item` as it stood before its last push.
`Item^^`	Before the push before that. One `^` too many answers void; it does not fail.
`Item^.Child` · `Item^Child`	A descendant of the snapshot. The `.` after a `^` is optional, as it is after a name.
`Item@Tag^`	A pushed attribute — attributes are items and carry stacks too.
`Item@^Tag` · `Item^@Tag`	The same object. `^` commutes with `@` and with `.`, because a snapshot is a whole item rather than a fragment of one.
`List@Tag` · `Vect@Tag`	A list and a vector are items, so they carry their own attributes and children.
`.Root.Item^`	Any of the above as a rooted path through `RootPath2Object`. `Path2Object` takes the same path relative to the manager.

Paging

Large datasets held outside the heap

A store and its data are normally the same thing: the tree is in the heap, the heap is the image, and the image is what you save. Paging breaks that identity for a chosen subtree. The tree still describes it — the nodes, the names, the shape are all there — but the bytes can live somewhere else entirely and arrive only when something reads them. Msgcore_c.h puts it in one line: a manager can delegate the residency of a subtree to its host, so a store may describe far more data than it holds.

Residency, delegated

You install two callbacks. When a paged dataset is about to be read, the core calls your page-in with the P2Pos of the item; you fetch it from wherever it really lives — a disk, a database, a socket — and answer 1 for resident or 0 for could not. When the work is finished it calls page-out, with a flag saying whether your copy needs writing back first. SafeDSetPaging is that same pair expressed as a scope, carrying the summary state built from PAGESUMM_DIRTY, CACHED, FLUSH, NOMERGE and DECACHE.

What it buys is that paths still resolve. The structure is addressable whether or not the bytes are home, so a consumer can walk it, search it and hand out addresses for nodes whose contents have never been fetched — and the fetch happens underneath, at the moment of the read, rather than being something the caller has to orchestrate.

#include "P2PmsgMgr.h"

// The host answers with the data. posItem names the node being asked
// about; nKey is the registration key the manager hands back to you.
BOOL CALLBACK PageIn  ( PINT_PTR nKey, P2Pos posItem );
BOOL CALLBACK PageOut ( PINT_PTR nKey, P2Pos posItem, BOOL bFlush );

P2PmsgMgr mgr ( L"catalogue.p2p" );
mgr.PageRegistration ( nKey, PageIn, PageOut );      // install the pair

mgr.PageDatasetIn  ( posRows );                    // fetch it now
mgr.PageDatasetOut ( posRows, /*bFlush*/ TRUE );   // write back, then drop

// Or let scope do it: in on construction, out on destruction.
{
    SafeDSetPaging oRows ( mgr, oRowsItem );        // resident from here
    oRows->r_Desc().GetCount ();                     // read it freely
}                                                    // ... to here

// Re-count after adding or removing elements, so the summary is honest.
mgr.PageSumm ( oRowsItem, /*additions*/ 12, /*removals*/ 0 );

#include "Msgcore_c.h"

// The same two callbacks, without the CALLBACK convention and without
// the key: the pUser you install is handed back to you unchanged.
static int PageIn  ( void* pUser, unsigned long long p2pos );
static int PageOut ( void* pUser, unsigned long long p2pos, int bFlush );

MsgMgrHandle hMgr = msgcore_mgr_open_file ( L"catalogue.p2p" );
msgcore_mgr_set_paging_sinks ( hMgr, PageIn, PageOut, pUser );

msgcore_mgr_page_dataset_in  ( hMgr, p2posRows );
msgcore_mgr_page_dataset_out ( hMgr, p2posRows, /*bFlush*/ 1 );

// 1 means "done, the data is resident"; 0 means "could not". With no
// sink installed both calls are successful no-ops, so paging costs
// nothing at all until you opt into it.

// The populate sink is the third one: it fills a subtree, rather than
// moving a whole dataset in and back out again.
msgcore_mgr_set_populate_sink ( hMgr, Populate, pUser );

// A Save walks the whole tree. With sinks installed, that walk would
// page the ENTIRE virtual dataset in just to write it out -- so the
// walk runs with the registration saved and cleared.
mgr.PageRegistrationPush ();      // save the sinks, clear them
mgr.Save ( L"catalogue.p2p" );    // nothing pages in
mgr.PageRegistrationPop  ();      // put them back

// The RAII form, which is what you want if anything between can throw.
{
    SafeRegistrationPush oQuiet ( &mgr );
    mgr.Save ( L"catalogue.p2p" );
}

// PUSH DOES NOT NEST. Pushing twice without an intervening pop asserts
// in a debug core, and loses the first saved set in a release one.

The contract — synchronous, and the core waits

This is not a background fetch. Msgcore_c.h is explicit about it: the core calls the sinks synchronously, on the thread performing the access, and waits for the answer — a page-in that has not returned yet is data that is not there, so there is no version of this that defers. A slow host is a stalled read, not a fetch in flight.

Two rules follow. A sink must not block for long, and it must not re-enter the manager that called it beyond the subtree it was asked to populate. The trigger sink’s contract is the opposite on every one of these points, and the header says in as many words not to model one on the other.

The paging surface
C++	Flat C	What it does
`PageRegistration ( key, in, out )`	`msgcore_mgr_set_paging_sinks`	Install the page-in / page-out pair. Passing null for both clears the registration and releases the record.
`PageRegistration ( key, populate )`	`msgcore_mgr_set_populate_sink`	Install the populate sink, which fills a subtree rather than moving a whole dataset.
`PageDatasetIn ( pos )`	`msgcore_mgr_page_dataset_in`	Ask the host for the dataset rooted at that node, now. A successful no-op when no sink is installed.
`PageDatasetOut ( pos, bFlush )`	`msgcore_mgr_page_dataset_out`	Hand it back — written out first if `bFlush`, simply dropped if not.
`PageSumm ( item, adds, removes )`	`msgcore_mgr_page_summ`	Re-count a paged item after a mutation; answers the new summary count.
`PageRegistrationPush ()`	`msgcore_mgr_page_registration_push`	Save the installed sinks and clear them, so a section of code runs with paging suppressed. Does not nest.
`PageRegistrationPop ()`	`msgcore_mgr_page_registration_pop`	Restore them.
`SafeDSetPaging`	—	The in/out pair as a scope: paged in by the constructor, out by the destructor, summary state carried between.
`SafeRegistrationPush`	—	Push and pop as a scope, for a suppressed section that can throw.

Watch the heap grow

Five fields go into one heap. The fifth does not fit, so the heap reallocates its base and copies — and every offset in the image is still correct, because none of them ever named an address. Then the whole thing is written to disk as the bytes it already is.

Caller code

Inside the manager

What the store gives you

Design properties

Addressing

Offsets, not pointers

Every reference inside the store is a displacement from the base of the heap. Growth reallocates that base — and every reference in the image is still correct, because none of them named an address in the first place.

Persistence

Save is a copy

The store is already contiguous and already self-describing, so writing it is writing bytes. Save goes to a temporary file and renames over the target, so a reader never sees a half-written store.

Shape

Fields, children, attributes

A field carries a name, a typed value, an ordered set of children and a set of @-qualified attributes. Native types, strings, blobs, XML and images all live in the one structure, addressed by field and by path.

Containers

Lists, vectors, stacks, cursors

The container types are not adapters over the store; they are laid out inside its heap. A cursor survives mutation of the ring it is scanning, which is what makes by-name traversal of a live store workable.

Portability

One image, two widths

An endian sentinel travels with every image and doubles as its layout generation, so a store written under one byte order or addressing width is recognised — rather than silently misread — by another.

Reach

C++, C, Java, COM

A C++ class API for code built with the same toolset, and a flat extern "C" ABI of 282 entry points for everyone else — C, .NET through P/Invoke, FUSE front ends. For Java all 282 are bound through java.lang.foreign, and a coverage test fails the build if the generated layer ever falls behind Msgcore_c.h. MsgFacade wraps the object model without macros, and MsgcoreCom puts a store one CreateObject away from PowerShell, VBScript and C#. See the facade page.

Diagnostics

One event system

P2Pevent carries errors and diagnostics as throwable C++ exceptions, or as data attached to a store — the same object Targetcore routes across a network.

Licence

Apache 2.0

Msgcore is licensed under the Apache 2.0 licence, alongside Targetcore. The integration libraries and reference applications are already free downloads with full source.

Scope

What it is not

The store is not internally synchronised — one store is serialised by its caller — and it has not been reviewed as a network-facing parser. Treat an image from a source you do not control as unsafe to load.

Path resolution is not on the flat C ABI. Msgcore_c.h carries the emitter, msgcore_mgr_p2pos2path, and no resolver — so a C, Panama or .NET caller can ask a node for its path, but addresses it back by P2Pos or by name. Path2Object and RootPath2Object are on the C++ classes, which are toolchain-pinned rather than a supported 1.x surface.

Two libraries, one lineage

How they fit

Msgcore and Targetcore are separate products with a one-way dependency. Msgcore knows nothing about networks; Targetcore does not define a payload format.

MsgcoreDefines the store, the fields and the P2Piomage frame. In service since 2000.

TargetcoreAddresses, encrypts and moves that frame between hubs. In service since 2002.

The ruleMsgcore does not depend on Targetcore. The layering is acyclic, and always has been.

In practiceThe message you build with Msgcore is the message a hub routes. There is no conversion step.

The type reference → Targetcore walkthrough

The pieces

Type model

Seven families carry the whole store. The full reference — the containers, the flat C ABI and the two rules a caller has to honour — is on the architecture page.

Msgcore type families
Type	Responsibility
`P2PmsgMgr`	The store manager. Owns the heap, the root field, the save and load paths, path lookup and triggers.
`P3PmsgField`	A named node: a value, an ordered set of children, and a set of attributes. The unit everything else is built from.
`P3PmsgData`	The typed value a field holds — native types, strings, BSTRs, blobs, XML and images.
`MsgVBHeap`	The offset-addressed heap and its block allocator: growth, the free list, and the addressing widths.
`MsgList` · `MsgVect` · `MsgStck` · `MsgCurs`	The container types over a field's children, laid out inside the heap rather than over it.
`P3PmsgBSTR` · `P2Piomage`	String storage, and the packed sync-header frame that leaves the process — the shape Targetcore encrypts and moves.
`P2Pevent`	Events and exceptions. Throwable, attachable to a store, and shared with Targetcore.

Full architecture → Downloads

Built on it

In production

Library

Targetcore

The routing kernel. Every message a hub queues, encrypts and dispatches is a Msgcore image; the kernel adds the address, the wire and the handler map.

See the walkthrough →

Application

Chartboard

Workspaces, chart stacks, portfolios and scanner results are stores — which is why a workspace file loads without a parse and a chart survives a round trip intact.

Explore Chartboard →

Reference server

AKAserva

An HTTP and WebSocket server where a parsed request is a field tree, decorated by handlers and translated back into a response.

One heap.Every field addressed.Nothing to serialise.