DAW JSON Link v2
Content
- Intro
- API Documentation - Member mapping classes and methods
- Cookbook Get cooking and putting it all together
- Arrays
- Classes
- Class from Array
- Dates
- Enums
- Graphs
- Key Values - Map and Dictionary like things
- Numbers
- Optional/Nullable Values
- Parsing Individual Members
- Parser Policies
- Strings
- Unknown JSON and Delayed Parsing - Browsing the JSON Document and delaying of parsing of specified members
- Variant
- Automatic Code Generation
- Intro
- Installing/Using
- Performance considerations
- Escaping/Unescaping of member names
- Differences between C++17 and C++20
- Using data types
- Error Handling
- Deserializing/Parsing
- Serialization
- Build Configuration Points
- Requirements
- Limitations
Intro
Top
The DAW JSON Link library provides multiple ways to serialization/deserialization JSON documents in C++. The primary one is parsing of JSON directly to your C++ data structures. This allows the known structure of the document to be exploited for greater checking and performance. Alternatively, there is an event passing(SAX) interface that can parse to generic types(double, string, bool,...) or can use the same type restricted parsers as the static parser previously mentioned. A generic DOM(lazy) based parser is provided that can be iterate over the document structure too, again it can use the generic parsers or the type based restricted versions. One can mix the three modes of parsing to form more complicated systems. For serialization, the first static mapping method is required, there is no json value type in the library. The library is, also, non-intrusive into your data structures and does not require member's to be declared/defined within them. This allows keeping the mapping in a separate header file from the data structures themselves.
The library is using the BSL licensed
When the structure of the JSON document is known, parsing is like the following:
MyThing thing = daw::json::from_json<MyThing>( json_string );
or for array documents, where the root of the document is an array, there is a helper method to make it easier and it can be parsed like the following:
std::vector<MyThing> things = daw::json::from_json_array<MyThing>( json_string2 );
If the structure of the JSON document is unknown, one can construct a json_value that acts as a container and allows iteration and parsing on demand. It is a lazy parser and will only parse when asked to. The following is an example of opening a json_value from JSON data:
json_value val = daw::json::json_value( json_string );
The from_json and to_json methods allow access most of the parsing needs.
The event based parser(SAX) can be called via daw::json::json_event_parser. It takes two arguments, a json document and an event handler. The event handler can opt into events by having the following members:
- handle_on_value
- handle_on_array_start
- handle_on_array_end
- handle_on_class_start
- handle_on_class_end
- handle_on_number
- handle_on_bool
- handle_on_string
- handle_on_null
- handle_on_error
Code Examples
- The Cookbook section has precanned tasks and working code examples
- Tests provide another source of working code samples.
- Some video walkthroughs
- Links to other examples
- Small samples below
Mapping of your classes to JSON documents is done by specializing the trait daw::json::json_data_contract. A class that is mapped does not need to be mapped again if it is a member of another mapped class. There are two parts to the trait json_data_contract, first is a type alias named type that maps the JSON members to our class's constructor. This gets around needing private access to the class, assuming that data we would serialize would also be needed to construct the class. For example:
struct Thing {
int a;
int b;
};
The construct for Thing requires 2 integers and if we had the following JSON:
{
"a": 42,
"b": 1234
}
We could do the mapping like the following:
namespace daw::json {
template<>
struct json_data_contract<Thing> {
static constexpr char const a[] = "a";
static constexpr char const b[] = "b";
using type = json_member_list<
json_number<a, int>,
json_number<b, int>
>;
};
}
This says that the JSON class will have at least two members "a", and "b" that will be numbers that are integers. They will be passed to the constructor of Thing when daw::json::from_json<Thing>( json_doc ); is called, or that another class has a json_class<MemberName, Thing> member mapping. The above is the C++17 mapping method for the names, it works in future C++ versions too. But, in C++20 and later the names can be inline in the mapping e.g. json_number<"a", int>. The above is all that is needed for parsing JSON, for serializing a static member function is needed in the trait. Taking the previous example and extending it we could serialize Thing with:
namespace daw::json {
template<>
struct json_data_contract<Thing> {
static constexpr char const a[] = "a";
static constexpr char const b[] = "b";
using type = json_member_list<
json_number<a, int>,
json_number<b, int>
>;
};
static auto to_json_data( Thing const & v ) {
return std::forward_as_tuple( v.a, v.b );
}
}
The ordering of the members returned as a tuple need to match the mapping in the type alias type. This allows for passing the result of accessor methods too, if the data members are not public.
- Note: The return type of
to_json_datadoes not have to return a tuple of references to the existing object members, but can return calculated values too.
The parsers work by constructing each argument in place in the call to the classes constructor. The individual argument parsers can be tuned for the specified circumstances of the data(e.g. floating point and integral numbers). Then with our type trait defining the arguments needed to construct the C++ class and their order we are able to look at each member in the JSON. Now we construct the value with the result of each parser; similar to T{ parse<0, json_string<"name">>( data ), parse<1, json_number<"age", unsigned>>( data ), parse<json_number<2, "number>>( data )}. For each member, the data stream will be moved forward until we find the member we need to parse, storing interested locations for later parsing. This process allows us to parse other classes as members too via the json_class<"member_name", Type> mapping type. So that each mapping trait only has to deal with it's specific members and not their details. 
Default mapping of types
Top
In unnamed contexts, such as the root value, array elements, some key value types, and variant element lists where the name would be no_name, one can use some native C++ data types instead of the the JSON mapping types. This includes, integer, floating point, bool, std::string, std::string_view, and previously mapped classes.
For example, to map an array of string's.
template<>
struct daw::json::json_data_contract<MyType> {
using type = json_member_list<json_array<"member_name", std::string>>;
};
Installing/Using
Top
Including in cmake project
To use daw_json_link in your cmake projects, adding the following should allow it to pull it in along with the dependencies:
include( FetchContent )
FetchContent_Declare(
daw_json_link
GIT_REPOSITORY https://github.com/beached/daw_json_link
GIT_TAG release
)
FetchContent_MakeAvailable(daw_json_link)
Then in the targets that need it:
target_link_libraries( MyTarget daw::json_link )
Installing
On a system with bash, it is similar on other systems too, the following can install for the system
git clone https://github.com/beached/daw_json_link
cd daw_json_link
mkdir build
cd build
cmake ..
cmake --install .
Testing
The following will build and run the tests.
git clone https://github.com/beached/daw_json_link
cd daw_json_link
mkdir build
cd build
cmake -DDAW_ENABLE_TESTING=On ..
cmake --build .
ctest .
After the build there the individual examples can be tested too. city_test_bin requires the path to the cities JSON file.
./tests/city_test_bin ../test_data/cities.json
Performance considerations
Top
The order of the members in the data structures should generally match that of the JSON data. The parser is faster if it doesn't have to back track for values. Optional values, when missing in the JSON data, can slow down the parsing too. If possible have them sent as null. The parser does not allocate. The parsed to data types may and this allows one to use custom allocators or a mix as their data structures will do the allocation. The defaults for arrays is to use the std::vector and if this isn't desirable, you must supply the type.
Benchmarks
- Kostya results using test_dawjsonlink.cpp See Kostya Benchmarks for latest results.
Escaping/Unescaping of member names
Top
The library, currently, does not unescape/escape member names when serializing, they are expected to be valid and unescaped. This may be a future optional addition, as it does have a cost.
Differences between C++17 and C++20
Top
There are slight differences between C++17 and C++20
Naming of JSON members
namespace daw::json {
template<>
struct json_data_contract<MyType> {
static constexpr char const member_name[] = "memberName";
using type = json_member_list<json_number<member_name>>;
};
}
C++ 20 Naming of JSON members
When compiled within C++20 compiler, in addition to passing a char const * as in C++17, the member names can be specified as string literals directly. C++20 compiler support is still really early and here be dragons. There are known issues with g++9.x and it's only tested with g++10. Here be dragons
namespace daw::json {
template<>
struct json_data_contract<MyType> {
using type = json_member_list<json_number<"member_name">>;
};
}
Using mapped data types
Top
Once a data type has been mapped with a json_data_contract, the library provides methods to parse JSON to them
MyClass my_class = from_json<MyClass>( json_str );
Alternatively, if the input is trusted, the less checked version can be faster
MyClass my_class = from_json<MyClass, NoCommentSkippingPolicyUnchecked>( json_str );
JSON documents with array root's use the from_json_array function to parse
std::vector<MyClass> my_data = from_json_array<MyClass>( json_str );
Alternatively, if the input is trusted, the less checked version can be faster
std::vector<MyClass> my_data = from_json_array<MyClass, std::vector<MyClass>, NoCommentSkippingPolicyUnchecked>( json_str );
If you want to work from JSON array data you can get an iterator and use the std algorithms to Iterating over array's in JSON data can be done via the json_array_iterator
using iterator_t = json_array_iterator<MyClass>;
auto pos = std::find( iterator_t( json_str ), iterator_t( ), MyClass( ... ) );
Alternatively, if the input is trusted you can called the less checked version
using iterator_t = daw::json::json_array_iterator_trusted<MyClass>;
auto pos = std::find( iterator_t( json_str ), iterator_t( ), MyClass( ... ) );
If you want to serialize to JSON
std::string my_json_data = to_json( MyClass{} );
Or serialize a collection of things
std::vector<MyClass> arry = ...;
std::string my_json_data = to_json_array( arry );
Error Handling
Parsing call
Top
Error checking can be modified on a per parse basis. the from_json/from_json_array calls can be supplied a Parser Policy. The current policies are
NoCommentSkippingPolicyChecked- No comments allowed, checks enabledNoCommentSkippingPolicyUnchecked- No comments allowed, assumes perfect JSONCppCommentSkippingPolicyChecked- C++ style comments/* commment */and// comment until end of line, checks enabledCppCommentSkippingPolicyUnchecked- C++ style comments/* commment */and// comment until end of line, assumes perfect JSONHashCommentSkippingPolicyChecked- Hash style comments# comment until end of line, checks enabledHashCommentSkippingPolicyUnchecked- Hash style comments# comment until end of line, assumes perfect JSON
The unchecked variants can sometimes provide a 5-15% performance increase, but at great risk when the data isn't perfect.
Global
Top
There are two possible ways of handling errors. The default is to throw a daw::json::json_exception on an error in the data. json_exception has a member function std::string_view reason( ) const akin to std::exception's what( ). Second, calling std::terminate( ); on an error in data. If you want to disable exceptions in an environment that has them, you can defined DAW_JSON_DONT_USE_EXCEPTIONS to disable exception throwing by the library.
Deserializing/Parsing
Top
This can be accomplished by writing a function called json_data_contract_for with a single argument that is your type. The library is only concerned with it's return value. For example:
#include <daw/json/daw_json_link.h>
struct TestClass {
int i = 0;
double d = 0.0;
bool b = false;
daw::string_view s{};
std::vector<int> y{};
TestClass( int Int, double Double, bool Bool, daw::string_view S, std::vector<int> Y )
: i( Int )
, d( Double )
, b( Bool )
, s( S )
, y( Y ) {}
};
namespace daw::json {
template<>
struct json_data_contract<TestClass> {
using type = json_member_list<
json_number<"i", int>,
json_number<"d">,
json_bool<"b">,
json_string<"s", daw::string_view>,
json_array<"y", int>
>;
};
}
int main( ) {
std::string test_001_t_json_data = R"({
"i":5,
"d":2.2e4,
"b":false,
"s":"hello world",
"y":[1,2,3,4]
})";
std::string json_array_data = R"([{
"i":5,
"d":2.2e4,
"b":false,
"s":"hello world",
"y":[1,2,3,4]
},{
"i":4,
"d":122e4,
"b":true,
"s":"goodbye world",
"y":[4,3,1,4]
}])";
TestClass test_class = daw::json::from_json<TestClass>( test_001_t_json_data );
std::vector<TestClass> arry_of_test_class = daw::json::from_json_array<TestClass>( test_001_t_json_data );
}
Both aggregate and user constructors are supported. The description provides the values needed to construct your type and the order. The order specified is the order they are placed into the constructor. There are customization points to provide a way of constructing your type too(TODO discuss customization points) A class like:
#include <daw/json/daw_json_link.h>
struct AggClass {
int a{};
double b{};
};
namespace daw::json {
template<>
struct json_data_contract<AggClass> {
using type = json_member_list<
json_number<"a", int>,
json_number<"b">
>;
};
}
Works too. Same but C++17
#include <daw/json/daw_json_link.h>
struct AggClass {
int a{};
double b{};
};
namespace daw::json {
template<>
struct json_data_contract<AggClass> {
static inline constexpr char const a[] = "a";
static inline constexpr char const b[] = "b";
using type = json_member_list<
json_number<a, int>,
json_number<b>
>;
};
}
The class descriptions are recursive with their submembers. Using the previous AggClass one can include it as a member of another class
// See above for AggClass
struct MyClass {
AggClass other;
std::string_view some_name;
};
namespace daw::json {
template<>
struct json_data_contract<MyClass> {
using type = json_member_list<
json_class<"other", AggClass>,
json_string<"id", std::string_view>
>;
};
}
The above maps a class MyClass that has another class that is described AggClass. Also, you can see that the member names of the C++ class do not have to match that of the mapped JSON names and that strings can use std::string_view as the result type. This is an important performance enhancement if you can guarantee the buffer containing the JSON file will exist as long as the class does.
Iterating over JSON arrays. The input iterator daw::json::json_array_iterator<JsonElement> allows one to iterator over the array of JSON elements. It is technically an input iterator but can be stored and reused like a forward iterator. It does not return a reference but a value.
#include <daw/json/daw_json_link.h>
struct AggClass {
int a{};
double b{};
};
namespace daw::json {
template<>
struct json_data_contract<AggClass> {
using type = json_member_list<
json_number<"a", int>,
json_number<"b">
>;
};
}
int main( ) {
std::string json_array_data = R"([
{"a":5,"b":2.2},
{"a":5,"b":3.14},
{"a":5,"b":0.122e44},
{"a":5334,"b":34342.2}
])";
using iterator_t = daw::json::json_array_iterator<AggClass>;
auto pos = std::find_if( iterator_t( json_array_data ), iterator_t( ),
[]( AggData const & element ) { return element.b > 1000.0; } );
if( pos == iterator_t( ) ) {
std::cout << "Not found\n";
} else {
std::cout << "Found\n";
}
}
Member Paths
Parsing can begin at a specific member. An optional member path to from_json_array, from_json_array_unchecked, from_json_array, or from_json_array_unchecked can be specified. The format is a dot separated list of member names and optionally an array index such as member0.member1 or member0[5].member1.
Comments
Comments are supported when the parser policy for them is used. Currently there are two forms of comment policies. C++ style // and /* */. Comments can be placed anywhere there is whitespace allowed
- Hash style
{ # This is a comment
"a" #this is also a comment
: "a's value"
}
- C++ style
{ // This is a comment
"a" /*this is also a comment*/: "a's value"
}
To change the parser policy, you add another argument to from_json and call like from_json<MyType, CppCommentParsePolicy>( json_data )
Serialization
Top
To enable serialization one must create an additional function in your specialization of json_data_contract called to_json_data( Thing const & ); It will provide a mapping from your type to the arguments provided in the class description. To serialize to a JSON string, one calls to_json( my_thing ); where value is a registered type or one of the fundamental types like string, bool, and numbers. The result of to_json_data( Thing const & ) is a tuple who's elements match order in json_data_contract's type alias type. Using the example above lets add that
#include <daw/json/daw_json_link.h>
#include <tuple>
struct AggClass {
int a{};
double b{};
};
namespace daw::json {
template<>
struct json_data_contract<AggClass> {
using type = json_member_list<
json_number<"a", int>,
json_number<"b">
>;
static inline auto to_json_data( AggClass const & value ) {
return std::forward_as_tuple( value.a, value.b );
}
};
}
//...
AggData value = //...;
std::string test_001_t_json_data = to_json( value );
// or
std::vector<AggData> values = //...;
std::string json_array_data = to_json_array( values );
Alternatively there is an optional iostreams interface. In you types json_data_constract add a type alias named opt_into_iostreams the type it aliases doesn't matter, and include daw_json_iostream.h . For example
#include <daw/json/daw_json_link.h>
#include <daw/json/daw_json_iostream.h>
#include <tuple>
struct AggClass {
int a{};
double b{};
};
namespace daw::json {
template<>
struct json_data_contract<AggClass> {
using opt_into_iostreams = void;
using type = json_member_list<
json_number<"a", int>,
json_number<"b">
>;
static inline auto to_json_data( AggClass const & value ) {
return std::forward_as_tuple( value.a, value.b );
}
};
}
//...
AggData value = //...;
std::cout << value << '\n';
// or
std::vector<AggData> values = //...;
std::cout << values << '\n';
A working example can be found at daw_json_iostream_test.cpp
Build configuration points
There are a few defines that affect how JSON Link operates
DAW_JSON_DONT_USE_EXCEPTIONS- Controls if exceptions are allowed. If they are not, anstd::terminate()on errors will occurDAW_ALLOW_SSE42- Allow experimental SSE3 modeDAW_JSON_NO_CONST_EXPR- This can be used to allow classes without move/copy special members to be constructed from JSON data prior to C++ 20. This mode does not work in a constant expression prior to C++20 when this flag is no longer needed.
Requirements
Top
- C++ 17 compiler
- GCC(8/9)/Clang(7/8/9/10) have been tested.
- MSVC 19.21 has been tested.
For building tests
- git
- cmake
Contact
Darrell Wright [email protected]evel.ca
Limitations
- When parsing classes, the first member with a mapped name will be used. If you want to parse a class that can have more than one of any member by name, either parse as a
json_valuesee or as ajson_key_valuethat is mapped to astd::multimapor astd::vectorwith a pair of key type(string) and value type(s). Cookbook Key Values demonstrates these methods. If ajson_key_valueis used and the mapped data type does not support duplicate keys, it will insert for each key. This may result in the last item being the value reflected after serializing. If the duplicate member is the tag type in ajson_tagged_variant, it is undefined what the behaviour for parsing is.
352 Jan 4, 2023
7 Oct 30, 2022
17 Oct 19, 2022
24 Sep 21, 2022
2.1k Jan 1, 2023
234 Jan 8, 2023
33 Dec 13, 2022
282 Jan 3, 2023
305 Dec 13, 2022
275 Dec 28, 2022
23 Jun 15, 2022
28 Dec 30, 2022
16.3k Dec 27, 2022
110 Dec 4, 2022
31 Nov 16, 2022
12.6k Dec 30, 2022
36 Dec 5, 2022