Hi!

During serialization, the size of a type provided is calculated by sizeof(value), here: https://github.com/felixguendling/cista/blob/707ef83b3302104a2aeac5714ff9148ac2e94dc4/include/cista/serialization.h#L194-L196

This is too early IMO to calculate size and alignment when working with custom type.

For example, let's consider these structs:

struct Simple { std::vector<int> data; } simple; template <typename Ctx> void serialize(Ctx& c, Simple const* el, offset_t const pos) { c.write(pos, el->data[0]); }

struct Complex { Simple simple1; Simple simple2; Simple simple3; } complex; template <typename Ctx> void serialize(Ctx& c, Complex const* el, offset_t const pos) { c.write(pos, (int)42); }

When serialized, they will create 0x20 and 0x60 bytes respectiviely (std::vector from MVSC 2017 15.9.0) for cista::byte_buf as a result of sizeof(Simple) and sizeof(Complex) - this cannot be easily overloaded in custom serialization since Context is already given (alocated) and it could only grow.

Hello!

I would like a little bit of help on serializing a class B that inherits A, both standard layout but need custom serialization functions.

For example, let's say we have this

struct A
{
  int _a;
  int _b;

  A(const int a, const int b): _a(a), _b(b) {}

  template <typename Ctx>
  friend inline void serialize(
      Ctx & context, A const* el,
      cista::offset_t const offset) {
    using cista::serialize;
    serialize(context, &el->_a,     offset + offsetof(A, _a));
    serialize(context, &el->_b,     offset + offsetof(A, _b));
  }
}

and this

struct B: A
{
  int _c;
  int _d;

  B(const in a, const int b, const int c, const int d): A(a, b), _c(c), _d(d) {}

  template <typename Ctx>
  friend inline void serialize(
      Ctx & context, B const* el,
      cista::offset_t const offset) {
    // What do I do here ?
  }

}

How do I write the serialization of B ?

Thanks!

Please take a look on this example: https://wandbox.org/permlink/l2iEAmTpiCWFtLb6

Works like a charm, i.e.: CISTA::vector<CISTA::ptr<ElementTest>> paths_ pointers are perfectly rebuild during deserialization. No issue...

Exactly same stuff under latest VS 17.2.6 with all features beyond C++20 enabled gives me two things:

• static_assert need for a custom serializer for ElementTest - whereas wandbox not;
• after ser/des empty fn provided for ElementTest, paths_ still points to memory before serialization;

I am still investigating this...

The following program

#include <iostream>
#include "cista.h"

struct a {
    static int x;
    int z;
    CISTA_PRINTABLE(a)
};

struct b : public a {
    int y;
    CISTA_PRINTABLE(b)
};

int main() {
  b instance;
  std::cout << instance;
  // "{1, 2, 100, hello}"
}

int a::x = 42;

yields the following error for me:

~/Downloads $ clang++-9 -std=c++2a bah.cc -o bah                                                                                                                  16:27:20
In file included from bah.cc:2:
./cista.h:2149:11: error: cannot decompose class type 'b': both it and its base class 'a' have non-static data members
    auto& [p1, p2] = t;
          ^
./cista.h:3734:58: note: in instantiation of function template specialization 'cista::to_tuple<const b>' requested here
    std::apply([&](auto&&... args) { (fn(args), ...); }, to_tuple(t));
                                                         ^
bah.cc:12:5: note: in instantiation of function template specialization 'cista::for_each_field<const b, (lambda at bah.cc:12:5)>' requested here
    CISTA_PRINTABLE(b)
    ^
./cista.h:6381:14: note: expanded from macro 'CISTA_PRINTABLE'
    ::cista::for_each_field(o, [&](auto&& f) {                              \
             ^
1 error generated.

I would like to use cista in a project where there is a lot of inheritance and static members, so this is a problem. I don't know enough about [] binding and templating to know if there is a way to work around this, or if it's a more fundamental limitation.

I have this little fellow here:

using Variant = CISTA::variant
	<
	bool,
	std::int64_t
	>;

Variant test{ true };

auto result = cista::get_if<bool>(test);

...and I am getting (on VS2019/VS2022prev, both C++20): "Error C2563 mismatch in formal parameter list" "Error C2568 '==': unable to resolve function overload" on: https://github.com/felixguendling/cista/blob/23875effbc4f3441304137515991dbcfbc52caed/include/cista/containers/variant.h#L380

I know that I could address bool by its index in Variant - namely 0, but how to make it happen with Type directly? I am trying to make this through cista::get_if implementations... test-case "variant_test.cc" helps me not.

[C2563] https://docs.microsoft.com/en-us/cpp/error-messages/compiler-errors-2/compiler-error-c2563?f1url=%3FappId%3DDev16IDEF1%26l%3DEN-US%26k%3Dk(C2563)%26rd%3Dtrue&view=msvc-160

[C2568] https://docs.microsoft.com/en-us/cpp/error-messages/compiler-errors-2/compiler-error-c2568?f1url=%3FappId%3DDev16IDEF1%26l%3DEN-US%26k%3Dk(C2568)%26rd%3Dtrue&view=msvc-160

1. mkdir test && cd test
2. wget https://github.com/felixguendling/cista/releases/download/v0.10/cista.h
3. echo '#include "cista.h"' > main.cpp
4. clang++ -std=c++17 main.cpp

leads to

In file included from main.cpp:1:
./cista.h:2949:9: error: use of undeclared identifier 'mi_malloc_aligned'
        CISTA_ALIGNED_ALLOC(ALIGNMENT, static_cast<size_t>(size)));
        ^

clang version 14.0.6 Target: x86_64-pc-linux-gnu

Seems like this line is missing from the amalgamated file.

Hi! I know, I know... how this issue summary sounds :) so let's go to an example. I have cista representation of JSON structure like this (cista::mode is NONE default):

namespace CISTA = cista::offset;

using Null = std::monostate;

struct MetaArray;
struct MetaObject;

using MetaValue = CISTA::variant
	<
	Null,
	CISTA::string,
	std::int64_t,
	std::uint64_t,
	double,
	bool,
	CISTA::unique_ptr<MetaArray>,
	CISTA::unique_ptr<MetaObject>
	>;

struct MetaArray : public CISTA::vector<MetaValue> {};

struct MetaObject : public CISTA::hash_map<CISTA::string, MetaValue> {};

Building it works (starting at MetaObject level) with no problem -> after this silly change #96 Serialization of MetaObject or directly MetaValue breaks at: https://github.com/felixguendling/cista/blob/23875effbc4f3441304137515991dbcfbc52caed/include/cista/reflection/to_tuple.h#L472 with (VS2019/VS2022prev output): "Error C3448 the number of identifiers must match the number of array elements or members in a structured binding declaration"

Same for both std::vector<>/void cista::serialize(...) approaches

Hi Felix!

It's been a while since my last interaction with Cista.... so many new features since, my congrats!

To the point... :) I have switched lately my project to the VS "/std:c++latest" and "/std:c++20" (both: 2019 and 2022 preview) and since that I noticed that this serialize(...) usage: https://github.com/felixguendling/cista/blob/23875effbc4f3441304137515991dbcfbc52caed/include/cista/serialization.h#L324-L326 produces hard error: "Error C1001 Internal compiler error".

I know, I know that Cista is C++17 project... but since everything else just works(!) and only this one(!) particular error occurs (even during clean single-header build/tests) I though it is worth mentioning here.

FYI: I did all the preliminary use-case analysis, all of them - even simple tryout of cista::variant serialization leads to "this".

I will investigate it further on my own (right now)... but I'm silently counting on your help here - even it is outside the target (C++17).

Hello!

By looking closer at the code, it appears to me that built-in support for type hashing and serializing std::array could be possible ? This has the benefit to avoid migrating downstream code bases that use std::array. Mirroring the code that already exists for cista::array in my tests seems to work

template <typename T, size_t Size>
hash_t type_hash(std::array<T, Size> const&, hash_t h,
                std::map<hash_t, unsigned>& done) noexcept
{
  h = hash_combine(h, hash("std::array"));
  h = hash_combine(h, Size);
  return type_hash(T{}, h, done);
}

template <typename Ctx, typename T, size_t Size>
void serialize(Ctx& c, std::array<T, Size> const* origin, offset_t const pos) {
  auto const size =
      static_cast<offset_t>(serialized_size<T>() * origin->size());
  auto i = 0u;
  for (auto it = pos; it != pos + size; it += serialized_size<T>()) {
    serialize(c, origin->data() + i++, it);
  }
}

Also, the code of cista::array seems to be standard enough so that std::array could replace it ?

I am probably missing some important piece of information that made it not possible to begin with.

Thanks for your time !

Adel

Not sure what's exactly causing the issue, but once I nest templates deep enough the following happens upon trying to call the cista::offset::to_vec(std::vector<double>) macro:

my.cpp: In function `cista::basic_vector<double, cista::offset_ptr<double, void>, false, unsigned int>::basic_vector()':
/usr/local/cista/include/cista/containers/vector.h:21: undefined reference to `cista::offset_ptr<double, void>::offset_ptr(decltype(nullptr))'
my.cpp: In function `cista::basic_vector<double, cista::offset_ptr<double, void>, false, unsigned int>::basic_vector(cista::basic_vector<double, cista::offset_ptr<double, void>, false, unsigned int> const&)':
/usr/local/cista/include/cista/containers/vector.h:47: undefined reference to `cista::offset_ptr<double, void>::offset_ptr(decltype(nullptr))'
collect2: error: ld returned 1 exit status

Error disappears once I create and call to_vec function instead of the to_vec macro:

inline auto to_vec(const std::vector<double> &vs) {
  message::data::vector<double> v;
  v.reserve(std::size(vs));
  std::copy(std::cbegin(vs), std::cend(vs), std::back_inserter(v));
  return v;
}

If I make the above function a template, i.e. to_vec<double>(std::vector), the link error comes back.

deserialize(CharT *in, CharT *out) only works with const char* in CAST mode. If I try setting the mode to non-CAST, deserialization_context requires the input to be char* and check only works with uint8_t.

Is it possible to make the API work with both const char* and const uint8_t* in all modes?

std::variant<
  unique_ptr<T> /* aus T const& angelegt und kopiert */,
  T const* /* pointer aus bereits alignedtem speicher gespeichert */,
  aligned_buf /* unaligned pointer musste kopiert werden */
>

I'm trying to use cista for a protocol with messages (structures) having between 8 and 88 bytes. I used cista::offset::vector<cista::variant<msg1, msg2, etc.>>.

Having a vector containing 10 messages of 8 bytes, what would be the memory used? I guess the serialized buffer will have approximately the same size.

Could it be possible to add support for more basic string types like std::wstring, std::u16string, std::u32string. Pretty much it, i would add this my self but im confused as to how the string gets turned to bytes. Is it safe to just add more genric_string defines for std::u16string?

In my application I have several big data structures, that I want to dump to the disk, and I can compute size of data in compile time. So Im using cista::array, but seem like there is a bug and Im receiving the invalid address error. Ive attached the example in the hastebin

The error Ive got look like this

* thread #1, name = 'a.out', stop reason = signal SIGSEGV: invalid address (fault address: 0x7ffffdda5d70)
    frame #0: 0x00005555555578a0 a.out`unsigned long cista::type_hash<cista_serializator>() at cista.h:4482:8
   4479
   4480 template <typename T>
   4481 hash_t type_hash() {
-> 4482   auto done = std::map<hash_t, unsigned>{};
   4483   return type_hash(T{}, BASE_HASH, done);
   4484 }
   4485

Process 61576 launched: 'a.out' (x86_64)
(lldb) bt
* thread #1, name = 'a.out', stop reason = signal SIGSEGV: invalid address (fault address: 0x7ffffdda5d70)
  * frame #0: 0x00005555555578a0 a.out`unsigned long cista::type_hash<cista_serializator>() at cista.h:4482:8
    frame #1: 0x0000555555557242 a.out`void cista::serialize<(cista::mode)6, cista::buf<std::vector<unsigned char, std::allocator<unsigned char> > >, cista_serializator>(t=0x00007fffffffe480, value=0x00007ffff582d010) at cista.h:4814:40
    frame #2: 0x0000555555556f31 a.out`std::vector<unsigned char, std::allocator<unsigned char> > cista::serialize<(el=0x00007ffff582d010)6, cista_serializator>(cista_serializator&) at cista.h:4845:18
    frame #3: 0x0000555555556401 a.out`main(argc=1, argv=0x00007fffffffe608) at main.cpp:41:45
    frame #4: 0x00007ffff7ab2152 libc.so.6`__libc_start_main + 242
    frame #5: 0x00005555555562ae a.out`_start + 46

OS: Linux x86_64, compiler: gcc version 10.2.0 (GCC), compile command: g++ -std=c++17 -I. -g main.cpp, Ive tried cista::offset and cista::raw, but no luck