A fast streaming JSON parsing library in C.

Last update: Jan 1, 2023

Related tags

Overview

**********************************************************************
        This is YAJL 2.  For the legacy version of YAJL see
              https://github.com/lloyd/yajl/tree/1.x
**********************************************************************

Welcome to Yet Another JSON Library (YAJL)

## Why does the world need another C library for parsing JSON?  

Good question.  In a review of current C JSON parsing libraries I was 
unable to find one that satisfies my requirements.  Those are, 
0. written in C
1. portable
2. robust -- as close to "crash proof" as possible
3. data representation independent
4. fast
5. generates verbose, useful error messages including context of where
   the error occurs in the input text.
6. can parse JSON data off a stream, incrementally
7. simple to use
8. tiny

Numbers 3, 5, 6, and 7 were particularly hard to find, and were what 
caused me to ultimately create YAJL.  This document is a tour of some
of the more important aspects of YAJL.

## YAJL is Free.

Permissive licensing means you can use it in open source and
commercial products alike without any fees.  My request beyond the
licensing is that if you find bugs drop me a email, or better yet,
fork and fix.

Porting YAJL should be trivial, the implementation is ANSI C.  If you
port to new systems I'd love to hear of it and integrate your patches.

## YAJL is data representation independent.

BYODR!  Many JSON libraries impose a structure based data representation
on you.  This is a benefit in some cases and a drawback in others.
YAJL uses callbacks to remain agnostic of the in-memory representation.
So if you wish to build up an in-memory representation, you may do so
using YAJL, but you must bring the code that defines and populates the
in memory structure.

This also means that YAJL can be used by other (higher level) JSON
libraries if so desired.

## YAJL supports stream parsing

This means you do not need to hold the whole JSON representation in
textual form in memory.  This makes YAJL ideal for filtering projects,
where you're converting YAJL from one form to another (i.e. XML).  The
included JSON pretty printer is an example of such a filter program.

## YAJL is fast

Minimal memory copying is performed.  YAJL, when possible, returns
pointers into the client provided text (i.e. for strings that have no
embedded escape chars, hopefully the common case).  I've put a lot of
effort into profiling and tuning performance, but I have ignored a
couple possible performance improvements to keep the interface clean,
small, and flexible.  My hope is that YAJL will perform comparably to
the fastest JSON parser out there.

YAJL should impose both minimal CPU and memory requirements on your
application.

## YAJL is tiny.

Fat free.  No whip.

enjoy,
Lloyd - July, 2007

Comments

Tests fail on 64 bit machines

Compilation is failing for me on 64 bit Ubuntu (SliceHost):

test case: 'cases/integers.json': --- cases/integers.json.gold 2009-04-24 21:32:45.000000000 +0000 +++ cases/integers.json.test 2009-05-14 22:50:49.000000000 +0000 @@ -10,5 +10,6 @@ integer: -123456789 integer: 2147483647 integer: -2147483647 -parse error: integer overflow +integer: 2147483648 +array close ']' memory leaks: 0 FAILURE

opened by lloyd 7
id field is not properly populated

i first opened this issue for twitter-stream api: http://github.com/intridea/tweetstream/issues#issue/5

and it was suggested that maybe the cause is in yajl. when twitter returns id in json, the value is correct, but id field (in the hash) is -1.

opened by zlu 5
Trouble using/installing 'yajl' on Ubuntu 14.04
Hey @lloyd @pykler

I have been trying to install yajl on my Ubuntu 14.04 system for implementing a Streaming API.

So, I used the following commands:

git clone https://github.com/lloyd/yajl cd yajl/ sudo ./configure sudo make install

Also added it to the shared library path by export LD_LIBRARY_PATH="/usr/local/lib/python3.4/dist-packages/yajl-0.3.5.egg-info"

Still, using it with jsonstreamer gives me the following :

File "/usr/local/lib/python3.4/dist-packages/jsonstreamer/yajl/parse.py", line 29, in load_lib raise OSError('Yajl cannot be found.') OSError: Yajl cannot be found.

Reference to the code-snippet which raises the OSError : github.com/kashifrazzaqui/json-streamer/jsonstreamer/yajl/parse.py#L18-L29

Running gcc -lyajl yields:

jigyasa@spin:~$ gcc -lyajl .... /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crt1.o: In function `_start': (.text+0x20): undefined reference to `main' collect2: error: ld returned 1 exit status

And sudo ldconfig -p | grep yajl results in:

jigyasa@spin:~$ sudo ldconfig -p | grep yajl libyajl.so.2 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libyajl.so.2

Any pointers on this one ?
opened by jigyasa-grover 4

error installing under ubuntu 12.10

hello i have reported and issue when installing a gem script (visage) and the author asked me to report the problem here. here is the original issue : https://github.com/auxesis/visage/issues/124 and here is the installation error :

# gem install visage-app
Building native extensions.  This could take a while...
ERROR:  Error installing visage-app:
    ERROR: Failed to build gem native extension.

        /usr/bin/ruby1.9.1 extconf.rb
creating Makefile

make
compiling yajl_ext.c
yajl_ext.c: In function ‘yajl_encode_part’:
yajl_ext.c:117:21: warning: variable ‘status’ set but not used [-Wunused-but-set-variable]
yajl_ext.c: In function ‘rb_yajl_parser_parse’:
yajl_ext.c:442:17: warning: variable ‘stat’ set but not used [-Wunused-but-set-variable]
compiling yajl_lex.c
compiling yajl_encode.c
compiling yajl.c
compiling yajl_parser.c
compiling yajl_alloc.c
compiling yajl_gen.c
compiling yajl_buf.c
compiling yajl_version.c
linking shared-object yajl/yajl.so

make install
/usr/bin/install -c -m 0755 yajl.so /var/lib/gems/1.9.1/gems/yajl-ruby-1.1.0/lib/yajl
installing default yajl libraries


Gem files will remain installed in /var/lib/gems/1.9.1/gems/yajl-ruby-1.1.0 for inspection.
Results logged to /var/lib/gems/1.9.1/gems/yajl-ruby-1.1.0/ext/yajl/gem_make.out

opened by ssm2017 4

Encode speedup, yajl_string_encode runtime cut from 150 ms to 100 ms

I've done some profiling of an application of mine that uses yajl, and yajl_string_encode was consistently the most cpu intensive part of yajl. These 2 commits cut the total time spent in and below yajl_string_encode from ~150 milliseconds to ~100 milliseconds or so.

opened by bpowers 4
Fixing locale-dependent conversions

(Sorry, I accidentally closed the original issue)

The functions strtod() (used in yajl_parser) and sprintf() (yajl_gen) are locale-dependent and thus cannot be reliably used to convert JSON values. The only reasonable solution I found to this was using setlocale() to insert the "C" locale. However, this can introduce side-effects, so I locally set and then restore it, like this:

char oldLocale; / Example for converting doubles / oldLocale = setlocale(LC_ALL, NULL); / saving old locale / setlocale(LC_ALL, "C"); / ensuring POSIX locale / sprintf(i, "%g", number); / now the conversion works as expected / setlocale(LC_ALL, oldLocale); / restoring locale */

This is, however, probably an inefficient quick fix. The locale should be set before starting the parsing/generation phases and restored at the end, if this does not affect string conversion.

opened by enhadzi 4

Clang warning with -Wenum-conversion

With recent versions of Clang, compiling YAJL 2.0.1 results in this warning:

third_party/yajl/yajl_gen.c:301:5: error: implicit conversion from enumeration type 'yajl_gen_state' to different enumeration type 'yajl_gen_status' [-Werror,-Wenum-conversion]
    DECREMENT_DEPTH;
    ^~~~~~~~~~~~~~~
third_party/yajl/yajl_gen.c:174:46: note: expanded from macro 'DECREMENT_DEPTH'
  if (--(g->depth) >= YAJL_MAX_DEPTH) return yajl_gen_error;
                                      ~~~~~~ ^~~~~~~~~~~~~~
third_party/yajl/yajl_gen.c:327:5: error: implicit conversion from enumeration type 'yajl_gen_state' to different enumeration type 'yajl_gen_status' [-Werror,-Wenum-conversion]
    DECREMENT_DEPTH;
    ^~~~~~~~~~~~~~~
third_party/yajl/yajl_gen.c:174:46: note: expanded from macro 'DECREMENT_DEPTH'
  if (--(g->depth) >= YAJL_MAX_DEPTH) return yajl_gen_error;
                                      ~~~~~~ ^~~~~~~~~~~~~~

This appears to be implicitly converting yajl_gen_state -> yajl_gen_status. I would submit a pull request but the correct fix is not clear to me.

opened by haberman 3

yajl_gen_double does not always emit decimal point

When using yajl to generate JSON, it will not include the decimal point on double-precision floating point numbers if only zeros follow the decimal. This can cause some other lenient JSON parsers to automatically parse such values as an integer rather than a double, which can cause unexpected integer rounding behaviors in that client app if that app was expecting a double to always be parsed.

opened by bovine 3
Allow yajl_gen to be reset

Haiving the ability to reset all internal state and clear the buffer of a given yajl_gen instance would allow me to reuse it even if it were in yajl_gen_in_error_state or yajl_gen_generation_complete.

The benefit if in the yajl_gen_generation_complete state, is that I can continue to create new JSON strings with a single yajl_gen instance, skipping the need to continually malloc/free new instances.

While if in the yajl_gen_in_error_state, I could simply "start over" without needing to free/malloc a new instance.

opened by brianmario 3
Is there any possibility to convert JSON to XML in YAJL ver 2.1.0

Hi Team,

In our implementation we download data from http streaming. As per YAJL documentation I read that, there is a way to convert JSON data to XML format. The below link describes this... /*** *https://github.com/lloyd/yajl *## YAJL supports stream parsing *This means you do not need to hold the whole JSON representation in *textual form in memory. This makes YAJL ideal for filtering projects, *where you're converting YAJL from one form to another (i.e. XML). The *included JSON pretty printer is an example of such a filter program. */

We used YAJL in our implementation to parse JSON data, by using yajl_tree_parse() API. But, this API is not converting our JSON data into XML format.

As per above documentation is there any way to get XML format from JSON data ? If so, then which API will help me to get XML format from JSON data ?

Kindly provide your input on this...

Thanks, Rambabu.

opened by RambabuTirumalasetty 2
Install pkg-config file into libdir

Most pkgconfig files contain a Libs: variable, which is either /usr/lib or /usr/lib64. If a 32bit and a 64bit variant of yajl libraries is installed, the last one wins. As a result compiling for the other bitsize will fail.

Instead of sharedir use libdir as install target.

Signed-off-by: Olaf Hering [email protected]

opened by olafhering 2

`2.1.0`: the same name used as branch name and git tag

Because of that

[tkloczko@devel-g2v SPECS]$ wget https://github.com/lloyd/yajl//archive/2.1.0/yajl-2.1.0.tar.gz
--2022-12-03 08:35:39--  https://github.com/lloyd/yajl//archive/2.1.0/yajl-2.1.0.tar.gz
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 300 Multiple Choices
Length: 105 [text/plain]
Saving to: ‘yajl-2.1.0.tar.gz’

yajl-2.1.0.tar.gz                          100%[========================================================================================>]     105  --.-KB/s    in 0s

2022-12-03 08:35:40 (33.1 MB/s) - ‘yajl-2.1.0.tar.gz’ saved [105/105]

[tkloczko@devel-g2v SPECS]$ cat yajl-2.1.0.tar.gz; echo
the given path has multiple possibilities: #<Git::Ref:0x00007fd90e5de0a0>, #<Git::Ref:0x00007fd90e51a100>

Solution could be rename 2.1.0 branch to 2.1 or 2.1.x.

opened by kloczek 0

legacy-install-failure with pip in Mac M1 chip, unable to pip install yajl

`pip install yajl DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621 Collecting yajl Using cached yajl-0.3.5.tar.gz (52 kB) Preparing metadata (setup.py) ... done Building wheels for collected packages: yajl Building wheel for yajl (setup.py) ... error error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [110 lines of output] fatal: not a git repository (or any of the parent directories): .git running bdist_wheel running build running build_ext building 'yajl' extension creating build creating build/temp.macosx-12-arm64-cpython-39 creating build/temp.macosx-12-arm64-cpython-39/yajl creating build/temp.macosx-12-arm64-cpython-39/yajl/src clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c decoder.c -o build/temp.macosx-12-arm64-cpython-39/decoder.o -Wall decoder.c:111:39: warning: comparison of integers of different signs: 'int' and 'unsigned int' [-Wsign-compare] for (floaty_char = 0; floaty_char < length; floaty_char++) { ~~~~~~~~~~~ ^ ~~~~~~ decoder.c:120:21: warning: comparison of integers of different signs: 'int' and 'unsigned int' [-Wsign-compare] if (floaty_char >= length) { ~~~~~~~~~~~ ^ ~~~~~~ 2 warnings generated. clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c encoder.c -o build/temp.macosx-12-arm64-cpython-39/encoder.o -Wall encoder.c:63:29: warning: '_PyUnicode_get_wstr_length' is deprecated [-Wdeprecated-declarations] Py_ssize_t length = PyUnicode_GET_SIZE(object); ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:261:7: note: expanded from macro 'PyUnicode_GET_SIZE' PyUnicode_WSTR_LENGTH(op) :
^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:451:35: note: expanded from macro 'PyUnicode_WSTR_LENGTH' #define PyUnicode_WSTR_LENGTH(op) _PyUnicode_get_wstr_length((PyObject*)op) ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:445:1: note: '_PyUnicode_get_wstr_length' has been explicitly marked deprecated here Py_DEPRECATED(3.3) ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/pyport.h:508:54: note: expanded from macro 'Py_DEPRECATED' #define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated)) ^ encoder.c:63:29: warning: 'PyUnicode_AsUnicode' is deprecated [-Wdeprecated-declarations] Py_ssize_t length = PyUnicode_GET_SIZE(object); ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:262:14: note: expanded from macro 'PyUnicode_GET_SIZE' ((void)PyUnicode_AsUnicode(_PyObject_CAST(op)),
^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:580:1: note: 'PyUnicode_AsUnicode' has been explicitly marked deprecated here Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE ) PyUnicode_AsUnicode( ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/pyport.h:508:54: note: expanded from macro 'Py_DEPRECATED' #define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated)) ^ encoder.c:63:29: warning: '_PyUnicode_get_wstr_length' is deprecated [-Wdeprecated-declarations] Py_ssize_t length = PyUnicode_GET_SIZE(object); ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:264:8: note: expanded from macro 'PyUnicode_GET_SIZE' PyUnicode_WSTR_LENGTH(op))) ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:451:35: note: expanded from macro 'PyUnicode_WSTR_LENGTH' #define PyUnicode_WSTR_LENGTH(op) _PyUnicode_get_wstr_length((PyObject)op) ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:445:1: note: '_PyUnicode_get_wstr_length' has been explicitly marked deprecated here Py_DEPRECATED(3.3) ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/pyport.h:508:54: note: expanded from macro 'Py_DEPRECATED' #define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated)) ^ encoder.c:64:35: warning: 'PyUnicode_AsUnicode' is deprecated [-Wdeprecated-declarations] Py_UNICODE *raw_unicode = PyUnicode_AS_UNICODE(object); ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:279:7: note: expanded from macro 'PyUnicode_AS_UNICODE' PyUnicode_AsUnicode(_PyObject_CAST(op))) ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:580:1: note: 'PyUnicode_AsUnicode' has been explicitly marked deprecated here Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode( ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/pyport.h:508:54: note: expanded from macro 'Py_DEPRECATED' #define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated)) ^ encoder.c:262:17: warning: comparison of integers of different signs: 'size_t' (aka 'unsigned long') and 'Py_ssize_t' (aka 'long') [-Wsign-compare] if (newsize != Py_SIZE(sauc->str)) { ~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~ 5 warnings generated. clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yajl.c -o build/temp.macosx-12-arm64-cpython-39/yajl.o -Wall yajl.c:347:15: warning: incompatible function pointer types initializing 'PyCFunction' (aka 'struct _object ()(struct _object *, struct _object *)') with an expression of type 'PyCFunctionWithKeywords' (aka 'struct _object ()(struct _object *, struct _object *, struct _object *)') [-Wincompatible-function-pointer-types] {"dumps", (PyCFunctionWithKeywords)(py_dumps), METH_VARARGS | METH_KEYWORDS, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ yajl.c:363:14: warning: incompatible function pointer types initializing 'PyCFunction' (aka 'struct _object ()(struct _object *, struct _object *)') with an expression of type 'PyCFunctionWithKeywords' (aka 'struct _object ()(struct _object *, struct _object *, struct _object *)') [-Wincompatible-function-pointer-types] {"dump", (PyCFunctionWithKeywords)(py_dump), METH_VARARGS | METH_KEYWORDS, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ yajl.c:286:18: warning: unused function 'py_iterload' [-Wunused-function] static PyObject *py_iterload(PYARGS) ^ 3 warnings generated. clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yajl/src/yajl.c -o build/temp.macosx-12-arm64-cpython-39/yajl/src/yajl.o -Wall clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yajl/src/yajl_alloc.c -o build/temp.macosx-12-arm64-cpython-39/yajl/src/yajl_alloc.o -Wall clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yajl/src/yajl_buf.c -o build/temp.macosx-12-arm64-cpython-39/yajl/src/yajl_buf.o -Wall clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yajl/src/yajl_encode.c -o build/temp.macosx-12-arm64-cpython-39/yajl/src/yajl_encode.o -Wall clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yajl/src/yajl_gen.c -o build/temp.macosx-12-arm64-cpython-39/yajl/src/yajl_gen.o -Wall clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yajl/src/yajl_lex.c -o build/temp.macosx-12-arm64-cpython-39/yajl/src/yajl_lex.o -Wall clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yajl/src/yajl_parser.c -o build/temp.macosx-12-arm64-cpython-39/yajl/src/yajl_parser.o -Wall clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yajl_hacks.c -o build/temp.macosx-12-arm64-cpython-39/yajl_hacks.o -Wall yajl_hacks.c:116:37: error: implicitly declaring library function 'strlen' with type 'unsigned long (const char *)' [-Werror,-Wimplicit-function-declaration] ENSURE_VALID_STATE; INSERT_SEP; INSERT_WHITESPACE; ^ yajl_hacks.c:79:26: note: expanded from macro 'INSERT_WHITESPACE' strlen(g->indentString));
^ yajl_hacks.c:116:37: note: include the header <string.h> or explicitly provide a declaration for 'strlen' yajl_hacks.c:79:26: note: expanded from macro 'INSERT_WHITESPACE' strlen(g->indentString));
^ yajl_hacks.c:118:22: warning: passing 'const unsigned char *' to parameter of type 'const char *' converts between pointers to integer types where one is of the unique plain 'char' type and the other is not [-Wpointer-sign] g->print(g->ctx, str, len); ^~~ 1 warning and 1 error generated. error: command '/usr/bin/clang' failed with exit code 1 [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for yajl Running setup.py clean for yajl Failed to build yajl Installing collected packages: yajl DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621 Running setup.py install for yajl ... error error: subprocess-exited-with-error

× Running setup.py install for yajl did not run successfully. │ exit code: 1 ╰─> [112 lines of output] fatal: not a git repository (or any of the parent directories): .git running install /opt/homebrew/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( running build running build_ext building 'yajl' extension creating build creating build/temp.macosx-12-arm64-cpython-39 creating build/temp.macosx-12-arm64-cpython-39/yajl creating build/temp.macosx-12-arm64-cpython-39/yajl/src clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c decoder.c -o build/temp.macosx-12-arm64-cpython-39/decoder.o -Wall decoder.c:111:39: warning: comparison of integers of different signs: 'int' and 'unsigned int' [-Wsign-compare] for (floaty_char = 0; floaty_char < length; floaty_char++) { ~~~~~~~~~~~ ^ ~~~~~~ decoder.c:120:21: warning: comparison of integers of different signs: 'int' and 'unsigned int' [-Wsign-compare] if (floaty_char >= length) { ~~~~~~~~~~~ ^ ~~~~~~ 2 warnings generated. clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c encoder.c -o build/temp.macosx-12-arm64-cpython-39/encoder.o -Wall encoder.c:63:29: warning: '_PyUnicode_get_wstr_length' is deprecated [-Wdeprecated-declarations] Py_ssize_t length = PyUnicode_GET_SIZE(object); ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:261:7: note: expanded from macro 'PyUnicode_GET_SIZE' PyUnicode_WSTR_LENGTH(op) :
^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:451:35: note: expanded from macro 'PyUnicode_WSTR_LENGTH' #define PyUnicode_WSTR_LENGTH(op) _PyUnicode_get_wstr_length((PyObject*)op) ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:445:1: note: '_PyUnicode_get_wstr_length' has been explicitly marked deprecated here Py_DEPRECATED(3.3) ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/pyport.h:508:54: note: expanded from macro 'Py_DEPRECATED' #define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated)) ^ encoder.c:63:29: warning: 'PyUnicode_AsUnicode' is deprecated [-Wdeprecated-declarations] Py_ssize_t length = PyUnicode_GET_SIZE(object); ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:262:14: note: expanded from macro 'PyUnicode_GET_SIZE' ((void)PyUnicode_AsUnicode(_PyObject_CAST(op)),
^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:580:1: note: 'PyUnicode_AsUnicode' has been explicitly marked deprecated here Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE ) PyUnicode_AsUnicode( ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/pyport.h:508:54: note: expanded from macro 'Py_DEPRECATED' #define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated)) ^ encoder.c:63:29: warning: '_PyUnicode_get_wstr_length' is deprecated [-Wdeprecated-declarations] Py_ssize_t length = PyUnicode_GET_SIZE(object); ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:264:8: note: expanded from macro 'PyUnicode_GET_SIZE' PyUnicode_WSTR_LENGTH(op))) ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:451:35: note: expanded from macro 'PyUnicode_WSTR_LENGTH' #define PyUnicode_WSTR_LENGTH(op) _PyUnicode_get_wstr_length((PyObject)op) ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:445:1: note: '_PyUnicode_get_wstr_length' has been explicitly marked deprecated here Py_DEPRECATED(3.3) ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/pyport.h:508:54: note: expanded from macro 'Py_DEPRECATED' #define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated)) ^ encoder.c:64:35: warning: 'PyUnicode_AsUnicode' is deprecated [-Wdeprecated-declarations] Py_UNICODE *raw_unicode = PyUnicode_AS_UNICODE(object); ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:279:7: note: expanded from macro 'PyUnicode_AS_UNICODE' PyUnicode_AsUnicode(_PyObject_CAST(op))) ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/cpython/unicodeobject.h:580:1: note: 'PyUnicode_AsUnicode' has been explicitly marked deprecated here Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode( ^ /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9/pyport.h:508:54: note: expanded from macro 'Py_DEPRECATED' #define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated)) ^ encoder.c:262:17: warning: comparison of integers of different signs: 'size_t' (aka 'unsigned long') and 'Py_ssize_t' (aka 'long') [-Wsign-compare] if (newsize != Py_SIZE(sauc->str)) { ~~~~~~~ ^ ~~~~~~~~~~~~~~~~~~ 5 warnings generated. clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yajl.c -o build/temp.macosx-12-arm64-cpython-39/yajl.o -Wall yajl.c:347:15: warning: incompatible function pointer types initializing 'PyCFunction' (aka 'struct _object ()(struct _object *, struct _object *)') with an expression of type 'PyCFunctionWithKeywords' (aka 'struct _object ()(struct _object *, struct _object *, struct _object *)') [-Wincompatible-function-pointer-types] {"dumps", (PyCFunctionWithKeywords)(py_dumps), METH_VARARGS | METH_KEYWORDS, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ yajl.c:363:14: warning: incompatible function pointer types initializing 'PyCFunction' (aka 'struct _object ()(struct _object *, struct _object *)') with an expression of type 'PyCFunctionWithKeywords' (aka 'struct _object ()(struct _object *, struct _object *, struct _object *)') [-Wincompatible-function-pointer-types] {"dump", (PyCFunctionWithKeywords)(py_dump), METH_VARARGS | METH_KEYWORDS, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ yajl.c:286:18: warning: unused function 'py_iterload' [-Wunused-function] static PyObject *py_iterload(PYARGS) ^ 3 warnings generated. clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yajl/src/yajl.c -o build/temp.macosx-12-arm64-cpython-39/yajl/src/yajl.o -Wall clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yajl/src/yajl_alloc.c -o build/temp.macosx-12-arm64-cpython-39/yajl/src/yajl_alloc.o -Wall clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yajl/src/yajl_buf.c -o build/temp.macosx-12-arm64-cpython-39/yajl/src/yajl_buf.o -Wall clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yajl/src/yajl_encode.c -o build/temp.macosx-12-arm64-cpython-39/yajl/src/yajl_encode.o -Wall clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yajl/src/yajl_gen.c -o build/temp.macosx-12-arm64-cpython-39/yajl/src/yajl_gen.o -Wall clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yajl/src/yajl_lex.c -o build/temp.macosx-12-arm64-cpython-39/yajl/src/yajl_lex.o -Wall clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yajl/src/yajl_parser.c -o build/temp.macosx-12-arm64-cpython-39/yajl/src/yajl_parser.o -Wall clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I. -Iincludes/ -Iyajl/src -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c yajl_hacks.c -o build/temp.macosx-12-arm64-cpython-39/yajl_hacks.o -Wall yajl_hacks.c:116:37: error: implicitly declaring library function 'strlen' with type 'unsigned long (const char *)' [-Werror,-Wimplicit-function-declaration] ENSURE_VALID_STATE; INSERT_SEP; INSERT_WHITESPACE; ^ yajl_hacks.c:79:26: note: expanded from macro 'INSERT_WHITESPACE' strlen(g->indentString));
^ yajl_hacks.c:116:37: note: include the header <string.h> or explicitly provide a declaration for 'strlen' yajl_hacks.c:79:26: note: expanded from macro 'INSERT_WHITESPACE' strlen(g->indentString));
^ yajl_hacks.c:118:22: warning: passing 'const unsigned char *' to parameter of type 'const char *' converts between pointers to integer types where one is of the unique plain 'char' type and the other is not [-Wpointer-sign] g->print(g->ctx, str, len); ^~~ 1 warning and 1 error generated. error: command '/usr/bin/clang' failed with exit code 1 [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure

× Encountered error while trying to install package. ╰─> yajl

note: This is an issue with the package mentioned above, not pip.`

opened by pajain-mdsol 0
Alternative fix for CVE-2022-24795

This is a "hybrid" of https://github.com/lloyd/yajl/pull/240, https://github.com/brianmario/yajl-ruby/pull/211 and https://github.com/robohack/yajl/commit/166b384aec1cf304859d69f03e42c3ab85c34858

Using abort() to avoid heap corruption/infinite loop while not adding new requirements for clients of this library and/or complex error handling mechanisms.

opened by skazi0 0
Fix CVE-2022-24795

There was an integer overflow in yajl_buf_ensure_available() leading to allocating less memory than requested. Then data were written past the allocated heap buffer in yajl_buf_append(), the only caller of yajl_buf_ensure_available(). Another result of the overflow was an infinite loop without a return from yajl_buf_ensure_available().

yajl-ruby project, which bundles yajl, fixed it https://github.com/brianmario/yajl-ruby/pull/211 by checking for the integer overflow, fortifying buffer allocations, and report the failures to a caller. But then the caller yajl_buf_append() skips a memory write if yajl_buf_ensure_available() failed leading to a data corruption.

A yajl fork mainter recommended calling memory allocation callbacks with the large memory request and let them to handle it. But that has the problem that it's not possible pass the overely large size to the callbacks.

This patch catches the integer overflow and terminates the process with abort().

https://github.com/lloyd/yajl/issues/239 https://github.com/brianmario/yajl-ruby/security/advisories/GHSA-jj47-x69x-mxrm

opened by ppisar 6
CVE-2022-24795

Hello, I am a member of the GitHub Security team and am seeking a maintainer contact for this project with regards to https://github.com/brianmario/yajl-ruby/security/advisories/GHSA-jj47-x69x-mxrm

We were unable to establish contact with a maintainer for this project during the coordinated disclosure process for CVE-2022-24795. If this project is still actively maintained, we'd like to offer our assistance in getting this issue resolved in your library. If it is no longer actively maintained, we recommend archiving this repository.

opened by anticomputer 3