An HTML5 parsing library in pure C99

Related tags

Networking and Internet gumbo-parser
Overview

Gumbo - A pure-C HTML5 parser.

Build Status Build status

Gumbo is an implementation of the HTML5 parsing algorithm implemented as a pure C99 library with no outside dependencies. It's designed to serve as a building block for other tools and libraries such as linters, validators, templating languages, and refactoring and analysis tools.

Goals & features:

  • Fully conformant with the HTML5 spec.
  • Robust and resilient to bad input.
  • Simple API that can be easily wrapped by other languages.
  • Support for source locations and pointers back to the original text.
  • Support for fragment parsing.
  • Relatively lightweight, with no outside dependencies.
  • Passes all html5lib tests, including the template tag.
  • Tested on over 2.5 billion pages from Google's index.

Non-goals:

  • Execution speed. Gumbo gains some of this by virtue of being written in C, but it is not an important consideration for the intended use-case, and was not a major design factor.
  • Support for encodings other than UTF-8. For the most part, client code can convert the input stream to UTF-8 text using another library before processing.
  • Mutability. Gumbo is intentionally designed to turn an HTML document into a parse tree, and free that parse tree all at once. It's not designed to persistently store nodes or subtrees outside of the parse tree, or to perform arbitrary DOM mutations within your program. If you need this functionality, we recommend translating the Gumbo parse tree into a mutable DOM representation more suited for the particular needs of your program before operating on it.
  • C89 support. Most major compilers support C99 by now; the major exception (Microsoft Visual Studio) should be able to compile this in C++ mode with relatively few changes. (Bug reports welcome.)
  • Security. Gumbo was initially designed for a product that worked with trusted input files only. We're working to harden this and make sure that it behaves as expected even on malicious input, but for now, Gumbo should only be run on trusted input or within a sandbox. Gumbo underwent a number of security fixes and passed Google's security review as of version 0.9.1.

Wishlist (aka "We couldn't get these into the original release, but are hoping to add them soon"):

  • Full-featured error reporting.
  • Additional performance improvements.
  • DOM wrapper library/libraries (possibly within other language bindings)
  • Query libraries, to extract information from parse trees using CSS or XPATH.

Installation

To build and install the library, issue the standard UNIX incantation from the root of the distribution:

$ ./autogen.sh
$ ./configure
$ make
$ sudo make install

Gumbo comes with full pkg-config support, so you can use the pkg-config to print the flags needed to link your program against it:

$ pkg-config --cflags gumbo         # print compiler flags
$ pkg-config --libs gumbo           # print linker flags
$ pkg-config --cflags --libs gumbo  # print both

For example:

$ gcc my_program.c `pkg-config --cflags --libs gumbo`

See the pkg-config man page for more info.

There are a number of sample programs in the examples/ directory. They're built automatically by 'make', but can also be made individually with make <programname> (eg. make clean_text).

To run the unit tests, you'll need to have googletest downloaded and unzipped. The googletest maintainers recommend against using make install; instead, symlink the root googletest directory to 'gtest' inside gumbo's root directory, and then make check:

$ unzip gtest-1.6.0.zip
$ cd gumbo-*
$ ln -s ../gtest-1.6.0 gtest
$ make check

Gumbo's make check has code to automatically configure & build gtest and then link in the library.

Debian and Fedora users can install libgtest with:

$ apt-get install libgtest-dev  # Debian/Ubuntu
$ yum install gtest-devel       # CentOS/Fedora

Note for Ubuntu users: libgtest-dev package only install source files. You have to make libraries yourself using cmake:

$ sudo apt-get install cmake
$ cd /usr/src/gtest
$ sudo cmake CMakeLists.txt
$ sudo make
$ sudo cp *.a /usr/lib

The configure script will detect the presence of the library and use that instead.

Note that you need to have super user privileges to execute these commands. On most distros, you can prefix the commands above with sudo to execute them as the super user.

Debian installs usually don't have sudo installed (Ubuntu however does.) Switch users first with su -, then run apt-get.

Basic Usage

Within your program, you need to include "gumbo.h" and then issue a call to gumbo_parse:

#include "gumbo.h"

int main() {
  GumboOutput* output = gumbo_parse("<h1>Hello, World!</h1>");
  // Do stuff with output->root
  gumbo_destroy_output(&kGumboDefaultOptions, output);
}

See the API documentation and sample programs for more details.

A note on API/ABI compatibility

We'll make a best effort to preserve API compatibility between releases. The initial release is a 0.9 (beta) release to solicit comments from early adopters, but if no major problems are found with the API, a 1.0 release will follow shortly, and the API of that should be considered stable. If changes are necessary, we follow semantic versioning.

We make no such guarantees about the ABI, and it's very likely that subsequent versions may require a recompile of client code. For this reason, we recommend NOT using Gumbo data structures throughout a program, and instead limiting them to a translation layer that picks out whatever data is needed from the parse tree and then converts that to persistent data structures more appropriate for the application. The API is structured to encourage this use, with a single delete function for the whole parse tree, and is not designed with mutation in mind.

Python usage

To install the python bindings, make sure that the C library is installed first, and then sudo python setup.py install from the root of the distro. This installs a 'gumbo' module; pydoc gumbo should tell you about it.

Recommended best-practice for Python usage is to use one of the adapters to an existing API (personally, I prefer BeautifulSoup) and write your program in terms of those. The raw CTypes bindings should be considered building blocks for higher-level libraries and rarely referenced directly.

External Bindings and other wrappers

The following language bindings or other tools/wrappers are maintained by various contributors in other repositories:

Comments
  • any interest in modifying code for binary search of tag names and for tag_in ?

    any interest in modifying code for binary search of tag names and for tag_in ?

    Hi, I have modified gumbo-parser to expand the TagNames to include the full set of presentation elements in MathML and the full SVG tag list. (according to the latest spec). Given the size of the new tag list, a binary search is needed to quickly assign GumboTag enums.

    But since we now can work with sorted lists of integers and complete sets of SVG and MathML tags we can easily use binary search on sorted integer lists to detect if a tag is a valid MathML tag (ie. part of the presentation subset) or a valid SVG tag.

    With this capability in place we can convert all of the calls to tag_in in parser.c and its related cousins to properly handle any C pre-processor determined list of tags (no more varargs needed) and to use binary search to determine if a tag is in that list.

    With this and the capability to quickly determine if something is a valid SVG or MathML element, you could easily modify the parser to properly prevent the confusion that comes when mislabling html tags as mathml or svg tags. which caused a number of issues.

    I have completed part one of this where I modify src/tag.c, add src/tag.h, and update the gumbo.h tag enum or expand its contents to properly match the list in tag.c.

    Before I go to the trouble of converting all of the varargs calls to tag_in and its relatives in parser.c to use pre-determined tag lists (pre-sorted), I was hoping someone would look at the part 1 patch and let me know if they are at all interested.

    opened by kevinhendricks 24
  • Altversion2 - an attempt to put vmg's approach on today's master

    Altversion2 - an attempt to put vmg's approach on today's master

    Hi,

    I tried to implement vmg's approach over currrent master without any of my changes. I also built things incrementally so you can pick and choose which commits you want for your tree if any.

    This should allow people to check out the speedups, size, etc using current master as the basis.

    I did not add any additional tags or reorder any to maintain your api per your wishes

    All credit goes to vmg here.

    opened by kevinhendricks 22
  • Match elements on qualified names, not just local names

    Match elements on qualified names, not just local names

    There are many places where the HTML spec says to look for an HTML element (i.e., an element in the HTML namespace) with a given tag, but where we were only looking at tag names and ignoring the namespace name.

    Now we always use qualified names where required. This is implemented using a new GumboQualName type, which is just a bit field that combines a GumboNamespaceEnum with a GumboTag. A series of macros make it easy to construct and inspect these bit fields. It's currently represented by a uintptr_t. This is larger than necessary; only 9 bits are required. At first I attempted to use a short but the compiler didn't like that being used with varargs, so then I tried an unsigned int but the compiler didn't like casting that to a pointer, so here we are. I also tried typedef enum { QNSIZE = SHORT_MAX } GumboQualName to induce more compiler warnings when mistakenly passing a tag to a function that expects a qualified name. This worked nicely but generated warnings about missing cases in switch statements. I'm not sure what the best option is, though I am quite tempted by the extra warnings the enum provides.

    Fixes #278

    /cc @gsnedders

    opened by aroben 22
  • Error in conversion from void* to GumboNode*

    Error in conversion from void* to GumboNode*

    I'm receiving the following error while trying to find the title from within a page being passed in as a char* by curl.

    Data_Handles.h:169:43: error:
invalid conversion from ‘void*’ to ‘GumboNode* {aka GumboInternalNode*}’ [-fpermissive]
GumboNode* child = root_children->data[i];

    I receive the same error in another location, but seeing how the code is similar, I figure that it'd be better to post once.

    The following is (with a small change) the same code as within the get_titles.c file in the examples directory of the repository.

    155:    char* toParse = get_src(url);
158:    GumboOutput* output = gumbo_parse(toParse);
161:    GumboNode* root = (output->root);
163:    assert(root->type == GUMBO_NODE_ELEMENT);
164:    assert(root->v.element.children.length >= 2);

166:    const GumboVector* root_children = &root->v.element.children;
167:    GumboNode* head = NULL;
168:    for (int i = 0; i < root_children->length; ++i) {
169:        GumboNode* child = root_children->data[i];
170:        if (child->type == GUMBO_NODE_ELEMENT &&
171:            child->v.element.tag == GUMBO_TAG_HEAD) {
172:            head = child;
173:            break;
172:        }
173:    }
174:    assert(head != NULL);

    I apologize for not being able to narrow the error down any further. I have tested the output of the curl and it does pass in valid source. I have used the clean_text.cc example file as a test for that. Therefore, I assume it's an issue with the way I'm using the structure of the GumboOutput and GumboNode.

    Thank you in advance for any help!

    opened by cleverNamesAreHard 12
  • Support parsing the <template> tag

    Support parsing the