Tilck is an educational monolithic x86 kernel designed to be Linux-compatible at binary leve

Overview

Tilck (Tiny Linux-Compatible Kernel)

Build Status codecov License

Tilck

Contents

Overview

What is Tilck?

Tilck is an educational monolithic x86 kernel designed to be Linux-compatible at binary level. Project's small-scale and simple design makes it the perfect playground for playing in kernel mode while retaining the ability to compare how the very same usermode bits run on the Linux kernel as well. That's a unique feature in the realm of educational kernels. Thanks to that, to build a program for Tilck it's enough to use a i686-musl toolchain from bootlin.com. Tilck has no need to have its own set of custom written applications, like most educational kernels do. It just runs mainstream Linux programs like the BusyBox suite. While the Linux-compatibility and the monolithic design might seem a limitation from the OS research point of view, on the other side, such design bring the whole project much closer to real-world applications in the future, compared to the case where some serious (or huge) effort is required to port pre-existing software on it. Also, nothing stops Tilck from implementing custom non-Linux syscalls that aware apps might take advantage of.

Future plans

In the long term, depending on how successful the project will be, Tilck might become suitable for embedded systems on which an extra-simple and fully deterministic kernel is required or, at least, it is considered the optimal solution. With a fair amount of luck, Tilck might be able to fill the gap between Embedded Linux and typical real-time operating systems like FreeRTOS or Zephyr. In any case, at some point it will be ported to the ARM family and it might be adapted to run on MMU-less CPUs as well. Tilck would be a perfect fit for that because consuming a tiny amount of RAM has always been a key point in Tilck's design. Indeed, the kernel can comfortably boot and run on a i686 QEMU machine with just 4 MB of memory today. Of course, that's pointless on x86, but on an ARM Cortex-R it won't be anymore the case.

What Tilck is NOT ?

An attempt to re-write and/or replace the Linux kernel. Tilck is a completely different kernel that has a partial compatibility with Linux just in order to take advantage of its programs and toolchains. Also, that helps a lot to validate its correctness: if a program works correctly on Linux, it must work the same way on Tilck as well (except for the not-implemented features). But, having a fair amount of Linux programs working on it, is just a starting point: after that, Tilck will evolve in a different way and it will have its own unique set of features as well. Tilck is fundamentally different from Linux in its design and its trade-offs as it does not aim to target multi-user server or desktop machines. Currently, it targets the educational world, while in the future it might target embedded systems or something else.

Features

Tilck is a preemptable monolithic (but with compile-time modules) *NIX kernel, implementing about ~100 Linux syscalls (both via int 0x80 and sysenter) on x86. At its core, the kernel is not x86-centric even if it runs only on x86 at the moment. Everything arch-specific is isolated. Because of that, most of kernel's code can be already compiled for any architecture and can be used in kernel's unit tests.

Hardware support

While the kernel uses a fair amount of legacy hardware like the 8259 PICs for IRQs, the legacy 8254 PIT for the system timer, the legacy 16550 UART for serial communication, the 8042 kb controller, the 8237 ISA DMA, and the Sound Blaster 16 sound card (QEMU only), it has also support for some recent hardware features like SSE, AVX and AVX2 fpu instructions, PAT, i686 sysenter, enumeration of PCI Express devices (via ECAM) and, above all, ACPI support via ACPICA. ACPI is currently used to receive power-button events, to reboot or power-off the machine, and to read the current parameters of machine's batteries (when implemented via ACPI control methods).

Comments about physical hardware

The operating system has been regularly tested on physical hardware from its inception by booting it with an USB stick (see the notes below). Test machines include actual i686 machines, older x86_64 machines with BIOS-only firmware, newer x86_64 machines with UEFI+CSM and finally super-recent pure-UEFI machines. For a long time, Tilck's development strictly complied with the following rule: if you cannot test it on real hardware, do not implement it in Tilck. Only recently, that rule has been relaxed a little in order to play with SB16. It is possible that, in the future, there might be a few other drivers that would be tested only on virtual machines: their development is justified by the educational value it will bring to the operating system and the infrastructure built for them will be reused for other drivers of the same kind. But that will never become a common practice. Tilck is designed to work on real hardware, where any kind of weird things happen. Being reliable there is critical for Tilck's success.

File systems

Tilck has a simple but full-featured (both soft and hard links, file holes, memory mapping, etc.) ramfs implementation, a minimalistic devfs implementation, read-only support for FAT16 and FAT32 (used for initrd) allowing memory-mapping of files, and a sysfs implementation used to provide a full view of ACPI's namespace, the list of all PCI(e) devices and Tilck's compile-time configuration. Clearly, in order to work with multiple file systems at once, Tilck has a simple VFS implementation as well.

Processes and signals

While Tilck uses internally the concept of thread, multi-threading is not currently exposed to userspace (kernel threads exist, of course). Both fork() and vfork() are properly implemented and copy-on-write is used for fork-ed processes. The waitpid() syscall is fully implemented (which implies process groups etc.). The support for POSIX signals is partial: custom signal handlers are supported using the rt_sigaction() interface, but most of the SA_* flags are not supported and handlers cannot interrupt each other, yet. rt_sigprocmask(), sys_rt_sigpending(), sys_rt_sigsuspend() work as expected, as well as special signals like SIGSTOP, SIGCONT and SIGCHLD. For more details, see the syscalls document.

One interesting feature in this area deserves a special mention: despite the lack of multi-threading in userspace, Tilck has full support for TLS (thread-local storage) via set_thread_area(), because libmusl requires it, even for classic single-threaded processes.

I/O

In addition to the classic read() and write() syscalls, Tilck supports vectored I/O via readv() and writev() as well. In addition to that, non blocking I/O, select() and poll() are supported too. Fortunately, no program so far needed epoll :-)

Console

Tilck has a console supporting more than 90% of Linux's console's features. It works in the same way (using layers of abstraction) both in text mode and in framebuffer mode. The effort to implement such a powerful console was driven by the goal to make Vim work smoothly on Tilck, with syntax highlighting etc. While it's true that such a thing has a little to do with "proper" kernel development, being able to run a "beast" like Vim on a simple kernel like Tilck, is a great achievement by itself because it shows that Tilck can run correctly programs having a fair amount of complexity.

Userspace applications

Tilck can run a fair amount of console applications like the BusyBox suite, Vim, TinyCC, Micropython, Lua, and framebuffer applications like a port of DOOM for the Linux console called fbDOOM. Check project's wiki page for more info about that.

Screenshots

Tilck screenshots

For full-size screenshots and much more stuff, check Tilck's wiki page.

Booting Tilck

Tilck's bootloader

Tilck comes with an interactive bootloader working both on legacy BIOS and on UEFI systems as well. The bootloader allows the user to choose the desired video mode, the kernel file itself and to edit kernel's cmdline.

Tilck's bootloader

3rd-party bootloaders

Tilck can be loaded by any bootloader supporting multiboot 1.0. For example, qemu's built-in bootloader works perfectly with Tilck:

qemu-system-i386 -kernel ./build/tilck -initrd ./build/fatpart

Actually that way of booting the kernel is used in the system tests. A shortcut for it is:

./build/run_multiboot_qemu

Grub support

Tilck can be easily booted with GRUB. Just edit your /etc/grub.d/40_custom file (or create another one) by adding an entry like:

/tilck module --nounzip /fatpart boot } ">
menuentry "Tilck" {
    multiboot /tilck
    module --nounzip /fatpart
    boot
}

After that, just run update-grub as root and reboot your machine.

Documentation and HOWTOs

Project's main documentation can be found in the docs/ directory. However, Tilck's wiki can be used to navigate through those documention files with the addition of much extra content like screenshots. Here below, instead, there's a quick starter guide, focusing on the most common scenarios.

Building Tilck

The project supports a fair amount of build configurations and customizations but building using its default configuration can be described in just a few steps. The only true requirement for building Tilck is having a Linux x86_64 host system or Microsoft's WSL. Steps:

  • Enter project's root directory.
  • Build the toolchain (just the first time) with: ./scripts/build_toolchain
  • Compile the kernel and prepare the bootable image with: make -j

At this point, there will be an image file named tilck.img in the build directory. The easiest way for actually trying Tilck at that point is to run: ./build/run_qemu.

Running it on physical hardware

The tilck.img image is, of course, bootable on physical machines as well, both on UEFI systems and on legacy ones. Just flush the image file with dd to a usb stick and reboot your machine.

Other configurations

To learn much more about how to build and configure Tilck, check the building guide in the docs/ directory.

Testing Tilck

Tilck has unit tests, kernel self-tests, system tests (using the syscall interface), and automated interactive system tests (simulating real user input through QEMU's monitor) all in the same repository, completely integrated with its build system. In addition to that, there's full code coverage support and useful scripts for generating HTML reports (see the coverage guide). Finally, Tilck is fully integrated with the Azure Pipelines CI, which validates each pushed branch with builds and test runs in a variety of configurations. Kernel's coverage data is also uploaded to CodeCov. Below, there are some basic instructions to run most of Tilck's tests. For the whole story, please read the testing document.

Running Tilck's tests

Running Tilck's tests is extremely simple: it just requires to have python 3 installed on the machine. For the self-tests and the classic system tests, run:

/st/run_all_tests -c

To run the unit tests instead:

  • Install the googletest library (once) with: ./scripts/build_toolchain -s build_gtest

  • Build the unit tests with: make -j gtests

  • Run them with: /gtests

To learn much more about Tilck's tests in general and to understand how to run its interactive system tests as well, read the testing document.

Debugging Tilck

With QEMU's integrated GDB server, it's possible to debug the Tilck kernel with GDB almost as if it were a regular process. It just gets tricky when context switches happen, but GDB cannot help with that. To debug it with GDB, follow the steps:

  • (Optional) Prepare a debug build of Tilck, for a better debugging experience.

  • Run Tilck's VM with: ./build/run_nokvm_qemu but, remain at the bootloader stage.

  • In a different terminal, run: gdb ./build/tilck_unstripped.

  • In GDB, run: target remote :1234 to connect to QEMU's gdb server.

  • Set one or more breakpoints using commands like: break kmain.

  • Type c to allow execution to continue and boot the OS by pressing ENTER in the bootloader.

In order to make the debugging experience better, Tilck comes with a set of GDB scripts (see other/gdb_scripts). With them, it's super-easy to list all the tasks on the system, the handles currently opened by any given process and more. In order to learn how to take advantage of those GDB scripts and anything else related to debugging the Tilck project, check the debugging document.

Tilck's debug panel

Tilck's debug panel

Debugging Tilck with GDB while it's running inside a VM is very convenient, but in other cases (e.g. Tilck on real hardware) we don't have GDB support. In addition to that, even when the kernel is running inside a VM, there are some features that are just much more convient to expose directly from the kernel itself rather than through GDB scripts. One way to expose kernel info to userspace is to use sysfs, but that's not necessarily the most convenient way for everything (still, Tilck does have sysfs implementation), especially when interaction with the kernel itself is needed for debugging purposes. To help in those cases, a debug panel has been introduced inside Tilck itself. It started as something like Linux's Magic SysRq which evolved in a sort of TUI application with debug info plus tracing capabilities for user processes. In the future, it will support some proper debugging features as well. To learn more about it, check the the debugging document.

A comment about user experience

Tilck particularly distinguishes itself from many open source projects in one way: it really cares about the user experience (where "user" means "developer"). It's not the typical super-cool low-level project that's insanely complex to build and configure; it's not a project requiring 200 things to be installed on the host machine. Building such projects may require hours or even days of effort (think about special configurations e.g. building with a cross-compiler). Tilck instead, has been designed to be trivial to build and test even by inexperienced people with basic knowledge of Linux. It has a sophisticated script for building its own toolchain that works on all the major Linux distributions and a powerful CMake-based build system. The build of Tilck produces an image ready to be tested with QEMU or written on a USB stick. (To some degree, it's like what the buildroot project does for Linux, but it's much simpler.) Finally, the project includes also scripts for running Tilck on QEMU with various configurations (BIOS boot, UEFI boot, direct (multi-)boot with QEMU's -kernel option, etc.).

Motivation

The reason for having the above mentioned features is to offer its users and potential contributors a really nice experience, avoiding any kind of frustration. Hopefully, even the most experienced engineers will enjoy a zero effort experience. But it's not all about reducing the frustration. It's also about not scaring students and junior developers who might be just curious to see what this project is all about and maybe eager to write a simple program for it and/or add a couple of printk()'s here and there in their fork. Hopefully, some of those people just playing with Tilck might actually want to contribute to its development.

In conclusion, even if some parts of the project itself are be pretty complex, at least building and running its tests must be something anyone can do.

FAQ (by vvaltchev)

Why Tilck does not have the feature/abstraction XYZ like other kernels do?

Tilck is not meant to be a full-featured production kernel. Please, refer to Linux (or other kernels) for that. The idea for the moment was just to implement an educational kernel able to run a class of Linux console applications on real hardware. At some point in the future and with a lot of luck, Tilck might actually have a chance to be used in production embedded environments (as mentioned above) but it will still be by design limited in terms of features compared to Embedded Linux. Tilck will always try to be different from Linux, simply because Linux is already great per se and it does not make any sense trying to reimplement it. Instead, it's worth trying to create something new while playing the "linux-compatibility card". What I expect is Tilck to start "stealing" ideas from hard real-time kernels, once it gets ported to ARM and MMU-less CPUs. But today, the project is not there yet.

Why Tilck runs only on x86 (ia-32)?

Actually Tilck runs only on x86 for the moment. The kernel was born as a purely educational project and the x86 architecture was already very friendly to me at the time. Moving from x86 usermode assembly to "kernel" mode (real-mode and the transitions back and forth to protected mode for the bootloader) required quite an effort, but it still was, in my opinion, easier than "jumping" directly into a completely unknown (for me) architecture, like ARM. I've also considered writing from the beginning a x86_64 kernel running completely in long mode but I decided to stick initially with the i686 architecture for the following reasons:

  • The long mode is, roughly, another "layer" added on the top of 32-bit protected mode: in order to have a full understanding of its complexity, I thought it was better to start first with its legacy.

  • The long mode does not have a full support for segmentation, while I wanted to get confident with this technology as well.

  • The long mode has a 4-level paging system, which is more complex to use that the classic 2-level paging supported by ia-32 (it was better to start with something simpler).

  • I never considered the idea of writing a kernel for desktop or server-class machines where supporting a huge amount of memory is a must. We already have Linux for that.

  • It seemed to me at the time, that more online "starters" documentation existed (like the articles on https://wiki.osdev.org/) for i686 compared to any other architecture.

Said that, likely I'll make Tilck to support also x86_64 and being able to run in long mode at some point but, given the long-term plans for it as a tiny kernel for embedded systems, making it to run on ARM machines has priority over supporting x86_64. Anyway, at this stage, making the kernel (mostly arch-independent code) powerful enough has absolute priority over the support for any specific architecture. x86 was just a pragmatic choice for its first archicture.

Why having support for FAT32?

The first reason for supporting FAT32 (and FAT16) was that a FAT partition is required for booting with UEFI. Therefore, it was convienent at the time to store there also all the rest of the "initrd" files (init, busybox etc.). After the boot, ramfs is mounted at root, while the FAT32 boot partition is mounted at /initrd. Part of the FAT32 code in the kernel is reused by the legacy bootloader in order to read the kernel file from the boot partition. Today, the "initrd" files are NOT stored anymore in the boot partition; there are two separate FAT partitions instead: bootpart, a small partition containing just the kernel file and the EFI bootloaders, and fatpart, a slightly bigger partition (depending on the configration) containing the initial ramdisk files (e.g. busybox) instead. After the boot, fatpart remains mounted at /initrd, while none of the contents of bootpart are kept.

Why keeping the initrd mounted?

To minimize the peak in memory usage during boot. Consider the idea of having a tgz archive and having to extract all of its files in the root directory: doing that will require, even for a short period of time, keeping both the archive and all of its contents in memory. This is against Tilck's effort to reduce its memory footprint as much as possible; part of project's goals is being able to run on very limited systems.

Why using 3 spaces as indentation?

Long story. It all started after using that coding style for years at VMware. Initially, it looked pretty weird to me, but at some point I felt in love with the way code looked. I got convinced that 2 spaces are just not enough, while 4 spaces are somehow "too much". Therefore, when I started the project in 2016, I decided to stick with the indentation size I liked most, even if I knew that using 4 spaces would have been better for most people. Today, I'm not sure if that was the right decision, but I still like the way Tilck's code looks.

Why many commit messages are so short?

It is well-known that all popular open source projects care about having good commit messages and nice git history. It is an investment that at some point pays off. A few years ago, I even wrote a blog post about that. The problem is that such investment actually starts paying off only when multiple people contribute to a project or the project is really mature enough. It took a long time for me to start considering Tilck as kind of mature. Actually, that's not even a binary value, it's a slow process instead: with time (and commits!), the project matured even if it still has a long way to go. Therefore, by looking at the commits from the initial one to today, it's possible to observe how they improved, both from the message point of view and from the content point of view as well. In particular, during the last ~1,000 commits I started not only re-ordering commits but to split, edit, and squash them all the time. Git's add -p became a friend too. That's because today Tilck is pretty stable and it starts to be a medium-sized project with its 97,000+ physical lines of code and a git history of 5,000+ commits. It deserves much more effort on each commit, compared to the past.

At the beginning, Tilck was just a small experimental and unstable project on which I worked alone in my free time. It had even a different name, ExperimentOS. Its source was also subject to drastic changes very often and I didn't have a clear roadmap for it either. It was an overkill to spend so much effort on each commit message as if I were preparing it for the Linux kernel. Tilck is still obviously not Linux so, don't expect to see 30+ lines of description for EVERY commit message from now on, BUT, the quality is raising through a gradual process and that's pretty natural. As other people start to contribute to the project, we all will have to raise further the bar in order to the collaboration to succeed and being able to understand each other's code faster. Today, the project still doesn't have regular contributors other than myself and that's why many commits still have short commit messages.

Issues
  • Kernel panic after the boot in Lenovo T440

    Kernel panic after the boot in Lenovo T440

    Describe the bug Kernel panic

    Host configuration Lenovo T440 BIOS: UEFI/Legacy Boot [UEFI Only] -CSM Support [No]

    Toolchain configuration Standard

    Tilck build configuration Commit number in attachment below

    Type of issue (select one)

    • Panic

    Reproduction details Boot, no actions, panic.

    Screenshots 20220608_175828

    bug 
    opened by duselguy 38
  • ACPI related problems

    ACPI related problems

    I included few different problems in one issue, because (may be) they relate to the same problem.

    Describe the bug ACPI related message appears only once.

    Host configuration Amilo UI 3520

    Toolchain configuration Standard

    Tilck build configuration See attachment below

    Type of issue (select one)

    • Incorrect behavior

    Reproduction details

    1. Boot with power cable is ON 1.1. Switch power cable OFF after the boot completed 1.2. The message:

    ACPI: got GPE #29

    1.3. Switch power cable ON/OFF/ON/etc. - no above message (not trapped or logged) 2. Boot with power cable is OFF 2.1. Switch power cable ON after the boot completed 2.2. The message:

    ACPI: got GPE #29

    2.3. Switch power cable OFF/ON/OFF/etc. - no above message (not trapped or logged)

    Expected behavior The message after each switch of the power cable.


    Describe the bug poweroff command is not completed

    Host configuration Amilo UI 3520

    Toolchain configuration Standard

    Tilck build configuration See attachment below

    Type of issue (select one)

    • Incorrect behavior

    Reproduction details

    1. Boot. Than issue: poweroff 1.1. Executing, black screen, but power and WiFi indicators are light.
    2. Boot. Than issue poweroff -f 20220608_201646

    Expected behavior Black screen and no light indicators.

    Describe the bug ACPI related message with exact number appears only once.

    Host configuration Lenovo G410

    Toolchain configuration Standard

    Tilck build configuration See attachment above

    Type of issue (select one)

    • Incorrect behavior

    Reproduction details

    1. poweroff and poweroff -f work OK
    2. After the boot:

    ACPI: got GPE #2 (?)

    ACPI: got GPE #27 (closing the screen cover)

    Closing the screen cover again - no ACPI message

    Expected behavior The message after each closing the screen cover.

    bug 
    opened by duselguy 15
  • The function AcpiOsStall doesn't work properly

    The function AcpiOsStall doesn't work properly

    Describe the bug According to the comment in hwsleep.c: ... /* * We wanted to sleep > S3, but it didn't happen (by virtue of the * fact that we are still executing!) * * Wait ten seconds, then try again. This is to get S4/S5 to work on * all machines. * * We wait so long to allow chipsets that poll this reg very slowly * to still read the right value. Ideally, this block would go * away entirely. */ AcpiOsStall (10 * ACPI_USEC_PER_SEC); ...

    It looks like we don't wait. I changed the code in hwsleep.c to check it:

    ...
        ACPI_DEBUG_PRINT ((ACPI_DB_INIT,
            "Entering sleep state [S%u]\n", SleepState));
        AcpiOsStall (10 * ACPI_USEC_PER_SEC); /* Temp changes */
        ACPI_DEBUG_PRINT ((ACPI_DB_INIT, /* Temp changes */
            "The second entering sleep state [S%u]\n", SleepState)); /* Temp changes */
        while (true) {}; /* Temp changes */
    
        /* Clear the SLP_EN and SLP_TYP fields */
    ...
    

    And I see that we don't wait: SAM_0605

    Host configuration Linux Veriton 5.13.0-51-generic #58~20.04.1-Ubuntu SMP Tue Jun 14 11:29:12 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)

    Toolchain configuration Standard.

    Tilck build configuration

    • Tilck commit: see screenshot above

    Type of issue (select one)

    • Incorrect behavior

    Reproduction details Reproducible on different hardware with code changes above.

    Expected behavior Wait requested time.

    Screenshots Screenshot is above.

    Additional context It looks like borrowed modules (for example, utosi.c, actypes.h, hwsleep.c, ... are out of date. Do you have any ideas/plans to synchronize these modules time to time?

    bug 
    opened by duselguy 11
  • Build tcc fails because the related tag is absent in git repo

    Build tcc fails because the related tag is absent in git repo

    Describe the bug [email protected]:~/Project2/tilck$ ./scripts/build_toolchain -s build_tcc

    ARCH=i386 HOST_ARCH=x86_64 GCC_TC_VER=8.4.0 ARCH_GCC_TC_VER=stable-2020.02-2

    CI Detected: None [Assuming user machine] [build_toolchain] Executing single function 'build_tcc' NOTE: Skipping GCC TOOLCHAIN (i386 - musl)

    *** TCC ***

    + local url=git://repo.or.cz/tinycc.git + local tag=15e9b7384e7091200aa33499b1ddcbf6e2f6a959 + local tarname=tcc.tgz + local dirname=tcc + '[' -f /home/vladimir/Project2/tilck/toolchain2/cache/tcc.tgz ']' + '[' -z 15e9b7384e7091200aa33499b1ddcbf6e2f6a959 ']' + git clone --branch 15e9b7384e7091200aa33499b1ddcbf6e2f6a959 --depth 1 git://repo.or.cz/tinycc.git tcc Cloning into 'tcc'... warning: Could not find remote branch 15e9b7384e7091200aa33499b1ddcbf6e2f6a959 to clone. fatal: Remote branch 15e9b7384e7091200aa33499b1ddcbf6e2f6a959 not found in upstream origin + tar cfz tcc.tgz tcc tar: tcc: Cannot stat: No such file or directory tar: Exiting with failure status due to previous errors + mv tcc.tgz /home/vladimir/Project2/tilck/toolchain2/cache + rm -rf tcc + set +x Extracting tcc.tgz... Extracting completed. /home/vladimir/Project2/tilck/scripts/bash_includes/tc/tcc: line 40: cd: tcc: No such file or directory sed: can't read Makefile: No such file or directory Running command: ./configure --enable-cross --cross-prefix=i686-linux- --enable-static --cpu=i386 --prefix=/ --extra-ldflags='-Wl,--oformat=elf32-i386 -static' --crtprefix=/lib/i686-tilck-musl --libpaths=/lib/i686-tilck-musl &> configure.log Running command: make cross-i386 &> build.log i686-linux-strip: 'i386-tcc': No such file [build_toolchain] Executing single function: DONE

    Host configuration

    • Linux distro and version Linux Veriton 5.13.0-41-generic 20.04.1-Ubuntu SMP Wed Apr 20 13:16:21 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

    • Compiler (name and version) gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)

    • Tilck commit: please indicate the exact commit used to reproduce the problem b4ba5e7b1c04bcedbe233252b79e964bc937a201

    Type of issue (select one)

    • Toolchain build problem

    P.S. tcc.tgz in cahe is empty. P.P.S. It's desirable to have minimal error processing in scripts to exit early after the critical errors. Best regards, Vladimir

    bug 
    opened by duselguy 7
  • bogoMips 0 causes a halt/freeze

    bogoMips 0 causes a halt/freeze

    Hi! Oh man what an awesome project. Your blog posts were nice too :D Quite a lot of questions spinning around in my head now, but mainly just wanted to drop in to say "hi, saw what you did there, i approve" :D

    I do have a small bug to report though. I did some testing in UniPCemu and found out that bogoMips goes to zero for some reason (EDIT: looks like it's caused by emulation slowness...__bogo_loops reaches value of 8, so that would do it).

    This causes Tilck to freeze at kb8042 kernel module init, because the nop loop counter (ecx) will flip over to max value. Forcing loops_per_us value to 1 enables Tilck to boot all the way to the shell prompt.

    opened by misutoneko 7
  • Memory leak in compiled GCC.

    Memory leak in compiled GCC.

    Builded the toolchain with a script. Changed it a bit to use pacman for my Artix Linux. The toolchain is builded and the package is installed. However, when the kernel compilation hits 53 percent, I run out of memory and Xorg killed.

    test bug 
    opened by turbocat2001 6
  • Building fbdoom

    Building fbdoom

    I am getting an error when I try to build fbdoom on Ubuntu 20.04 x86_64. Maybe I am missing some 16bit libraries, does anyone knows which ones I need to install ?

    $ ./scripts/build_toolchain -s build_fbdoom
    
    ARCH=i386
    GCC_TC_VER=7.3.0
    ARCH_GCC_TC_VER=stable-2018.11-1
    
    CI Detected: None [Assuming user machine]
    [build_toolchain] Executing single function 'build_fbdoom'
    NOTE: Skipping FBDOOM
    [build_toolchain] Executing single function: DONE
    
    $ cat ./toolchain2/i386/fbDOOM/fbdoom/build.log
    mkdir -p build
    [Compiling i_video_fbdev.c]
    [Compiling i_input_tty.c]
    [Compiling i_main.c]
    [Compiling dummy.c]
    /tmp/tilck/scripts/build_scripts/i686-linux-gcc: 2: exec: -m32: not found
    make: *** [Makefile:61: build/i_video_fbdev.o] Error 127
    /tmp/tilck/scripts/build_scripts/i686-linux-gcc: 2: exec: -m32: not found
    make: *** Waiting for unfinished jobs....
    make: *** [Makefile:61: build/i_input_tty.o] Error 127
    /tmp/tilck/scripts/build_scripts/i686-linux-gcc: 2: exec: -m32: not found
    make: *** [Makefile:61: build/i_main.o] Error 127
    /tmp/tilck/scripts/build_scripts/i686-linux-gcc: 2: exec: -m32: not found
    make: *** [Makefile:61: build/dummy.o] Error 127
    
    opened by johanburati 5
  • Added Lua support

    Added Lua support

    Hey! I got interested in this project and I would like to make a small contribution. I am making this PR to add Lua interpreter support. I have added all the necessary scripts in scripts/bash_includes/tc/lua and also I have added add_lua routine in templates/build_fatpart .

    As you know Lua is a tiny interpreter and it is a cool nice feature to have in the toolchain. Having Lua in the toolchain would allow anyone to build cool new features with Lua scripting language. (Most of the hobby operating systems I have seen has a Lua port).

    None of the kernel functions are modified as a part of this PR.

    I have tried my best to maintain the same coding standards as yours, so that there will be a uniformity. Please review the PR, I am happy to make any changes you request.

    Regards, Prasanna

    opened by Narasimha1997 5
  • False warning about absence of KVM on Arch Linux

    False warning about absence of KVM on Arch Linux

    scripts/templates/qemu/run_qemu checks for presence of KVM in the system with which kvm:

    https://github.com/vvaltchev/tilck/blob/3cde30be0f7d92760a66c99b677868ec822c4a2f/scripts/templates/qemu/run_qemu#L12-L20

    I have KVM installed and the KVM module is loaded into the kernel:

    $ lsmod | grep kvm
    kvm_intel             335872  0
    kvm                  1036288  1 kvm_intel
    irqbypass              16384  1 kvm
    

    But there is no kvm binary, so the check above fails.

    The script also actually runs QEMU with -enable-kvm successfully and info kvm in the QEMU monitor outputs kvm support: enabled.

    I understand that this is only a false warning, not a false error, but maybe there is another way of checking for absence of KVM that works on Arch Linux too?

    opened by ytret 4
  • A note on pointer-to-void to pointer-to-function cast

    A note on pointer-to-void to pointer-to-function cast

    This is a follow up on our discussion on [0] from the other day. I was interested in the details and did some research and I wanted to share what I found, which I hope may be of interest to you as well.

    I checked several answers in StackOverflow and Reddit and then I went forward straight to the source - the C99 standard [1] and I didn't find any clause explicitly saying that casting between pointer to data and pointer to function may yield undefined behavior. It was just not listed explicitly as allowed in the main normative text:

    6.3.2.3 Pointers 1. A pointer to void may be converted to or from a pointer to any incomplete or object type. A pointer to any incomplete or object type may be converted to a pointer to void and back again; the result shall compare equal to the original pointer.

    It is here that pointer to function types are not mentioned and I think this paragraph is what causes some people to consider this undefined behavior.

    [...] 5. An integer may be converted to any pointer type (Petar: including pointer-to-function?). Except as previously specified, the result is implementation-defined, might not be correctly aligned, might not point to an entity of the referenced type, and might be a trap representation.

    I interpret this to mean that if you know the address of a function, the compiler should happily implicitly convert the integer literal to the desired pointer to function type:

    // Create a pointer to the function at address 128:
    int (*fp)() = 128;
    

    And as expected, compiling this with gcc with -Wall -std=c99 yielded no warnings for me.

    1. Any pointer type may be converted to an integer type. Except as previously specified, the result is implementation-defined. If the result cannot be represented in the integer type, the behavior is undefined. The result need not be in the range of values of any integer type.

    Given 5. and 6., one could conclude that if a given compiler provides with an integer type large enough to hold a pointer value (i.e.. intptr_t exists), then converting a pointer-to-data to an integer and then converting that number to a pointer-to-function should be allowed.

    1. A pointer to a function of one type may be converted to a pointer to a function of another type and back again; the result shall compare equal to the original pointer. If a converted pointer is used to call a function whose type is not compatible with the pointed-to type, the behavior is undefined.

    This essentially is saying that you can store a pointer-to-function of one type in a pointer-to-function of a different type, as long as you don't try to call it through the wrong type. (Initially I thought that this property (which I'm sure that you're well aware of) could save you from that cast, and that's why I'm mentioning it, but that's not the case. Anyways...)

    Going a bit further, the Posix standard, which essentially operates on top of a subset of the C standard (I mean it requires things that in theory won't be true on all things that C targets), effectively requires that you to be able to convert between pointer-to-void and pointer-to-function in cases like dlsym [2]:

    The ISO C standard does not require that pointers to functions can be cast back and forth to pointers to data. Indeed, the ISO C standard does not require that an object of type void * can hold a pointer to a function. Implementations supporting the XSI extension, however, do require that an object of type void * can hold a pointer to a function. The result of converting a pointer to a function into a pointer to another data type (except void *) is still undefined, however. Note that compilers conforming to the ISO C standard are required to generate a warning if a conversion from a void * pointer to a function pointer is attempted as in:

    fptr = (int (*)(int))dlsym(handle, "my_function");
    

    From [3]:

    The standard defines two levels of conformance: POSIX conformance, which is a baseline set of interfaces required of a conforming system; and XSI Conformance, which additionally mandates a set of interfaces (the "XSI extension") which are only optional for POSIX conformance. XSI-conformant systems can be branded UNIX 03. (XSI conformance constitutes the Single UNIX Specification version 3 (SUSv3).)

    And indeed, not only most C compilers allow it for typical (i.e. non-exotic and non-embedded targets), it is also mentioned in the C99 standard under Annex J (informative (Petar: note: not normative)) Portability issues, J.5 Common extensions:

    J.5.7 Function pointer casts

    1. A pointer to an object or to void may be cast to a pointer to a function, allowing data to be invoked as a function (6.5.4).
    2. A pointer to a function may be cast to a pointer to an object or to void, allowing a function to be inspected or modified (for example, by a debugger) (6.5.4).

    So, in summary, I believe that the reason that the C standard doesn't explicitly mandate this behavior has nothing to do with the strict aliasing rules, which were a late, poorly executed effort to catch-up in performance with Fortran, but with the fact that back then pure Harvard architectures [4] were something that C needed to support.

    In some systems, there is much more instruction memory than data memory so instruction addresses are wider than data addresses.

    [..]

    Microcontrollers are characterized by having small amounts of program (flash memory) and data (SRAM) memory, and take advantage of the Harvard architecture to speed processing by concurrent instruction and data access. The separate storage means the program and data memories may feature different bit widths, for example using 16-bit-wide instructions and 8-bit-wide data. They also mean that instruction prefetch can be performed in parallel with other activities.

    So in practice, what this means for Tilck is that as long as you only are targeting only modified Harvard architecture systems (main memory holds both instruction and data, but the CPU may have different internal data paths for instructions and data) and also keeping the ELF constructor section technique I think it's reasonable to say that you should be ok with a naive cast. Also I think that if you're targeting an architecture where indeed functions and data are in completely separate address spaces and use different bus widths, the current cast will be just as undefined behavior, as the naive one.

    I don't have any preference about the code style, so you can leave it as it is, I wanted to share my findings, as I know you appreciate understanding things deeply like me ;)

    Cheers, Petar

    opened by PetarKirov 2
  • Stack validation fails sometimes

    Stack validation fails sometimes

    Reproduction:

    [parent] waiting the child to exit...
    [kernel] Pid 12 will WAIT until pid 13 dies
    *** DEBUG: attempt to write COW page at 0xbffff000 (addr: 0xbffffef0)
    *** DEBUG: the page was not shared anymore. Making it writable.
    Fork returned 0
    ############## I'm the child!
    [thread signal]: under lock, signal_all!
    [thread signal]: exit
    ****** [kernel thread] EXIT (pid: 8)
    [validate stack] real stack page:       0xc01ff000
    [validate stack] current->kernel_stack: 0xc441c000
    ************************ KERNEL PANIC ************************
    Invalid kernel stack pointer.
    File /home/travis/build/darthvader88/experimentOs/kernel/interrupts.c at line 67
    Current process: 8 [KERNEL]
    Interrupts: [ ]
    *** KERNEL STACKTRACE ***
    frame[0]: 0xc0102ee5
    frame[1]: 0xc0105432
    frame[2]: 0xc0103658
    frame[3]: 0xc0100473
    *** END STACKTRACE ***
    
    bug 
    opened by vvaltchev 2
Owner
Vladislav Valtchev
Kernel hacker, former Senior C++ platform engineer at VMware.
Vladislav Valtchev
A port of the Linux x86 IOLI crackme challenges to x86-64

This is a port of the original Linux x86 IOLI crackme binaries to x86-64. The original set of IOLI crackmes can be found here: https://github.com/Maij

Julian Daeumer 4 Mar 19, 2022
C library designed for the TI MSP432P401R microprocessor and the TI Educational Booster Pack, to easily play and control songs with the integrated Piezo Buzzer.

MusicLib C library designed for the TI MSP432P401R microprocessor and the TI Educational Booster Pack, to easily play and control songs with the integ

Matteo Merler 1 Nov 24, 2021
TinyDBR is meant for tiny dynamic binary rewriter fox x86 instruction set

TinyDBR What is TinyDBR? TinyDBR is meant for tiny dynamic binary rewriter fox x86 instruction set. This is a port to the TinyInst by Google Project Z

Asuka 34 Jun 20, 2022
A low level Operating System designed using Linux Kernel

Conqueror A low level Operating System designed using Linux Kernel To develop the basic low level operating system, we need following Virtual Machine

mahendra gandham 8 May 27, 2022
💧 A simple x86-64 kernel written in C

HydrOS About HydrOS aims to be a simple yet well documented kernel for the x86-64 architecture. Why HydrOS Hydros (that's why OS looks so nice there)

dacousb 15 Jun 15, 2022
A C library for runtime-flippable feature flags on Linux/x86-64, with negligible overhead in the common case

Biased runtime-flippable flags, with cross-modifying code The dynamic_flag library is a two-file "C" library that offers efficient biased conditionals

Backtrace Labs 57 May 24, 2022
Emergency alert and tracer for realtime high-performance computing app (work in progress, currently supported env is only Linux x86-64).

HPC Emerg Emergency alert and tracer for realtime high-performance computing app (work in progress, currently supported env is only Linux x86-64). Exa

Ammar Faizi 7 Jan 19, 2022
Unix pager (with very rich functionality) designed for work with tables. Designed for PostgreSQL, but MySQL is supported too. Works well with pgcli too. Can be used as CSV or TSV viewer too. It supports searching, selecting rows, columns, or block and export selected area to clipboard.

Unix pager (with very rich functionality) designed for work with tables. Designed for PostgreSQL, but MySQL is supported too. Works well with pgcli too. Can be used as CSV or TSV viewer too. It supports searching, selecting rows, columns, or block and export selected area to clipboard.

Pavel Stehule 1.8k Jun 21, 2022
The purpose of these streams is to be educational and entertaining for viewers to learn about systems architecture, reverse engineering, software security, etc., and NOT to encourage nor endorse malicious game hacking.

Memestream This repository holds the code that I develop during my live game "modding" ?? sessions. When I stream, I like to speedrun making a success

Stephen Tong 27 Jun 17, 2022
Educational material and examples for those interested in learning the C programming language

Learn C Educational material and examples for those interested in learning the C programming language Files: examples.c Various examples of programs w

Michael Kolesidis 14 May 15, 2022
Accepted solutions to the Atcoder Educational DP Contest

atcoder-educational-dp Accepted solutions to the Atcoder Educational DP Contest A - Frog 1 B - Frog 2 C - Vacation D - Knapsack 1 E - Knapsack 2 F - L

Sachin Srivastava 1 Dec 10, 2021
Unox is an educational unix-like operating system. #JustForFun

Unox Unox is a x86_64 unix-like operating system written in c and assembly for learn about operating systems. Roadmap Write bootloader and linker Writ

Hasan Kashi 19 May 3, 2022
Educational Administration Platform Emulator(TUI)

Educational Administration Platform Emulator This is an simple emulator aim to implement Educational Administration Platform base on terminal user int

null 1 Dec 30, 2021
C parsing, semantic analys, generate a graph from a source code. An educational project during my third year of Computer Science Licence.

Pour compiler le programme, il suffit d'exécuter compiler.sh avec la commande "./compiler.sh" en se trouvant dans le dossier racine du projet. Un fich

Jean Philippe Carlens 3 Feb 3, 2022
Made for educational purposes.

Tested on Win10x64 20H2 and not latest gepard Last test some months ago Features: Breakpoints (HWBP & INT3 trap flag) Integrity of modified files on g

Charles 4 May 14, 2022
A kernel module that patches Linux kernel "on-the-fly" to skip TASK_RSS_EVENTS_THRESH check in check_sync_rss_stat

split-rss-counting-patch A kernel module that patches Linux kernel "on-the-fly" to skip TASK_RSS_EVENTS_THRESH check in check_sync_rss_stat. Why? Read

Bao-Hiep Le 3 Mar 6, 2022
The source for the Linux kernel used in Windows Subsystem for Linux 2 (WSL2)

Introduction The WSL2-Linux-Kernel repo contains the kernel source code and configuration files for the WSL2 kernel. Reporting Bugs If you discover an

Microsoft 5.8k Jun 23, 2022
a small C library for x86 CPU detection and feature extraction

libcpuid libcpuid provides CPU identification for the x86 (and x86_64). For details about the programming API, you might want to take a look at the pr

Veselin Georgiev 329 May 31, 2022