Perf-ninja - This is an online course where you can learn and master the skill of low-level performance analysis and tuning.

Overview

Performance Ninja Class

This is an online course where you can learn to find and fix low-level performance issues, for example CPU cache misses and branch mispredictions. It's all about practice. So we offer you this course in a form of lab assignments and youtube videos. You will spend at least 90% of the time analyzing performance of the code and trying to improve it.

Each lab assignment focuses on a specific performance problem and can take anywhere from 30 mins up to 4 hours depending on your background and the complexity of the lab assignment itself. Once you're done improving the code, you can submit your solution to Github for automated benchmarking and verification.

Before you start working on lab assignments, make sure you read Get Started page and watch the warmup video. Join our discord channel to collaborate with others.

Lab assignments

Support the project

Performance Ninja is in a very much work-in-progress state. We will be adding new lab assignments and videos! The course is free by default, but we ask you to support us on Github Sponsors, Patreon or PayPal. Your sponsorship will speed up adding new lab assignments.

Current sponsors:

  • Pavel Davydov
  • Maya Lekova (@MayaLekova)
  • Aaron St. George (@AaronStGeorge)

Thanks to Mansur Mavliutov (@Mansur) for providing an AMD-based machine for running CI jobs.

Contributing

We warmly welcome contributions! See Contributing.md for the details.

Please write to [email protected] with suggestions.

Copyright © 2021 by Denis Bakhvalov under Creative Commons license (CC BY 4.0).

Issues
  • MSVC 2019 fails to build warmup lab

    MSVC 2019 fails to build warmup lab

    Hello and thank you for the outstanding course!

    I built a benchmark with the use of make_benchmark_library_vs2019.cmd script first. But then, when I tried to build a warmup library, it failed. Am I missing anything? The full log is below:

    PS C:\Work\perf-ninja\labs\misc\warmup> cmake -E make_directory build
    PS C:\Work\perf-ninja\labs\misc\warmup> cd build
    PS C:\Work\perf-ninja\labs\misc\warmup\build> cmake -DCMAKE_BUILD_TYPE=Release ..
    -- Building for: Visual Studio 16 2019
    -- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.22000.
    -- The C compiler identification is MSVC 19.29.30141.0
    -- The CXX compiler identification is MSVC 19.29.30141.0
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped
    -- Detecting C compile features
    -- Detecting C compile features - done
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    -- Performing Test SUPPORT_MSVC_AVX512
    -- Performing Test SUPPORT_MSVC_AVX512 - Failed
    -- Performing Test SUPPORT_MSVC_AVX2
    -- Performing Test SUPPORT_MSVC_AVX2 - Success
    -- Performing Test SUPPORT_MSVC_AVX
    -- Performing Test SUPPORT_MSVC_AVX - Success
    -- Configuring done
    -- Generating done
    -- Build files have been written to: C:/Work/perf-ninja/labs/misc/warmup/build
    PS C:\Work\perf-ninja\labs\misc\warmup\build> cmake --build . --config Release --parallel 8
    Microsoft (R) Build Engine version 16.11.2+f32259642 for .NET Framework
    Copyright (C) Microsoft Corporation. All rights reserved.
    
      Checking Build System
      Building Custom Rule C:/Work/perf-ninja/labs/misc/warmup/CMakeLists.txt
      Building Custom Rule C:/Work/perf-ninja/labs/misc/warmup/CMakeLists.txt
      bench.cpp
      validate.cpp
    C:\Work\perf-ninja\tools\benchmark\include\benchmark/benchmark.h(190,10): fatal error C1083: Cannot open include file: 'benchmark/export.h': No such file or d
    irectory [C:\Work\perf-ninja\labs\misc\warmup\build\lab.vcxproj]
      solution.cpp
      Generating Code...
      solution.cpp
      Generating Code...
      validate.vcxproj -> C:\Work\perf-ninja\labs\misc\warmup\build\Release\validate.exe
    
    opened by operasfantom 8
  • Provide a script to locally benchmark against the baseline

    Provide a script to locally benchmark against the baseline

    Request I recieved: "It would be nice if there was a script or something to run the baseline program and the improved program to tell me how well I'm faring, instead of having to take note what was the latest score, what was the baseline score etc. It would reduce the ping-ponging with CI."

    opened by dendibakh 8
  • The solution of `data packing` in the summary video fail to pass the CI.

    The solution of `data packing` in the summary video fail to pass the CI.

    Hello Denis,

    Following solution in the summary video failed to pass the CI for the data packing lab.

    struct S {
      float d;
      long long l:16;
      int i:8;
      short s:7;
      bool b:1;
    
      bool operator<(const S &s) const { return this->i < s.i; }
    };
    

    In particular, short s:7; failed to pass the validation.

    Scanning dependencies of target validateLab
    Validation Failed. Value s is -28. Expected is 100 for intialization values 0 and 100
    Validation Failed. Value s is -29. Expected is 99 for intialization values 1 and 99
    Validation Failed. Value s is -28. Expected is 100 for intialization values 100 and 100
    make[3]: *** [CMakeFiles/validateLab.dir/build.make:57: CMakeFiles/validateLab] Error 2
    make[2]: *** [CMakeFiles/Makefile2:134: CMakeFiles/validateLab.dir/all] Error 2
    make[1]: *** [CMakeFiles/Makefile2:141: CMakeFiles/validateLab.dir/rule] Error 2
    make: *** [Makefile:144: validateLab] Error 2
    Will benchmark the lab: memory_bound:data_packing
    Build and Validate the solution
    Prepare build directory - OK
    CMake - OK
    Build - OK
    Validation - Failed
    

    If we just set short s;, then the new version would be slower:

    Benchmark                   Time             CPU      Time Old      Time New       CPU Old       CPU New
    --------------------------------------------------------------------------------------------------------
    bench1                   +0.0307         -0.0028           419           432           420           419
    New version is slower. Submission for the lab memory_bound:data_packing failed.
    
    opened by fuyw 6
  • Permission denied

    Permission denied

    Hi! I tried to push my private branch, but I get the following error :

    ERROR: Permission to dendibakh/perf-ninja.git denied to goandrei.
    fatal: Could not read from remote repository.
    
    Please make sure you have the correct access rights
    and the repository exists.
    

    Is the course still free to use?

    opened by goandrei 3
  • [Bug] Local experiments do not run

    [Bug] Local experiments do not run

    Hi, thanks again for this wonderful repository.

    Running the local experiments (as described in GetStarted.md) fails with the following output.

    λ python3 tools/check_speedup.py -lab_path labs/misc/warmup -bench_lib_path tools/benchmark -num_runs 1 -v
    Running: cmake -E make_directory baselineBuild0
    Running: cmake -G Ninja -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER=clang -DCMAKE_BUILD_TYPE=Release  "labs/misc/warmup"
    CMake Error: The source directory "/Users/tp5/code/perf-ninja/baselineBuild0/labs/misc/warmup" does not exist.
    Specify --help for usage, or press the help button on the CMake GUI.
    baseline: iteration 0 - Failed
    Running: cmake -E make_directory solutionBuild0
    Running: cmake -G Ninja -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER=clang -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS=-DSOLUTION "labs/misc/warmup"
    CMake Error: The source directory "/Users/tp5/code/perf-ninja/solutionBuild0/labs/misc/warmup" does not exist.
    Specify --help for usage, or press the help button on the CMake GUI.
    solution: iteration 0 - Failed
    Running: python "tools/benchmark/tools/compare.py" benchmarks "/Users/tp5/code/perf-ninja/baselineBuild0/result.json" "/Users/tp5/code/perf-ninja/solutionBuild0/result.json"
    usage: compare.py benchmarks [-h] test_baseline test_contender ...
    compare.py benchmarks: error: argument test_baseline: can't open '/Users/tp5/code/perf-ninja/baselineBuild0/result.json': [Errno 2] No such file or directory: '/Users/tp5/code/perf-ninja/baselineBuild0/result.json'
    

    The link to the source directory seems incorrect. I could fix the error locally on my machine by modifying the following

    https://github.com/dendibakh/perf-ninja/blob/d2d6b12f734a897f95fb3e6a568cd2b6f5088263/tools/check_speedup.py#L62

    to

        callWrapper("cmake -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER=clang -DCMAKE_BUILD_TYPE=Release " + cmakeFlags + " \"../" + labRootPath + "\"")
    

    (I added a ../) before the labRootPath. Can you take a look if this bug is reproducible or if it's something that's incorrectly configured on my machine? Thanks.

    opened by tp5uiuc 3
  • Address differences in measurements between different platforms

    Address differences in measurements between different platforms

    Sometimes we have to deal with different speedups on different platforms, e.g. Linux vs. Windows. This may cause CI failures on one platform but success on another, which is annoying.

    Example: measurements for two CI jobs for the same commit differ by 2x: Linux: https://github.com/dendibakh/perf-ninja/actions/runs/1382087200 Windows: https://github.com/dendibakh/perf-ninja/actions/runs/1382087204

    Possible solutions:

    • Define two sets of thresholds for Windows and Linux
    • For big differences (like 2x) investigate the reason. Maybe something is not configured properly on one of the platforms.
    opened by dendibakh 3
  • Support pre-AVX2 machines

    Support pre-AVX2 machines

    This fixes issue #2. For GCC/Clang compilers it is safe to leave -march=native since they will automatically determine the target arch. But for MSVC I'm not sure since the default for x86 in MSVC is SSE2. For now, let's not touch it.

    opened by dendibakh 3
  • workflow CI is not running.

    workflow CI is not running.

    Hello Dan,

    The build is not running when pushing the code. The terminal is showing the following message

    Requested labels: self-hosted, Linux Job defined at: dendibakh/perf-ninja/.github/workflows/[email protected]/heads/private/aschhabra Waiting for a runner to pick up this job...

    Benchmark on Linux

    opened by aschhabra 1
  • Windows, multiconfig generator: improve comparison

    Windows, multiconfig generator: improve comparison

    I added some cmake options to suppress DEBUG configuration warning and to simplify comparisons under MSVC. I hope someone will check workability under Linux: -B and -S options.

    opened by andrewevstyukhin 1
  • Allow to define CI benchmarking time for each test individually

    Allow to define CI benchmarking time for each test individually

    Right now the execution time is hardcoded as 1 sec for each benchmark. I think we should allow increasing time for individual benchmarks to make them more stable in CI. Changes in https://github.com/dendibakh/perf-ninja/blob/main/buildbot/runCI.py needed. We need to extend class LabParams.

    opened by dendibakh 1
  • Illegal instruction

    Illegal instruction

    Hi, I was building the warm up lab and encountered this error:

    $cmake --build . --target validateLab
    Consolidate compiler generated dependencies of target validate
    [100%] Built target validate
    make[3]: *** [CMakeFiles/validateLab.dir/build.make:70: CMakeFiles/validateLab] Illegal instruction (core dumped)
    make[2]: *** [CMakeFiles/Makefile2:139: CMakeFiles/validateLab.dir/all] Error 2
    make[1]: *** [CMakeFiles/Makefile2:146: CMakeFiles/validateLab.dir/rule] Error 2
    make: *** [Makefile:150: validateLab] Error 2
    

    My system info

    $uname -a
    Linux pc 5.12.19-1-MANJARO #1 SMP PREEMPT Tue Jul 20 20:57:37 UTC 2021 x86_64 GNU/Linux
    
    $lscpu
    Architecture:            x86_64
      CPU op-mode(s):        32-bit, 64-bit
      Address sizes:         36 bits physical, 48 bits virtual
      Byte Order:            Little Endian
    CPU(s):                  8
      On-line CPU(s) list:   0-7
    Vendor ID:               GenuineIntel
      Model name:            Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
        CPU family:          6
        Model:               58
        Thread(s) per core:  2
        Core(s) per socket:  4
        Socket(s):           1
        Stepping:            9
        CPU max MHz:         3400,0000
        CPU min MHz:         1600,0000
        BogoMIPS:            6787.59
        Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr s
                             se sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopolog
                             y nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr
                             pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fa
                             ult epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt
                             dtherm arat pln pts md_clear flush_l1d
    
    opened by tomekwes 1
  • Having trouble with warmup

    Having trouble with warmup

    Hello, I'm looking forward following the course, but I ran into an issue at the start:

    When I do cmake --build . --config Release --parallel 8 for the warmup, I get the error: undefined reference to 'main'

    With --verbose=1:

    ~/workspace/perf-ninja/labs/misc/warmup/build$ cmake --build . --config Release --parallel 8 -v
    /snap/cmake/1005/bin/cmake -S/home/jhovan/workspace/perf-ninja/labs/misc/warmup -B/home/jhovan/workspace/perf-ninja/labs/misc/warmup/build --check-build-system CMakeFiles/Makefile.cmake 0
    /snap/cmake/1005/bin/cmake -E cmake_progress_start /home/jhovan/workspace/perf-ninja/labs/misc/warmup/build/CMakeFiles /home/jhovan/workspace/perf-ninja/labs/misc/warmup/build//CMakeFiles/progress.marks
    /usr/bin/make  -f CMakeFiles/Makefile2 all
    make[1]: Entering directory '/home/jhovan/workspace/perf-ninja/labs/misc/warmup/build'
    /usr/bin/make  -f CMakeFiles/lab.dir/build.make CMakeFiles/lab.dir/depend
    /usr/bin/make  -f CMakeFiles/validate.dir/build.make CMakeFiles/validate.dir/depend
    make[2]: Entering directory '/home/jhovan/workspace/perf-ninja/labs/misc/warmup/build'
    cd /home/jhovan/workspace/perf-ninja/labs/misc/warmup/build && /snap/cmake/1005/bin/cmake -E cmake_depends "Unix Makefiles" /home/jhovan/workspace/perf-ninja/labs/misc/warmup /home/jhovan/workspace/perf-ninja/labs/misc/warmup /home/jhovan/workspace/perf-ninja/labs/misc/warmup/build /home/jhovan/workspace/perf-ninja/labs/misc/warmup/build /home/jhovan/workspace/perf-ninja/labs/misc/warmup/build/CMakeFiles/lab.dir/DependInfo.cmake --color=
    make[2]: Entering directory '/home/jhovan/workspace/perf-ninja/labs/misc/warmup/build'
    cd /home/jhovan/workspace/perf-ninja/labs/misc/warmup/build && /snap/cmake/1005/bin/cmake -E cmake_depends "Unix Makefiles" /home/jhovan/workspace/perf-ninja/labs/misc/warmup /home/jhovan/workspace/perf-ninja/labs/misc/warmup /home/jhovan/workspace/perf-ninja/labs/misc/warmup/build /home/jhovan/workspace/perf-ninja/labs/misc/warmup/build /home/jhovan/workspace/perf-ninja/labs/misc/warmup/build/CMakeFiles/validate.dir/DependInfo.cmake --color=
    make[2]: Leaving directory '/home/jhovan/workspace/perf-ninja/labs/misc/warmup/build'
    make[2]: Leaving directory '/home/jhovan/workspace/perf-ninja/labs/misc/warmup/build'
    /usr/bin/make  -f CMakeFiles/lab.dir/build.make CMakeFiles/lab.dir/build
    /usr/bin/make  -f CMakeFiles/validate.dir/build.make CMakeFiles/validate.dir/build
    make[2]: Entering directory '/home/jhovan/workspace/perf-ninja/labs/misc/warmup/build'
    make[2]: Nothing to be done for 'CMakeFiles/validate.dir/build'.
    make[2]: Leaving directory '/home/jhovan/workspace/perf-ninja/labs/misc/warmup/build'
    make[2]: Entering directory '/home/jhovan/workspace/perf-ninja/labs/misc/warmup/build'
    [ 50%] Linking CXX executable lab
    [ 66%] Built target validate
    /snap/cmake/1005/bin/cmake -E cmake_link_script CMakeFiles/lab.dir/link.txt --verbose=1
    /usr/bin/clang++-12 -O3 -ffast-math -march=native   -rdynamic CMakeFiles/lab.dir/bench.cpp.o CMakeFiles/lab.dir/solution.cpp.o -o lab  /usr/local/lib/libbenchmark.a -lpthread -lm 
    /usr/bin/ld: /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/crt1.o: in function `_start':
    (.text+0x24): undefined reference to `main'
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    make[2]: *** [CMakeFiles/lab.dir/build.make:114: lab] Error 1
    make[2]: Leaving directory '/home/jhovan/workspace/perf-ninja/labs/misc/warmup/build'
    make[1]: *** [CMakeFiles/Makefile2:87: CMakeFiles/lab.dir/all] Error 2
    make[1]: Leaving directory '/home/jhovan/workspace/perf-ninja/labs/misc/warmup/build'
    make: *** [Makefile:91: all] Error 2
    

    Any help would be greatly appreciated!

    opened by nidpez 4
  • Add MAC support

    Add MAC support

    We need to have Quickstart instructions, similar to https://github.com/dendibakh/perf-ninja/blob/main/QuickstartWindows.md It may be hard to find an idle Mac machine for the CI, but it would be nice to have one. :)

    opened by dendibakh 0
  • Improve validation in loop_interchange_1

    Improve validation in loop_interchange_1

    Comment by @OleksandrKvl :

    Current validation is strange in a way that you're using other functions from solution.cpp, not just the main power(). Ideally, you should have separate implementation inside validate.cpp, multiple some matrices with functions from validate.cpp and solution.cpp and just compare their results.

    opened by dendibakh 6
Owner
Denis Bakhvalov
Denis Bakhvalov
Boost::ASIO low-level redis client (connector)

bredis Boost::ASIO low-level redis client (connector), github gitee Features header only zero-copy (currently only for received replies from Redis) lo

Ivan Baidakou 142 Jul 21, 2022
TLS 1.3 implementation in C (master supports RFC8446 as well as draft-26, -27, -28)

picotls Picotls is a TLS 1.3 (RFC 8446) protocol stack written in C, with the following features: support for three crypto engines "OpenSSL" backend u

H2O 392 Jul 31, 2022
Encapsulates the two protocols of OpenVpn and Ikev2, you only need to enter the server IP and port number to realize the connection and status display, and the specific situation of the connection can be displayed at the same time。

NewVpnCore 封装了OpenVpn和Ikev2两种协议,只需要输入服务器IP和端口号即可实现连接和状态显示,同时可以显示连接的具体情况。 UniteVpn Core(第一版) 1. 模块说明 unitevpn:封装了vpn的操作和统一不同协议信息的模块 ikev2:IKEV2协议的源码 op

ZFashion 3 Jun 8, 2022
Online chess platform (client-server) in Python with StockFish API

PyChess Gra w szachy tylko w Pythonie :) Wymagania Python 3.8 Instalacja Wchodzimy i pobieramy najnowsze wydanie aplikacji. https://github.com/Rafixe

Rafał Hrabia 6 Oct 7, 2021
Run WPS PIN attacks (Pixie Dust, online bruteforce, PIN prediction) without monitor mode with the wpa_supplicant

Overview OneShot-C - implementation of OneShot on C OneShot-С performs Pixie Dust attack without having to switch to monitor mode. Features Pixie Dust

null 14 Aug 3, 2022
LANDrop is a cross-platform tool that you can use to conveniently transfer photos, videos, and other types of files to other devices on the same local network.

LANDrop is a cross-platform tool that you can use to conveniently transfer photos, videos, and other types of files to other devices on the same local network.

LANDrop 2.9k Aug 12, 2022
Simple useful interoperability tests for WebRTC libraries. If you are a WebRTC library developer we'd love to include you!

Overview This project aims to be a convenient location for WebRTC library developers to perform interoperability tests. Who can Participate The projec

Aaron Clauson 94 Jul 29, 2022
About Add any Program in any language you like or add a hello world Program ❣️ if you like give us ⭐

Hello-World About Add any Program in any language you like or add a hello world Program ❣️ if you like give us ⭐ Give this Project a Star ⭐ If you lik

Lokesh Jangid 15 Aug 5, 2022
As a Teaching Assistant, this is a sample project about socket programming for my teaching in a capstone course in NTUST(National Taiwan University of Science and Technology)

socket-programming As a Teaching Assistant, this is a sample project about socket programming for my teaching in a capstone course in NTUST(National T

Chang Wei 2 Oct 26, 2021
OTUS C++ course demo day examples

coroutines-epoll-example OTUS C++ course demo day examples Инструкция по сборке Необходимы следующие версии компонентов g++11 cmake >= 3.10 git clone

sdukshis 2 Dec 19, 2021
This is a group project from my CPE353 course. It is done in C++ intended for use with QtCreator.

fishinggame This is a group project from my CPE353 course. It is done in C++ intended for use with QtCreator. Credit is absolutely due to my teammates

Ben Bruzewski 1 Jan 8, 2022
Ultra fast and low latency asynchronous socket server & client C++ library with support TCP, SSL, UDP, HTTP, HTTPS, WebSocket protocols and 10K connections problem solution

CppServer Ultra fast and low latency asynchronous socket server & client C++ library with support TCP, SSL, UDP, HTTP, HTTPS, WebSocket protocols and

Ivan Shynkarenka 867 Aug 8, 2022
LAppS - Lua Application Server for micro-services with default communication over WebSockets. The fastest and most vertically scalable WebSockets server implementation ever. Low latency C++ <-> Lua stack roundtrip.

LAppS - Lua Application Server This is an attempt to provide very easy to use Lua Application Server working over WebSockets protocol (RFC 6455). LApp

null 47 Apr 25, 2022
Flexible low latency all-in-one arcade USB adaptor

Imperium Imperium aims to be a flexible low latency all-in-one arcade USB adaptor. The project started out as a replacement controller for the Micro C

Doug Stevens 4 May 24, 2022
Winpcap-based network packet capture tool, support TLS (part), UDP, ICMP, TCP, ARP, DNS and other protocol analysis, interface reference wireshark.

Winpcap-based network packet capture tool, support TLS (part), UDP, ICMP, TCP, ARP, DNS and other protocol analysis, interface reference wireshark.

null 38 Aug 5, 2022
A software C library designed to extract data attributes from network packets, server logs, and from structured events in general, in order to make them available for analysis

MMT-DPI A software C library desinged to extract data attributes from network packets, server logs, and from structured events in general, in odrder t

Montimage 3 Apr 14, 2022
High-level networking API for real-time simulations with primitives for remote procedure call and object state replication

tnl2 - Torque Network Library version 2 tnl2 is a high-level networking API for real-time simulations with primitives for remote procedure call and o

Mark Frohnmayer 23 Apr 10, 2022
Linux Application Level Firewall based on eBPF and NFQUEUE.

eBPFSnitch eBPFSnitch is a Linux Application Level Firewall based on eBPF and NFQUEUE. It is inspired by OpenSnitch, and Douane, but utilizing modern

Harpo Roeder 650 Aug 3, 2022
Level up your Beat Saber experience on Quest! AnyTweaks provides various tweaks to help boost your experience on Quest, such as Bloom, FPS Counter and more.

Need help/support? Ask in one of BSMG's support channels for Quest, or join my Discord server! AnyTweaks Level up your Beat Saber experience on Quest!

kaitlyn~ 14 Jul 28, 2022