Screenshots
Only screenshots and sample for negative Fontsizes. @astiob: can you add the "large positive fontsize" sample mentioned on IRC?
xy-VSFilter (xy-VSFilter bundled in Aegisub r8250, Windows 7, 32-bit binary):

libass since #615 (before it crashed)

Description of the issue
For negative fontsizes VSFilter/GDI uses something like the absolute size, but not quite. The rendered size is a bit larger than expected and the difference is larger than just a different rounding direction of the final value.
For large positive values apparently something similar happens.
libass version
8e80f49ba70d6f16d8f244a7f41dfeee98d416df
ASS Sample
Font is version 2.37 of DejaVu Sans
[Script Info]
ScriptType: v4.00+
WrapStyle: 0
ScaledBorderAndShadow: yes
YCbCr Matrix: None
PlayResX: 1280
PlayResY: 720
Video File: ?dummy:25.000000:40000:1280:720:47:191:225:
Video AR Value: 1.777778
[Aegisub Project Garbage]
Video File: ?dummy:25.000000:40000:1280:720:47:191:225:
Video AR Value: 1.777778
[V4+ Styles]
Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
Style: sm48,DejaVu Sans,-48,&H00FFFFFF,&H000000FF,&H00000000,&H00000000,-1,0,0,0,100,100,0,0,1,4,2,8,10,10,10,1
Style: sm16,DejaVu Sans,-16,&H00FFFFFF,&H000000FF,&H00000000,&H00000000,-1,0,0,0,100,100,0,0,1,4,2,8,10,10,10,1
Style: sm4,DejaVu Sans,-4,&H00FFFFFF,&H000000FF,&H00000000,&H00000000,-1,0,0,0,100,100,0,0,1,4,2,8,10,10,10,1
Style: sm1,DejaVu Sans,-1,&H00FFFFFF,&H000000FF,&H00000000,&H00000000,-1,0,0,0,100,100,0,0,1,4,2,8,10,10,10,1
Style: Default,DejaVu Sans,100,&H0011FF11,&H000000FF,&H00000000,&H00000000,-1,0,0,0,100,100,0,0,1,4,2,8,10,10,10,1
Style: s48,DejaVu Sans,48,&H00FFFFFF,&H000000FF,&H00000000,&H00000000,-1,0,0,0,100,100,0,0,1,4,2,8,10,10,10,1
Style: s16,DejaVu Sans,16,&H00FFFFFF,&H000000FF,&H00000000,&H00000000,-1,0,0,0,100,100,0,0,1,4,2,8,10,10,10,1
Style: s4,DejaVu Sans,4,&H00FFFFFF,&H000000FF,&H00000000,&H00000000,-1,0,0,0,100,100,0,0,1,4,2,8,10,10,10,1
Style: s1,DejaVu Sans,1,&H00FFFFFF,&H000000FF,&H00000000,&H00000000,-1,0,0,0,100,100,0,0,1,4,2,8,10,10,10,1
Style: bordstyle3,DejaVu Sans,42,&H00EEEEEE,&H000000FF,&H00000000,&H00000000,-1,0,0,0,100,100,0,0,3,0,0,8,10,10,10,1
Style: s22,DejaVu Sans,22,&H00FFFFFF,&H000000FF,&H00000000,&H00000000,-1,0,0,0,100,100,0,0,1,4,0,9,10,10,10,1
Style: sm22,DejaVu Sans,-22,&H00FFFFFF,&H000000FF,&H00000000,&H00000000,-1,0,0,0,100,100,0,0,1,4,0,7,10,10,10,1
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
Dialogue: 0,0:00:00.00,0:00:05.00,sm48,,0,0,0,,{\an7}A
Dialogue: 0,0:00:00.00,0:00:05.00,sm48,,0,0,0,,{\an7}B
Dialogue: 0,0:00:00.00,0:00:05.00,Default,,0,0,0,,{\an7}{\fs-48}D
Dialogue: 0,0:00:00.00,0:00:05.00,sm48,,0,0,0,,{\an7}C
Dialogue: 0,0:00:00.00,0:00:05.00,sm16,,0,0,0,,{\an7}E
Dialogue: 0,0:00:00.00,0:00:05.00,sm16,,0,0,0,,{\an7}F
Dialogue: 0,0:00:00.00,0:00:05.00,Default,,0,0,0,,{\an7}{\fs-16}G
Dialogue: 0,0:00:00.00,0:00:05.00,sm16,,0,0,0,,{\an7}H
Dialogue: 0,0:00:00.00,0:00:05.00,sm4,,0,0,0,,{\an7}I
Dialogue: 0,0:00:00.00,0:00:05.00,sm4,,0,0,0,,{\an7}J
Dialogue: 0,0:00:00.00,0:00:05.00,Default,,0,0,0,,{\an7}{\fs-4}K
Dialogue: 0,0:00:00.00,0:00:05.00,sm4,,0,0,0,,{\an7}L
Dialogue: 0,0:00:00.00,0:00:05.00,sm1,,0,0,0,,{\an7}M
Dialogue: 0,0:00:00.00,0:00:05.00,sm1,,0,0,0,,{\an7}N
Dialogue: 0,0:00:00.00,0:00:05.00,Default,,0,0,0,,{\an7}{\fs-1}O
Dialogue: 0,0:00:00.00,0:00:05.00,sm1,,0,0,0,,{\an7}P
Dialogue: 0,0:00:00.00,0:00:05.00,s48,,0,0,0,,{\an9}A
Dialogue: 0,0:00:00.00,0:00:05.00,s48,,0,0,0,,{\an9}B
Dialogue: 0,0:00:00.00,0:00:05.00,Default,,0,0,0,,{\an9}{\fs48}D
Dialogue: 0,0:00:00.00,0:00:05.00,s48,,0,0,0,,{\an9}C
Dialogue: 0,0:00:00.00,0:00:05.00,s16,,0,0,0,,{\an9}E
Dialogue: 0,0:00:00.00,0:00:05.00,s16,,0,0,0,,{\an9}F
Dialogue: 0,0:00:00.00,0:00:05.00,Default,,0,0,0,,{\an9}{\fs16}G
Dialogue: 0,0:00:00.00,0:00:05.00,s16,,0,0,0,,{\an9}H
Dialogue: 0,0:00:00.00,0:00:05.00,s4,,0,0,0,,{\an9}I
Dialogue: 0,0:00:00.00,0:00:05.00,s4,,0,0,0,,{\an9}J
Dialogue: 0,0:00:00.00,0:00:05.00,Default,,0,0,0,,{\an9}{\fs4}K
Dialogue: 0,0:00:00.00,0:00:05.00,s4,,0,0,0,,{\an9}L
Dialogue: 0,0:00:00.00,0:00:05.00,s1,,0,0,0,,{\an9}M
Dialogue: 0,0:00:00.00,0:00:05.00,s1,,0,0,0,,{\an9}N
Dialogue: 0,0:00:00.00,0:00:05.00,Default,,0,0,0,,{\an9}{\fs1}O
Dialogue: 0,0:00:00.00,0:00:05.00,s1,,0,0,0,,{\an9}P
Dialogue: 0,0:00:00.00,0:00:15.00,bordstyle3,,0,0,0,,{\an2}Borderstyle=3 Outline=0{test if classic/xy VSF or MPC-HC ISR?}
Dialogue: 0,0:00:05.01,0:00:10.00,sm22,,0,0,0,,Style Neg:
Dialogue: 0,0:00:05.01,0:00:10.00,sm22,,0,0,0,,{\fs-36}W{ 57.2}
Dialogue: 0,0:00:05.01,0:00:10.00,sm22,,0,0,0,,{\fs-22}W{ 26.4}
Dialogue: 0,0:00:05.01,0:00:10.00,sm22,,0,0,0,,{\fs-11}W{ 2.2}
Dialogue: 0,0:00:05.01,0:00:10.00,sm22,,0,0,0,,W
Dialogue: 0,0:00:05.01,0:00:10.00,sm22,,0,0,0,,{\fs+11}W{ -46.2}
Dialogue: 0,0:00:05.01,0:00:10.00,sm22,,0,0,0,,{\fs+22}W{ -70.4}
Dialogue: 0,0:00:05.01,0:00:10.00,sm22,,0,0,0,,{\fs+36}W{ -101.2}
Dialogue: 0,0:00:05.01,0:00:10.00,s22,,0,0,0,,{\an9}Style Pos:
Dialogue: 0,0:00:05.01,0:00:10.00,s22,,0,0,0,,{\fs-36}W{ -57.2}
Dialogue: 0,0:00:05.01,0:00:10.00,s22,,0,0,0,,{\fs-22}W{ -26.4}
Dialogue: 0,0:00:05.01,0:00:10.00,s22,,0,0,0,,{\fs-11}W{ -2.2}
Dialogue: 0,0:00:05.01,0:00:10.00,s22,,0,0,0,,W
Dialogue: 0,0:00:05.01,0:00:10.00,s22,,0,0,0,,{\fs+11}W{ -46.2}
Dialogue: 0,0:00:05.01,0:00:10.00,s22,,0,0,0,,{\fs+22}W{ -70.4}
Dialogue: 0,0:00:05.01,0:00:10.00,s22,,0,0,0,,{\fs+36}W{ 101.2}
Special Fonts
I uploaded or linked to the required font
System Information
Native CPU-Arch: x86_64
VSFilter System: Windows 7, Aegisub+VSFilter 32-bit binary
libass System: glibc-Linux with freetype 2.10.4, fontconfig 2.13.1
compatibility fonts