New VPS specific benchmark [Update: Download avail.]

New VPS specific benchmark [Update: Download avail.]

jsgjsg Member
edited May 22 in General

Hello all

I have had enough from what I felt to be a quite unsatisfactory situation wrt. benchmarks. One of my major worries that seemed to not be addressed by any widely available server benchmark is the fact that VPSs are somewhat special and "sensitive animals" because there are other users on a node and because providers and neighbours are (understandably) easily angered by a VPS all but blocking a node during benchmarking.

vpsbench changes that. Using a microsecond based timer it can afford to and does break up the benchmark tests into many small slices with small pauses in between so that a node is never blocked by it.

Another gripe I had was that connectivity benchmarks virtually always include host resolution and server setup on the other side. As that can be quite a large amount of time the results are in between cooked and questionable. vpsbench doesn't do that but starts measuring only once the session is set up and data are beginning to flow so it shows the real throughput.

Probably only rarely useful it also shows the "build up" which can be helpful both to get an impression of congestion and especially in saving bandwidth. I have done a lot of testing and found out that virtually all connections are within 1% or 2% of their top speed after 32 - 64 MB. vpsbench has a default of 64 MB but allows any size between about 10 MB and about 2 GB, depending on the target file size of course. This feature also allows to test against a 1GB file but using only 64 MB or whatever one chose.

Oh and you can have different "target sets", say, a mainly Europe one, a west coast one, a small and fast one, etc. Moreover you can run all the tests (in a very simple way, see below) or you can chose which ones to run.

Another point I often missed was useful and relevant info on a VPS. Things like how many cores, how much real memory, model and family of the CPU, flags (useful to identify "hidden" vCores), etc. Well, vpsbench provides all that.

Regarding disks vpsbench does the usual sequential read/write and the somewhat less common random read/write and I do mean "random" (evil grin). If vpsbench shows good random read/write result you can bet that your database will fly.

My final issue was that I'm not interested in integer or floating point performance but in areas that are important for a server which is moving lots of data around, string operations and increasingly crypto. So that's what vpsbench does plus it looks at single core and at multi core performance.

I'm willing to open source vpsbench and I also have binaries (+- 1MB size) made for and tested with linux and FreeBSD both i386 and x64. Right now I'm planning on putting vpsbench binaries on some server for download but if there is significant interest I'll happily provide sources too.

Here is an example output from a real benchmark done on a 2 vCore 512MB VPS in Bucarest (the command was "vpsb ntargets" with ntargets being a file with servers to test for connectivity).

Let me know what you think and if you are interested.

Machine: amd64, Arch.: amd64, Model: Intel(R) Xeon(R) CPU     L5630  @ 2.13GHz
OS, version: linux 4.10.5, Mem.: 479 MB
CPU - Cores: 2, Family/Model/Stepping: 6/44/2
Cache: 32K/32K L1d/L1i, 256K L2, 12M L3
Std. Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
          pse36 cflsh mmx fxsr sse sse2 ss htt pbe sse3 pclmulqdq dtes64 ds_cpl
          ssse3 cx16 xtpr pcid dca sse4_1 sse4_2 popcnt aes hypervisor
Ext. Flags: syscall nx pdpe1gb lm lahf_lm

--- proc/mem/performance test single core ---
64 rounds~ 1.00 GB ->  116.98 MB/s
--- proc/mem/performance test multi-core ---
4 times 64 rounds ~ 4.00 GB ->  312.24 MB/s
--- disk test ---
Sequential writing 216.60 MB/s
Random writing     37.11 MB/s
Sequential reading 904.19 MB/s
Random reading     711.21 MB/s
--- network test - target   100KB  1MB  10MB   -> 64 MB ---
http://speedtest.lon02.softlayer.com/downloads/test100.zip  OK,LON:
    6.0 Mb/s   14.2 Mb/s   43.9 Mb/s    -> 111.9 Mb/s
http://speedtest.mel01.softlayer.com/downloads/test100.zip  AU,MEL:
    747.0 Kb/s   1.7 Mb/s   5.3 Mb/s    -> 13.6 Mb/s
http://speedtest.che01.softlayer.com/downloads/test100.zip  IN,CHN:
    1.6 Mb/s   3.7 Mb/s   10.9 Mb/s    -> 28.5 Mb/s
http://speedtest.fra02.softlayer.com/downloads/test100.zip  DE,FRA:
    11.1 Mb/s   26.1 Mb/s   78.9 Mb/s    -> 189.7 Mb/s
http://speedtest.mil01.softlayer.com/downloads/test100.zip  IT,MIL:
    5.9 Mb/s   13.9 Mb/s   40.2 Mb/s    -> 106.1 Mb/s
http://speedtest.par01.softlayer.com/downloads/test100.zip  FR,PAR:
    7.0 Mb/s   16.5 Mb/s   47.7 Mb/s    -> 124.5 Mb/s
http://93.95.100.190/test100mb.bin  RUS,MOS:
    2.3 Mb/s   5.5 Mb/s   7.7 Mb/s    -> 6.0 Mb/s
http://speedtest.sao01.softlayer.com/downloads/test100.zip  BR,SAO:
    1.2 Mb/s   2.7 Mb/s   8.4 Mb/s    -> 22.0 Mb/s
http://speedtest.dal05.softlayer.com/downloads/test100.zip  US,DAL:
    1.8 Mb/s   4.4 Mb/s   13.5 Mb/s    -> 34.5 Mb/s
http://speedtest.sjc01.softlayer.com/downloads/test100.zip  US,SJC:
    1.5 Mb/s   3.4 Mb/s   11.0 Mb/s    -> 28.8 Mb/s
http://speedtest.wdc01.softlayer.com/downloads/test100.zip  US,WDC:
    2.3 Mb/s   5.2 Mb/s   16.8 Mb/s    -> 42.4 Mb/s
http://speedtest.tokyo.linode.com/100MB-tokyo.bin   JP,TOK:
    1.0 Mb/s   2.4 Mb/s   7.7 Mb/s    -> 18.8 Mb/s
http://lg-ro.vps2day.com/100MB.test RO,BUK:
    22.5 Mb/s   49.7 Mb/s   130.8 Mb/s    -> 273.6 Mb/s
http://speedtest.ftp.otenet.gr/files/test100Mb.db   GR,UNK:
    2.0 Mb/s   6.7 Mb/s   21.5 Mb/s    -> 61.3 Mb/s
http://speedtest.osl01.softlayer.com/downloads/test100.zip  NO,OSL:
    4.3 Mb/s   9.4 Mb/s   29.0 Mb/s    -> 75.9 Mb/s
Thanked by 4FHR MasonR Aidan Shot2

Comments

  • FHRFHR Member, Provider

    This looks pretty good!
    It's just download a binary and run it, no external dependencies needed, right?

    Affordable Semi-Dedicated VPS - Enjoy the performance to the fullest extent. | 40% OFF promo

    Thanked by 1Aidan
  • FlamesRunnerFlamesRunner Member
    edited May 21

    @jsh

    I'd rather compile it myself + inspect the code before running it. You're a rather new member to LET so you don't exactly have my trust yet.

    @FHR

    Ahem, my sig.

    Thanked by 1Aidan
  • FHRFHR Member, Provider

    FlamesRunner said: Ahem, my sig.

    You don't want to compile code on every system you want to benchmark. Installing build tools and dependencies leaves quite a mess.

    On the other hand, you are right that embedding / bundling malicious software is very easy and, like your script suggests, can have fatal consequences for the software of the VM.

    Affordable Semi-Dedicated VPS - Enjoy the performance to the fullest extent. | 40% OFF promo

    Thanked by 1FlamesRunner
  • FalzoFalzo Member

    this looks promising, though only softlayer for most of the download test I personally find less than ideal, as peerings might differ a lot with other providers.

    also sadly still lacking some checks for upload speed, which would really something new to the benchmark script world.

    keep up the good work and hopefully improve even further! also agree with @FlamesRunner, will only run it if I can read the source and compile me a binary myself. the less dependencies, the better ;-)

    UltraVPS.eu KVM in US/UK/NL/DE: 15% off first 6 month and cheap 750G / 2TB storage offers
    Netcup KVM: 2GB 40TB BW - 16,14€ 6m or 2 dedCore 6GB 320GB - 78,88€ 12m /w 5€ off: 36nc15279180197 | 36nc15292244387

  • @FHR

    You're blowing my cover :)

  • jsgjsg Member
    edited May 21

    @FHR said: This looks pretty good!
    It's just download a binary and run it, no external dependencies needed, right?

    Yes, that was one point I considered important. No dependencies, no matter the shell no libthis.so or libthat.so needed, no rpm or deb or other distro specifics, no installation. You just download/upload the vps binary plus your target list file, which I assume most will want to have a couple of and that's it.

    Also, maybe important but I forgot to mention it. There is a short but complete small manual.

    @FlamesRunner said: @jsh

    I'd rather compile it myself + inspect the code before running it. You're a rather new member to LET so you don't exactly have my trust yet.

    jsG please (not jsH). Well, your reason isn't the most flattering one but I stand by what I said. If there's enough interest I will provide the sources, too. Freepascal and C btw.

    @Falzo said: this looks promising, though only softlayer for most of the download test ...

    Sorry for my bad wording. The target file is made by YOU, the user. It's just that I happen to know quite some softlayer test servers/files so I used them in my example test but you can put any provider/server you like.

    Thanked by 2FHR Falzo
  • @jsg

    Apologies. Can't fix it as I cannot edit my post.

  • jsgjsg Member

    OK, done. vpsbench (incl source) can now be downloaded via yandex disk. -> https://yadi.sk/d/GS1rPud83WKtWs

  • Why not Github?

    hi, I was wondering how much dedotated wam I need to the server.

  • jsgjsg Member

    @dedotatedwam said: Why not Github?

    Because I don't like it and because there is no need. This is not a group project.

    I've made it publicly available so everyone can download it and those who like to do so can play with the source. I have shared my work so that others can use my benchmark too, not to meta-discuss pro or anti this or that way of sharing my code.

  • jsgjsg Member

    While testing vpsbench I also looked again at an old AMD E350 System: dual core, incl. some graphics engine, quite low power consumption. I thought it might be interesting for some here to see the vpsbench results of that thingy. And they are not at all lousy and in fact at about the level of many VPS vCores.

    Machine: amd64, Arch.: x86_64, Model: AMD E-350 Processor
    OS, version: Linux 4.10.5, Mem.: 3.738 GB
    CPU - Cores: 2, Family/Model/Stepping: 20/1/0
    Cache: 32K/32K L1d/L1i, 512K L2, ? L3
    Std. Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
              pse36 cflsh mmx fxsr sse sse2 htt sse3 monitor ssse3 cx16 popcnt
    Ext. Flags: syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm lahf_lm cmp_legacy svm
              extapic cr8_legacy lzcnt sse4a misalignsse 3dnowprefetch ibs skinit
              wdt
    
    --- proc/mem/performance test single core ---
    ................................................................
    64 rounds~ 1.00 GB ->  117.96 MB/s
    --- proc/mem/performance test multi-core ---
    ................
    4 times 64 rounds ~ 4.00 GB ->  229.27 MB/s
    
  • jsgjsg Member
    edited August 9

    I just came across some benchmark called "nensh.sh" and thought I'd have a somewhat deeper look at it. One major reason for my interest were results that looked quite strange.

    My first grief is that it needs bash which is (a lousy and fat shell and) by far not everywhere installed or even available.

    The next minus is that it uses the 'time' command which is quite crude and more importantly those tests are run in 1 chunk. On a VPS that's (a) impolite and (b) quite worthless because you certainly get some value but its meaning is in between vague and ridiculous.

    Finally let's have a look at one CPU benchmark: AES

    nench uses openssl for that ( openssl enc -e -aes-256-cbc -pass pass:12345678 to be precise). So let's look at that: what exactly does this test do? To find out more I called the command with different 'bs' and 'count' values but always 500 MB in total. And low and behold the times were relatively consistent plus, and that's the really interesting part user time was (a) roughly about 50% of real time and (b) consistently considerably less than sys time plus the user time reported corresponded MUCH more to AES's real performance on the given system.

    Another bad habit is dd. timing dd - especially with relatively small sample sizes like 500 MB (which is well within the reach of today OS caches) and with /dev/null as source - may provide you some hints (like how good your system is caching reads and writes) but it does NOT tell you anything meaningful about the drives of the system under test.

    In other words nench (and most probably the other bash benchmarks too) tells you a whole lot which you didn't ask for and which many users will not even be aware of/understand - one thing however it reliably does NOT: it doesn't tell you what it promised to tell you.

    In short, those scripts can give you a VERY vage 1st impression of a system but they are worthless for reliably judging a system/VPS.

    I didn't write vpsbench for the fun of it. I wrote it because I had enough of toys and wanted an actually useful tool.

  • FHRFHR Member, Provider

    jsg said: My first grief is that it needs bash which is (a lousy and fat shell and) by far not everywhere installed or even available.

    bash is available in all VPS templates and all standard installations of normal server OSes.

    Affordable Semi-Dedicated VPS - Enjoy the performance to the fullest extent. | 40% OFF promo

    Thanked by 1vimalware
  • jsgjsg Member

    @FHR said:

    jsg said: My first grief is that it needs bash which is (a lousy and fat shell and) by far not everywhere installed or even available.

    bash is available in all VPS templates and all standard installations of normal server OSes.

    Linux != "[all] normal server OSes".

    Vpsbench needs NO installation at all. No files spread all over diverse directories, no fuzz, just uploading the binary no matter where to and maybe deleting it after use if one likes that.

    I mentioned that point for two reasons: (a) those benchmarks scripts are supposed to be simple and easy. Down/upload it run it, done (just like Vpsbench) but needing to install bash kills that advantage. sh on the other hand is indeed supposed to be available everywhere (but less powerful/luxurious). And (b) it's just one detail but it was one of the factors that made me want a real no fuzz, just download one file/binary and go.

    Btw. my comment was NOT meant to smear nench.

  • Shot2Shot2 Member
    edited August 9

    What I miss is:

    • a simpler overview of CPU flags, considering how cryptic and unsorted the output is (I'd rather have something like: AES:√ RNG:× VT:× SSE:√(4.2) )

    • some "cpu score" arbitrary and real-life rating, for quick comparison purposes... I don't know, "how many times some stable pattern can be deflate-compressed in 5 seconds...

    • output clearly some info about the context: at least hostname (without the domain part, privacy rulez), date/time of the test, version of the benchmark program... so I can archive the results

    Still, I appreciate the reasoning behind the bench tool. Well done.

    edit: I hate formatting, formatting hates me

    Providing less than /64 means "we are clueless about IPv6".

  • jsgjsg Member

    @Shot2 said: What I miss is:

    • a simpler overview of CPU flags, considering how cryptic and unsorted the output is (I'd rather have something like: AES:√ RNG:× VT:× SSE:√(4.2) )

    • some "cpu score" arbitrary and real-life rating, for quick comparison purposes... I don't know, "how many times some stable pattern can be deflate-compressed in 5 seconds...

    • output clearly some info about the context: at least hostname (without the domain part, privacy rulez), date/time of the test, version of the benchmark program... so I can archive the results

    Still, I appreciate the reasoning behind the bench tool. Well done.

    edit: I hate formatting, formatting hates me

    Thanks for the feedback.

    Re. your first point the truth is that I was (and am) too lazy. My thinking was that most don't really look at the flags anyway and those who do typically look for a specific flag (like AES) using grep or whatever.

    Re. your second point the problem is that there is no real (and reasonable) standard. What I did though was to design the CPU/mem test in such a way as to test a typical server workload. And those numbers can be compared and make sense.

    Re. the last point I actually did think a bit about that but decided against it as I'm a privacy fanatic and was always annoyed that many benchmarks show e.g. a servers IP. But those who really want that can easily choose one of diverse available routes to add that kind of information to the final output.

  • Shot2Shot2 Member

    Point 2: makes sense. 1 and 3: agreed. I hate it too when benchmark results include personal stuff :) Anyway, with the source, I might hack sth to fit my needs. Much obliged!

    Providing less than /64 means "we are clueless about IPv6".

  • jsgjsg Member

    @Shot2 said: Point 2: makes sense. 1 and 3: agreed. I hate it too when benchmark results include personal stuff :) Anyway, with the source, I might hack sth to fit my needs. Much obliged!

    You are welcome.

    Regarding your desire to have something like a list of flags with some "Yes" or "No" marker one simple way to get that would be to have a array with a structure with all possible flags and a (default "No") marker and then walking through the list of found flags that is already in the code and to set those to "Yes".

    Feel free to PM me if you have a question.

  • I try to compile it from source. Get this error:

    Free Pascal Compiler version 3.0.0+dfsg-2 [2016/01/28] for x86_64 Copyright (c) 1993-2015 by Florian Klaempfl and others (1002) Target OS: Linux for x86-64 (3104) Compiling vpsb.pas 2 199/736 Kb Used (3104) Compiling diskb.pas 100 21.631/24.096 Kb Used 200 22.299/24.736 Kb Used (3104) Compiling procmb.pas /root/compile/vpsbench/procmb.pas(90,25) Error: (5038) identifier idents no member "ExecuteInThread" 100 21.532/24.736 Kb Used procmb.pas(113) Fatal: (10026) There were 1 errors compiling module, stopping Fatal: (1018) Compilation aborted

    @jsg Any idea?

    Thank you.

  • jsgjsg Member
    edited August 12

    @try4lontalk

    Sorry, I can't reproduce that error. Looking at what you provide I note that you run a pretty old fpc version. I compiled with 3.0.4+dfsg-20 from about May 2018. Plus you might be missing some fpc packages like pthreads or rtl-extra. Anyway the error sounds like you either have no TThreads unit/module or you have an old one without ExecuteInThread.

  • jsgjsg Member
    edited August 12

    @Shot2 said: I don't know, "how many times some stable pattern can be deflate-compressed in 5 seconds...

    Good point! Sorry, I had it in mind when I responded but then forgot to write it.

    You probably confused nench and Vpsbench. My benchmark does not do any funny [b|g]zip routines. In fact, vpsbenchmark is expressely designed to avoid issues like (unwanted) caching, pattern repetition, etc. It uses in fact some quite good quality random generator (xoroshift128) both in the disk and in the processor/memory tests to trash caches. This is also a major reason for my disk benchmark being by far more reliable and providing better results than dd and friends (which are tainted by OS caching and easily tricked, especially when if=/dev/zero). And that was another major reason why I wanted and designed & created my own benchmarking tool vpsbench.

    You'll also note that unlike many benchmarks I expressly tried to avoid mixing up thing like involving the file system when testing CPU performance (unlike using bzip2 to test processor performance and involving the OS file system, even if only virtual).

Sign In or Register to comment.