feat: Allow running parallel transfer benchmarks #1928
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
By randomizing the port used for a bench. There is still a small chance for a collision, but seems acceptable.
Failed Interop TestsQUIC Interop Runner, client vs. server
All resultsSucceeded Interop TestsQUIC Interop Runner, client vs. server
Unsupported Interop TestsQUIC Interop Runner, client vs. server
|
larseggert
commented
Jun 17, 2024
| @@ -20,7 +20,8 @@ fn transfer(c: &mut Criterion) { | |||
| neqo_common::log::init(Some(log::LevelFilter::Off)); | |||
| neqo_crypto::init_db(PathBuf::from_str("../test-fixture/db").unwrap()).unwrap(); | |||
|
|
|||
| let done_sender = spawn_server(); | |||
| let port = 49151 + (rand::random::<u16>() % 16384); | |||
There was a problem hiding this comment.
Should we use neqo-crypto here? It doesn't seem to have a clean way to return randomized types, just u8?
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1928 +/- ##
=======================================
Coverage 94.82% 94.82%
=======================================
Files 110 110
Lines 35792 35792
=======================================
Hits 33938 33938
Misses 1854 1854 ☔ View full report in Codecov by Sentry. |
Benchmark resultsPerformance differences relative to 6664452. coalesce_acked_from_zero 1+1 entries: No change in performance detected. time: [196.78 ns 197.51 ns 198.56 ns]
change: [-0.2088% +0.4191% +1.3858%] (p = 0.36 > 0.05)
Found 17 outliers among 100 measurements (17.00%)
7 (7.00%) high mild
10 (10.00%) high severecoalesce_acked_from_zero 3+1 entries: No change in performance detected. time: [238.92 ns 239.73 ns 240.54 ns]
change: [-0.3174% +0.1125% +0.5175%] (p = 0.59 > 0.05)
Found 18 outliers among 100 measurements (18.00%)
12 (12.00%) high mild
6 (6.00%) high severecoalesce_acked_from_zero 10+1 entries: Change within noise threshold. time: [237.42 ns 238.41 ns 239.52 ns]
change: [+0.1697% +0.6985% +1.3026%] (p = 0.01 < 0.05)
Found 10 outliers among 100 measurements (10.00%)
10 (10.00%) high severecoalesce_acked_from_zero 1000+1 entries: No change in performance detected. time: [216.15 ns 216.30 ns 216.47 ns]
change: [-4.8684% -1.5903% +0.4282%] (p = 0.38 > 0.05)
Found 13 outliers among 100 measurements (13.00%)
5 (5.00%) high mild
8 (8.00%) high severeRxStreamOrderer::inbound_frame(): Change within noise threshold. time: [119.57 ms 119.73 ms 119.98 ms]
change: [+0.3022% +0.4625% +0.6768%] (p = 0.00 < 0.05)
Found 4 outliers among 100 measurements (4.00%)
3 (3.00%) high mild
1 (1.00%) high severetransfer/Run multiple transfers with varying seeds: No change in performance detected. time: [119.82 ms 120.09 ms 120.36 ms]
thrpt: [33.233 MiB/s 33.308 MiB/s 33.384 MiB/s]
change:
time: [-0.2439% +0.0963% +0.4498%] (p = 0.58 > 0.05)
thrpt: [-0.4478% -0.0962% +0.2445%]
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) low mildtransfer/Run multiple transfers with the same seed: Change within noise threshold. time: [120.53 ms 120.72 ms 120.92 ms]
thrpt: [33.080 MiB/s 33.134 MiB/s 33.186 MiB/s]
change:
time: [+0.1633% +0.3680% +0.5772%] (p = 0.00 < 0.05)
thrpt: [-0.5739% -0.3667% -0.1630%]
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high severe1-conn/1-100mb-resp (aka. Download)/client: No change in performance detected. time: [1.1028 s 1.1319 s 1.1596 s]
thrpt: [86.236 MiB/s 88.344 MiB/s 90.682 MiB/s]
change:
time: [-4.0277% -0.6115% +2.4658%] (p = 0.73 > 0.05)
thrpt: [-2.4065% +0.6152% +4.1967%]
1-conn/1-1b-resp (aka. HPS)/client: No change in performance detected. time: [41.766 ms 41.897 ms 42.047 ms]
thrpt: [23.783 elem/s 23.868 elem/s 23.943 elem/s]
change:
time: [-0.7282% -0.2213% +0.2747%] (p = 0.39 > 0.05)
thrpt: [-0.2740% +0.2217% +0.7335%]
Found 32 outliers among 100 measurements (32.00%)
20 (20.00%) low severe
2 (2.00%) low mild
1 (1.00%) high mild
9 (9.00%) high severeClient/server transfer resultsTransfer of 33554432 bytes over loopback.
|
|
Closing this. It was useful while debugging a spurious failure, but it's often more/also useful to have predictable ports for tcpdumping. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
By randomizing the port used for a bench. There is still a small chance for a collision, but seems acceptable.