Skip to content

Out of sample testing#5

Open
ArneTR wants to merge 1 commit intofixesfrom
out-of-sample-testing
Open

Out of sample testing#5
ArneTR wants to merge 1 commit intofixesfrom
out-of-sample-testing

Conversation

@ArneTR
Copy link
Collaborator

@ArneTR ArneTR commented Mar 18, 2026

This PR builds upon the bin approach in #4

It adds the --test-data param that points to a testing only energy logger data file.

This allows proper out of sample testing.

First results show this for non same workloads (idle vs benchmark.sh):

python3 model.py /tmp/energy-7hyUsZ9f.log --mode delta --alpha 10 --trim=1.0 --test-data /tmp/energy-3G05W1dt.log
# mode=delta alpha=10.0
# rows_train=1326 rows_test=418

# --- test metrics (current module defaults) ---
R2     : -15.25236
RMSE   : 946323.788 uJ
MAE    : 916790.084 uJ
MAPE   : 99.927 %

# --- test metrics (linear non-negative ridge) ---
R2     : -12.23006
RMSE   : 853812.158 uJ
MAE    : 830840.973 uJ
MAPE   : 100.021 %

# --- test metrics (nonlinear reference) ---
R2     : -0.43362
RMSE   : 281059.228 uJ
MAE    : 254916.150 uJ
MAPE   : 29.056 %

# --- test metrics (distilled kernel model) ---
R2     : -31.53800
RMSE   : 1338988.676 uJ
MAE    : 795537.625 uJ
MAPE   : 94.943 %

# --- train metrics (distilled kernel model) ---
R2     : -575.81784
RMSE   : 44334482.657 uJ
MAE    : 2338971.645 uJ
MAPE   : 79.459 %

For repeated runs of benchmark.sh it is this:

arne@framebook:~/Sites/procpower/tools$ python3 model.py /tmp/energy-7hyUsZ9f.log --mode delta --alpha 10 --trim=1.0 --test-data /tmp/energy-gKpkxu81.log
# mode=delta alpha=10.0
# rows_train=1326 rows_test=1246

# --- test metrics (current module defaults) ---
R2     : -2.88214
RMSE   : 3388222.120 uJ
MAE    : 2922949.824 uJ
MAPE   : 99.826 %

# --- test metrics (linear non-negative ridge) ---
R2     : 0.86565
RMSE   : 630306.004 uJ
MAE    : 576792.573 uJ
MAPE   : 32.694 %

# --- test metrics (nonlinear reference) ---
R2     : 0.96917
RMSE   : 301920.783 uJ
MAE    : 261693.641 uJ
MAPE   : 12.957 %

# --- test metrics (distilled kernel model) ---
R2     : 0.20774
RMSE   : 1530631.787 uJ
MAE    : 625044.750 uJ
MAPE   : 33.205 %

# --- train metrics (distilled kernel model) ---
R2     : -575.81784
RMSE   : 44334482.657 uJ
MAE    : 2338971.645 uJ
MAPE   : 79.459 %

@ArneTR ArneTR changed the base branch from main to fixes March 18, 2026 12:33
@ArneTR ArneTR mentioned this pull request Mar 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant