Script for reproducing an existing stack for a new CPU target#163
Script for reproducing an existing stack for a new CPU target#163bedroge merged 19 commits intoEESSI:mainfrom
Conversation
…at it now creates files in an actual easystack format and that it only does one write per file instead of one write per easyconfig
…dir from the reproduction dir
…systacks produced matches the original build order
…es that the easyconfig and easyblocks are used as they are specified in the easystack files - which is typically not the case if something is build as a dependency for something else
|
Just to leave it as a note here as well: in EESSI/software-layer#1412 (comment) we found that poetry was rebuilt with an additional dependency, and due to the setup of our reprod script this is not being handled well. We retain the original build order, but always build with the last used easyconfig/easyblock/EB version. In this particular case it means poetry couldn't be built, because that additional dependency ( It would be really tricky to automatically get that right. As an alternative we could simply process rebuilds afterwards, in the same way as it was originally done. That does involve more work though, which is the main reason why we had decided to process rebuilds immediately (i.e. build with the latest version). We could reconsider that, though this probably doesn't happen very often, and it can be easily fixed manually by moving some stuff around in the generated easystacks. |
|
A very similar but slighly more complicated issue popped up for gst-plugins-bad in EESSI/software-layer#1440. Again, it had to be solved by moving additional dependencies around in the easystacks, which was done in EESSI/software-layer#1443. As this one was more complex, I'm beginning to lean towards changing the approach and handle rebuilds at the end. But, for now, I think we should go ahead and merge this, as it has been used quite successfully for zen5. Issues that we encountered are listed in EESSI/software-layer#1430. There are certainly things that can be improved in the script, but we can do that in follow-up PRs. |
…rrently have with rebuilds
This script is the result of joint development with @bedroge on implementing what was discussed in https://gitlab.com/eessi/support/-/issues/233
In summary: it generates easystack files that will allow you to replicate the software stack installed for a reference architecture. We will use this in practice to deploy software for
zen5, which will also serve as a test for these scripts.Known limitations (see EESSI/software-layer#1430): doing the rebuild at the time of the original build (as will be done when generating easystacks with this script) will fail if the rebuild added new dependencies. To resolve this, we should do an iterative improvement of this script in which we replay rebuilds as well. This takes slightly more build time, but avoids these issues, thus resulting in a more seamless experience in reproducing the stack for a new target. This is listed as a TODO in EESSI/software-layer#1430 (comment)