Skip to content

CORS-4336: Add CI jobs for AWS European Sovereign Cloud (EUSC)#75568

Open
liweinan wants to merge 16 commits intoopenshift:mainfrom
liweinan:add-aws-eusc-ci-jobs
Open

CORS-4336: Add CI jobs for AWS European Sovereign Cloud (EUSC)#75568
liweinan wants to merge 16 commits intoopenshift:mainfrom
liweinan:add-aws-eusc-ci-jobs

Conversation

@liweinan
Copy link
Contributor

@liweinan liweinan commented Mar 2, 2026

Implement continuous integration support for AWS EUSC partition (aws-eusc) in eusc-de-east-1 region. Includes cluster profile definition, service endpoints configuration, custom AMI handling, and periodic test jobs.

This enables OpenShift testing on AWS's new European Sovereign Cloud infrastructure, which requires explicit endpoint configuration and custom RHCOS AMIs not available in public regions.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 2, 2026
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 2, 2026 •
edited by openshift-ci bot
Loading

@liweinan: This pull request references CORS-4336 which is a valid jira issue.

Details

In response to this:

Implement continuous integration support for AWS EUSC partition (aws-eusc) in eusc-de-east-1 region. Includes cluster profile definition, service endpoints configuration, custom AMI handling, and periodic test jobs.

This enables OpenShift testing on AWS's new European Sovereign Cloud infrastructure, which requires explicit endpoint configuration and custom RHCOS AMIs not available in public regions.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested review from neisw and xingxingxia March 2, 2026 18:05
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 2, 2026

@liweinan, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: configuration has 2 errors:

 * tests[125]: invalid cluster profile "aws-eusc-qe"
 * tests[126]: invalid cluster profile "aws-eusc-qe"
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@yunjiang29
Copy link
Contributor

yunjiang29 commented Mar 5, 2026

@liweinan as we discussed offline, for the new partition we need three types of cluster:

  1. common IPI cluster
  2. private cluster
  3. disconnected (private) cluster
    and based on above basic cluster, we also need to cover STS, custom KMS key, FIPS and minimum permission, you can refer to existing jobs.

@liweinan
Copy link
Contributor Author

liweinan commented Mar 5, 2026

@yunjiang29 Thanks for the review! I'll refactor this PR today.

@liweinan liweinan force-pushed the add-aws-eusc-ci-jobs branch from 24fed80 to de00d69 Compare March 5, 2026 12:16
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 5, 2026

@liweinan, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: configuration has 2 errors:

 * tests[126]: invalid cluster profile "aws-eusc"
 * tests[127]: invalid cluster profile "aws-eusc"
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@liweinan
Copy link
Contributor Author

liweinan commented Mar 6, 2026

@yunjiang29 Thanks for the detailed review! I'll update the PR recordingly.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 6, 2026

@liweinan, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: configuration has 2 errors:

 * tests[126]: invalid cluster profile "aws-eusc"
 * tests[127]: invalid cluster profile "aws-eusc"
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

liweinan added a commit to liweinan/release that referenced this pull request Mar 6, 2026
@liweinan
Refactor AWS EUSC CI jobs based on review feedback
7f83d83 
Address yunfei's review comments on PR openshift#75568:

1. Job naming convention:
   - Rename jobs from -f60 to -f7 suffix (non-destructive tests)
   - Update cron schedule to standard f7 pattern: 7,14,23,30

2. Private cluster configuration:
   - Add complete private cluster setup with bastion host
   - Add VPC, security groups, and proxy configuration
   - Set PUBLISH=Internal for private cluster access
   - Add minimal IAM permission provisioning
   - Follow pattern from cucushift-installer-rehearse-aws-ipi-private-provision

3. AMI configuration fix:
   - Replace deprecated compute.platform.aws.amiID field
   - Use platform.aws.defaultMachinePlatform.amiID instead
@liweinan liweinan force-pushed the add-aws-eusc-ci-jobs branch from 4b73bfe to 7f83d83 Compare March 6, 2026 06:38
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 6, 2026

@liweinan, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: configuration has 2 errors:

 * tests[126]: invalid cluster profile "aws-eusc"
 * tests[127]: invalid cluster profile "aws-eusc"
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

liweinan added a commit to liweinan/release that referenced this pull request Mar 6, 2026
@liweinan
Address yunfei's review comments on PR openshift#75568:
55daf83 
1. Job naming convention:
   - Rename jobs from -f60 to -f7 suffix (non-destructive tests)
   - Update cron schedule to standard f7 pattern: 7,14,23,30

2. Private cluster configuration:
   - Add complete private cluster setup with bastion host
   - Add VPC, security groups, and proxy configuration
   - Set PUBLISH=Internal for private cluster access
   - Add minimal IAM permission provisioning
   - Follow pattern from cucushift-installer-rehearse-aws-ipi-private-provision

3. AMI configuration fix:
   - Replace deprecated compute.platform.aws.amiID field
   - Use platform.aws.defaultMachinePlatform.amiID instead

4. Generalize step registry components for reusability:
   - Enhance ipi-conf-aws-custom-endpoints to support multiple AWS partitions
     * Add AWS_DOMAIN_SUFFIX env var (defaults to amazonaws.com)
     * Support amazonaws.eu for EUSC, amazonaws.com.cn for China
     * Allow full URLs for maximum flexibility
   - Make ipi-conf-aws-eusc-ami more generic
     * Support AWS_CUSTOM_AMI_ID for general use
     * Maintain AWS_EUSC_AMI_ID for backward compatibility
     * Can be used for EUSC, China, GovCloud, or custom AMI scenarios
   - Use generic steps in EUSC provision chain with partition-specific config
   - Remove obsolete ipi-conf-aws-eusc-endpoints (replaced by generic version)
@liweinan liweinan force-pushed the add-aws-eusc-ci-jobs branch from 7f83d83 to 55daf83 Compare March 6, 2026 06:58
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 6, 2026

@liweinan, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: configuration has 2 errors:

 * tests[126]: invalid cluster profile "aws-eusc"
 * tests[127]: invalid cluster profile "aws-eusc"
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

liweinan added a commit to liweinan/release that referenced this pull request Mar 6, 2026
@liweinan
Address yunfei's review comments on PR openshift#75568:
c6c4827 
1. Job naming convention:
   - Rename jobs from -f60 to -f7 suffix (non-destructive tests)
   - Update cron schedule to standard f7 pattern: 7,14,23,30

2. Private cluster configuration:
   - Add complete private cluster setup with bastion host
   - Add VPC, security groups, and proxy configuration
   - Set PUBLISH=Internal for private cluster access
   - Add minimal IAM permission provisioning
   - Follow pattern from cucushift-installer-rehearse-aws-ipi-private-provision

3. Generalize step registry components for maximum reusability:

   a) Enhance ipi-conf-aws-custom-endpoints for all AWS partitions:
      - Add AWS_DOMAIN_SUFFIX env var (defaults to amazonaws.com)
      - Support amazonaws.eu (EUSC), amazonaws.com.cn (China)
      - Allow full URLs for maximum flexibility
      - Remove obsolete ipi-conf-aws-eusc-endpoints step

   b) Extend ipi-conf-aws to support custom AMI configuration:
      - Add AWS_AMI_ID env var for custom RHCOS AMI
      - Useful for EUSC, China, GovCloud, or any partition without public AMIs
      - Fix deprecated amiID field -> defaultMachinePlatform.amiID
      - Auto-detection still works for C2S/SC2S
      - Remove obsolete ipi-conf-aws-eusc-ami step

   c) EUSC provision chain now uses only generic steps with env config

This refactoring reduces code duplication (net -59 lines) and makes step
components reusable across all AWS partitions.
@liweinan liweinan force-pushed the add-aws-eusc-ci-jobs branch from 55daf83 to c6c4827 Compare March 6, 2026 07:10
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 6, 2026

@liweinan, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: configuration has 2 errors:

 * tests[126]: invalid cluster profile "aws-eusc"
 * tests[127]: invalid cluster profile "aws-eusc"
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@liweinan
Copy link
Contributor Author

liweinan commented Mar 10, 2026

Relative PRs merged: #75441 / openshift/ci-tools#4973

@yunjiang29
Copy link
Contributor

yunjiang29 commented Mar 23, 2026

@tthvo's initial EUSC PR has merged, so we might actually be able to rehearse this, if credentials are setup:

I guess the credentials are not yet ready? The installer is asking for the access key and secrets in the ci/rehearse/openshift/installer/main/e2e-aws-eusc-techpreview.

level=warning msg=Found override for release image (registry.build10.ci.openshift.org/ci-op-n8lpz8gb/release@sha256:02c954c753c39c185094a8253214485c9dcd4aa17202bf680bb158d06db15032). Release Image Architecture is unknown
? AWS Access Key ID [? for help] �[?25l�[?25l�7�8�[?25h�[?25h? AWS Access Key ID [? for help] �[?25l�[?25l�7�8�[?25h�[?25h? AWS Access Key ID [? for help] �[?25l�[?25l�7�8�[?25h�[?25hlevel=error msg=failed to fetch Master Machines: failed to load asset "Install Config": failed to create install config: [platform.aws.region: Internal error: failed to get list of regions: failed to create EC2 client: EOF, controlPlane.platform.aws: Internal error: failed to create EC2 client: EOF, compute[0].platform.aws: Internal error: failed to create EC2 client: EOF]
Create manifests exit code: 3

@yunjiang29 Do you have any idea on this?

We can see Unsupported cluster type 'aws-eusc' in the build log, aws-eusc is new cluster type, but install steps can't recognize it, it needs to be added to

release/ci-operator/step-registry/ipi/install/install/aws/ipi-install-install-aws-commands.sh

Line 233 in a8b5667

aws|aws-arm64|aws-usgov)
and

release/ci-operator/step-registry/ipi/install/install/ipi-install-install-commands.sh

Line 621 in a8b5667

aws|aws-arm64|aws-usgov)

@openshift-ci openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 23, 2026
@yunjiang29
Copy link
Contributor

yunjiang29 commented Mar 23, 2026

@tthvo @patrickdillon btw, this PR is not ready for testing because it hasn't configured AMI_ID yet.

ami-0c5e051fb7dc39f8d is ready for test, you can config it in job config via:

CONTROL_PLANE_AMI: ami-0c5e051fb7dc39f8d
COMPUTE_AMI: ami-0c5e051fb7dc39f8d

@liweinan
Copy link
Contributor Author

liweinan commented Mar 23, 2026 

ami-0b78302f83217d149

After discussing with Yunfei locally, we decided to use the AMI noted here: #75568 (comment)

liweinan added 15 commits March 23, 2026 18:29
@liweinan
Add AWS European Sovereign Cloud (EUSC) CI support
c95cf5d 
Implement comprehensive CI infrastructure for AWS EUSC partition in eusc-de-east-1 region.

Job coverage (9 jobs):
- Common IPI: aws-eusc-ipi-f7, aws-eusc-ipi-f28-destructive, aws-eusc-ipi-fips-f7
- Private: aws-eusc-ipi-private-f7, aws-eusc-ipi-private-f28-destructive, aws-eusc-ipi-private-fips-f7
- Disconnected: aws-eusc-ipi-disconnected-private-f7
- STS: aws-eusc-ipi-sts-f7
- KMS: aws-eusc-ipi-byo-kms-f7

Key features:
- Dynamic service endpoint auto-detection from AWS API
- Split AMI variables (CONTROL_PLANE_AMI, COMPUTE_AMI) for flexible configuration
- Complete private cluster deprovision cleanup (bastion, security groups, stacks, S3)
- Support for FIPS-enabled clusters
- Disconnected (air-gapped) private cluster support
- STS (Security Token Service) authentication with OIDC
- Custom KMS key encryption for etcd
- Both non-destructive (f7) and destructive (f28) test variants

Technical implementation:
- Cluster profile: aws-eusc with automatic region detection
- Custom RHCOS AMI support for control plane and compute nodes separately
- Endpoint auto-detection from AWS API (no hardcoded values)
- Manual credentials mode for CCO
- Minimal IAM permissions
- Mirror registry for disconnected environments
- Backward compatible with existing AWS partitions

Workflows created:
- cucushift-installer-rehearse-aws-eusc-ipi (common IPI)
- cucushift-installer-rehearse-aws-eusc-ipi-private (private cluster)
- cucushift-installer-rehearse-aws-eusc-ipi-disconnected-private (disconnected)
- cucushift-installer-rehearse-aws-eusc-ipi-sts (STS authentication)
- cucushift-installer-rehearse-aws-eusc-ipi-byo-kms (custom KMS key)

Signed-off-by: Wei Li <weli@redhat.com>
@liweinan
Refactor AWS EUSC CI jobs based on review feedback
76bf602 
This refactors the AWS European Sovereign Cloud (EUSC) CI configuration
to maximize reuse of standard AWS workflows and reduce maintenance burden.

Changes based on @yunjiang29's review feedback:

- Reduced from 9 to 6 jobs following the pattern: 3 cluster types × 2 test types
- Improved FIPS coverage from 1/9 (11%) to 2/6 (33%) jobs:
  * aws-eusc-ipi-fips-f7 (IPI + FIPS)
  * aws-eusc-ipi-private-sts-fips-f7 (Private + STS + FIPS)
- Combined features across jobs:
  * aws-eusc-ipi-f28-destructive (destructive testing)
  * aws-eusc-ipi-private-mini-perm-f28 (Private + minimal permissions)
  * aws-eusc-ipi-disc-priv-kms-f7 (Disconnected + KMS)
  * aws-eusc-ipi-disc-priv-f28 (Disconnected destructive)
- All jobs cover: FIPS, STS, KMS, minimal permissions across 3 cluster types

- Deleted 15 EUSC-specific files, created 8 new ones (net reduction: -7 files)
- Maximized reuse of standard AWS workflows:
  * Basic IPI: reuses cucushift-installer-rehearse-aws-ipi-deprovision
  * Private: reuses cucushift-installer-rehearse-aws-ipi-private-deprovision
  * Disconnected: reuses cucushift-installer-rehearse-aws-ipi-disconnected-private-provision
  * Private-STS: reuses cucushift-installer-rehearse-aws-ipi-private-cco-manual-security-token-service
- EUSC-specific changes limited to:
  * Inserting ipi-conf-aws-custom-endpoints ref for service endpoint configuration
  * Custom provision chain for disconnected-private-kms (combines disconnected + KMS)
- Deleted all EUSC-specific deprovision chains (reuse standard chains)
- Removed unnecessary byo-kms and STS specific directory structures

1. Custom endpoints (ipi-conf-aws-custom-endpoints-commands.sh):
   - Removed auto-detection logic for AWS_DOMAIN_SUFFIX
   - Simplified to use environment variable or default to "amazonaws.com"
   - Removed Route53 endpoint configuration (global service)
   - Designed for easy removal when installer adds native EUSC support

2. AMI configuration (ipi-conf-aws-commands.sh):
   - Simplified from split variables (CONTROL_PLANE_AMI/COMPUTE_AMI) to single CONTROL_PLANE_AMI
   - Preserved C2S/SC2S auto-detection logic
   - Removed complex heredoc patching, kept simple approach
   - Updated documentation for clarity

1. **Minimize EUSC-specific code**: Only 8 workflow files vs 15 previously
2. **Maximize standard workflow reuse**: Follows USGov pattern, not C2S pattern
3. **Prepare for future evolution**: Custom endpoints easy to remove when installer supports EUSC natively
4. **FIPS coverage aligned with USGov**: 33% vs USGov's 18%, not C2S's 100%

- make update completed successfully
- All 6 jobs generated in ci-operator/jobs/.../periodics.yaml
- Step registry validation passed

Addresses: openshift#75568
@liweinan
Simplify AWS custom endpoints and AMI configuration logic
301b04a 
Address review feedback to simplify configuration scripts:
- Remove AWS_DOMAIN_SUFFIX from step parameters (use cluster profile)
- Support separate CONTROL_PLANE_AMI and COMPUTE_AMI configuration
- Replace yq-go with yq v4 for YAML manipulation
- Eliminate unnecessary fallback logic, rely on correct parameter passing
- Remove intermediate variables (RHCOS_AMI) in C2S auto-detection

These changes follow existing script patterns and maintain compatibility
with C2S/SC2S auto-detection while enabling flexible AMI configuration
for partitions like EUSC.
@liweinan
Add missing OWNERS files for AWS EUSC step-registry directories
061dc24 
Fixes CI check error that enforces OWNERS files for all component
configuration directories.
@liweinan
Add FEATURE_SET=TechPreviewNoUpgrade to EUSC CI jobs
ad28f5d
@liweinan
Update AWS EUSC CI jobs with correct domain and add multi-arch support
905d49e 
- Update BASE_DOMAIN from qe.devcluster.openshift.com to ci-eusc.devcluster.openshift.com
  for all AWS EUSC CI jobs to use the dedicated delegated subdomain for CI/QE account

- Add 8 multi-arch EUSC CI jobs in openshift-tests-private release-4.22 multi-nightly:
  * BYO KMS encryption with FIPS (ARM f7, AMD f28-destructive)
  * Disconnected private (ARM f7, AMD f28-destructive)
  * Private STS (ARM f7, AMD f28-destructive)
  * Custom DNS with minimal permissions (ARM f7, AMD f28-destructive)

- Add e2e-aws-eusc-techpreview jobs to openshift/installer configs:
  * release-4.22, release-4.23, release-5.0, and main

- Add installer repo to aws-eusc cluster profile owners

- Restore version info comments in ipi-conf-aws-commands.sh

All jobs use cluster_profile: aws-eusc with BASE_DOMAIN: ci-eusc.devcluster.openshift.com
and FEATURE_SET: TechPreviewNoUpgrade.
@liweinan
Remove manual AWS endpoint configuration for EUSC workflows
b2da59b 
The installer now configures service endpoints implicitly for EUSC partition,
so manual endpoint configuration via ipi-conf-aws-custom-endpoints is no longer needed.

Changes:
- Remove ipi-conf-aws-custom-endpoints from all 5 EUSC workflow files
- Update documentation to reflect implicit endpoint configuration
- Simplify workflow by relying on installer's built-in EUSC support

This addresses review feedback from yunjiang29 that the installer handles
endpoints automatically for special AWS partitions like EUSC.
@liweinan
Regenerate jobs after rebase to origin/main
f00b18e 
Update generated Prow job configurations after rebasing to the latest origin/main.
Changes include:
- Updated cluster assignments to match current build cluster distribution
- EUSC jobs properly integrated with latest job generation logic
@liweinan
Refactor: Use standard AWS workflows for EUSC jobs
806ccd1 
Delete 4 EUSC-specific workflows and 2 provision chains, replacing them
with standard AWS workflows. This reduces maintenance burden and ensures
consistency with standard AWS job configurations.

Changes:
- Delete cucushift-installer-rehearse-aws-eusc-ipi workflow
- Delete cucushift-installer-rehearse-aws-eusc-ipi-private workflow
- Delete cucushift-installer-rehearse-aws-eusc-ipi-private-sts workflow
- Delete cucushift-installer-rehearse-aws-eusc-ipi-disconnected-private workflow
- Delete cucushift-installer-rehearse-aws-eusc-ipi provision chain
- Delete cucushift-installer-rehearse-aws-eusc-ipi-private provision chain

Modified 5 jobs to use standard AWS workflows:
- aws-eusc-ipi-fips-f7 → cucushift-installer-rehearse-aws-ipi
- aws-eusc-ipi-f28-destructive → cucushift-installer-rehearse-aws-ipi
- aws-eusc-ipi-private-sts-fips-f7 → aws-ipi-private-cco-manual-security-token-service
- aws-eusc-ipi-private-mini-perm-f28 → cucushift-installer-rehearse-aws-ipi-private
- aws-eusc-ipi-disc-priv-f28 → cucushift-installer-rehearse-aws-ipi-disconnected-private

All modified jobs now include:
- cluster_profile: aws-eusc (handles region and AMI configuration)
- COMPUTE_NODE_TYPE: m5.xlarge
- CONTROL_PLANE_INSTANCE_TYPE: m6i.xlarge

Preserved for further discussion:
- cucushift-installer-rehearse-aws-eusc-ipi-disconnected-private-kms
  (unique combination not available in standard AWS workflows)

Result: -300 lines, 100% workflow reuse for modified jobs
@liweinan
Complete EUSC workflow cleanup: use standard AWS workflows
8c2a0ab 
- Delete last EUSC-specific workflow: cucushift-installer-rehearse-aws-eusc-ipi-disconnected-private-kms
- Delete associated provision chain
- Update aws-eusc-ipi-disc-priv-kms-f7 job to use standard cucushift-installer-rehearse-aws-ipi-disconnected-private workflow
- Add COMPUTE_NODE_TYPE and CONTROL_PLANE_INSTANCE_TYPE env vars to the job

All EUSC jobs now use standard AWS workflows with cluster_profile: aws-eusc.
This completes the refactoring based on review feedback.
@liweinan
Fix AMI configuration bugs: respect user-provided AMI values
59290dc 
Two critical bug fixes in ipi-conf-aws-commands.sh:

1. Fix CONTROL_PLANE_AMI being unconditionally overwritten
   - Before: Always fetched from GitHub in C2S/SC2S environments
   - After: Only auto-detect if user hasn't provided CONTROL_PLANE_AMI
   - Impact: Users can now override AMI for control plane nodes

2. Fix COMPUTE_AMI being unconditionally overwritten
   - Before: COMPUTE_AMI="${CONTROL_PLANE_AMI}" (always overwrites)
   - After: COMPUTE_AMI="${COMPUTE_AMI:-${CONTROL_PLANE_AMI}}" (respects user value)
   - Impact: Users can now specify different AMIs for compute nodes

Both fixes are 100% backward compatible with existing jobs.
All current C2S/SC2S jobs don't set these env vars, so behavior unchanged.
@liweinan
Address review feedback: use multi-nightly for all EUSC jobs
990f321 
Changes per yunjiang29's review comments:

1. Remove all 6 EUSC jobs from amd64-nightly.yaml
   - All EUSC jobs now run against multi-nightly payload only
   - ARM for non-destructive (f7), AMD for destructive (f28)

2. Fix ipi-conf-aws-commands.sh for C2S/SC2S:
   - Restore version info comment: "# custom rhcos ami for non-public regions"
   - Restore inline comments: "# 4.9 and below" and "# 4.10 and above"
   - Add COMPUTE_AMI and echo in C2S block
   - Remove unreasonable default COMPUTE_AMI logic outside C2S block

3. Fix multi-nightly.yaml jobs:
   a) Rename KMS job to include "etcd" and meet 61-char limit:
      aws-eusc-ipi-byo-kms-encryption-fips-tp-amd-f28-destructive
      → aws-eusc-ipi-byo-kms-etcd-encryption-fips-tp-f28-destructive
   b) Fix KMS config for destructive job:
      ENABLE_AWS_KMS_KEY_COMPUTE/CONTROL_PLANE: yes → no
      ENABLE_AWS_KMS_KEY_DEFAULT_MACHINE: no → yes
   c) Add -mini-perm to STS job names (they use AWS_INSTALL_USE_MINIMAL_PERMISSIONS):
      aws-eusc-ipi-private-sts-tp-arm-f7
      → aws-eusc-ipi-private-sts-mini-perm-tp-arm-f7
      aws-eusc-ipi-private-sts-tp-amd-f28-destructive
      → aws-eusc-ipi-private-sts-mini-perm-tp-amd-f28-destructive

Result:
- 8 EUSC jobs in multi-nightly (4 ARM f7 + 4 AMD f28-destructive)
- 4 installer presubmit jobs (unchanged)
- 0 EUSC jobs in amd64-nightly
- Total: 12 EUSC jobs (down from 18)
@liweinan
Address review feedback: fix AMI check and add EUSC jobs to 4.23/5.0
debebb2 
Fix the AMI configuration condition to check both CONTROL_PLANE_AMI and
COMPUTE_AMI are empty before auto-fetching RHCOS AMIs for C2S/SC2S regions.
Add AWS EUSC CI jobs to release-4.23 and release-5.0 multi-nightly configs.

Regenerate jobs after rebase to latest main
@liweinan
Address review feedback: Custom DNS domain, aws-eusc support, and AMI…
669ad59 
… config

1. Configure Custom DNS jobs to use GCP Cloud DNS domain
   - Changed BASE_DOMAIN to qe.gcp.devcluster.openshift.com
   - Modified 6 custom DNS jobs across 4.22, 4.23, 5.0
   - Custom DNS workflow requires external GCP Cloud DNS for testing

2. Add aws-eusc cluster type support to install scripts
   - Modified ipi-install-install-aws-commands.sh (1 location)
   - Modified ipi-install-install-commands.sh (2 locations)
   - Fixes "Unsupported cluster type 'aws-eusc'" error

3. Configure verified AMI for all EUSC jobs
   - Use ami-0b78302f83217d149 (120GB gp3 volume, verified working)
   - Added COMPUTE_AMI and CONTROL_PLANE_AMI to 28 jobs:
     - 24 openshift-tests-private multi-nightly jobs
     - 4 installer presubmit jobs
   - Do not use ami-0c5e051fb7dc39f8d (has 2GB volume issue)
@liweinan
Regenerate jobs after rebase to origin/main
2faef2b
@liweinan liweinan force-pushed the add-aws-eusc-ci-jobs branch from 66c8136 to 2faef2b Compare March 23, 2026 10:48
@openshift-ci openshift-ci bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 23, 2026
@liweinan
Fix remaining custom DNS jobs in 4.22 to use GCP domain
1a90406 
The rebase accidentally reverted two custom DNS jobs back to ci-eusc
domain. This commit fixes them to use qe.gcp.devcluster.openshift.com
as required by the custom DNS workflow.
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 23, 2026

[REHEARSALNOTIFIER]
@liweinan: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-console-crontab-plugin-main-e2e-aws-console-crontab-plugin openshift/console-crontab-plugin presubmit Registry content changed
pull-ci-openshift-console-crontab-plugin-release-5.0-e2e-aws-console-crontab-plugin openshift/console-crontab-plugin presubmit Registry content changed
pull-ci-openshift-console-crontab-plugin-release-4.23-e2e-aws-console-crontab-plugin openshift/console-crontab-plugin presubmit Registry content changed
pull-ci-openshift-console-crontab-plugin-release-4.22-e2e-aws-console-crontab-plugin openshift/console-crontab-plugin presubmit Registry content changed
pull-ci-openshift-console-crontab-plugin-release-4.21-e2e-aws-console-crontab-plugin openshift/console-crontab-plugin presubmit Registry content changed
pull-ci-openshift-console-crontab-plugin-release-4.20-e2e-aws-console-crontab-plugin openshift/console-crontab-plugin presubmit Registry content changed
pull-ci-openshift-console-crontab-plugin-release-4.19-e2e-aws-console-crontab-plugin openshift/console-crontab-plugin presubmit Registry content changed
pull-ci-openshift-console-crontab-plugin-release-4.18-e2e-aws-console-crontab-plugin openshift/console-crontab-plugin presubmit Registry content changed
pull-ci-openshift-console-crontab-plugin-release-4.17-e2e-aws-console-crontab-plugin openshift/console-crontab-plugin presubmit Registry content changed
pull-ci-openshift-console-crontab-plugin-release-4.16-e2e-aws-console-crontab-plugin openshift/console-crontab-plugin presubmit Registry content changed
pull-ci-openshift-console-crontab-plugin-release-4.15-e2e-aws-console-crontab-plugin openshift/console-crontab-plugin presubmit Registry content changed
pull-ci-openshift-console-crontab-plugin-release-4.14-e2e-aws-console-crontab-plugin openshift/console-crontab-plugin presubmit Registry content changed
pull-ci-openshift-prometheus-operator-main-e2e-agnostic-cmo openshift/prometheus-operator presubmit Registry content changed
pull-ci-openshift-prometheus-operator-main-ginkgo-tests-cmo openshift/prometheus-operator presubmit Registry content changed
pull-ci-openshift-prometheus-operator-release-5.0-e2e-agnostic-cmo openshift/prometheus-operator presubmit Registry content changed
pull-ci-openshift-prometheus-operator-release-5.0-ginkgo-tests-cmo openshift/prometheus-operator presubmit Registry content changed
pull-ci-openshift-prometheus-operator-release-4.23-e2e-agnostic-cmo openshift/prometheus-operator presubmit Registry content changed
pull-ci-openshift-prometheus-operator-release-4.23-ginkgo-tests-cmo openshift/prometheus-operator presubmit Registry content changed
pull-ci-openshift-prometheus-operator-release-4.22-e2e-agnostic-cmo openshift/prometheus-operator presubmit Registry content changed
pull-ci-openshift-prometheus-operator-release-4.22-ginkgo-tests-cmo openshift/prometheus-operator presubmit Registry content changed
pull-ci-openshift-prometheus-operator-release-4.21-e2e-agnostic-cmo openshift/prometheus-operator presubmit Registry content changed
pull-ci-openshift-prometheus-operator-release-4.21-ginkgo-tests-cmo openshift/prometheus-operator presubmit Registry content changed
pull-ci-openshift-prometheus-operator-release-4.20-e2e-agnostic-cmo openshift/prometheus-operator presubmit Registry content changed
pull-ci-openshift-prometheus-operator-release-4.19-e2e-agnostic-cmo openshift/prometheus-operator presubmit Registry content changed
pull-ci-openshift-prometheus-operator-release-4.18-e2e-agnostic-cmo openshift/prometheus-operator presubmit Registry content changed

A total of 31942 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@liweinan
Copy link
Contributor Author

liweinan commented Mar 23, 2026

@yunjiang29 Thanks for the detailed review! I have updated the PR accordingly.

@patrickdillon @tthvo As openshift/cluster-ingress-operator#1360 is verified locally: #75568 (comment), after that PR is merged, I guess we can use the job here for testing then.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 23, 2026

[REHEARSALNOTIFIER]
@liweinan: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-lws-operator-main-e2e-aws-operator openshift/lws-operator presubmit Registry content changed
pull-ci-openshift-lws-operator-main-e2e-aws-operand openshift/lws-operator presubmit Registry content changed
pull-ci-openshift-lws-operator-release-5.0-e2e-aws-operator openshift/lws-operator presubmit Registry content changed
pull-ci-openshift-lws-operator-release-5.0-e2e-aws-operand openshift/lws-operator presubmit Registry content changed
pull-ci-openshift-lws-operator-release-4.23-e2e-aws-operator openshift/lws-operator presubmit Registry content changed
pull-ci-openshift-lws-operator-release-4.23-e2e-aws-operand openshift/lws-operator presubmit Registry content changed
pull-ci-openshift-lws-operator-release-4.22-e2e-aws-operator openshift/lws-operator presubmit Registry content changed
pull-ci-openshift-lws-operator-release-4.22-e2e-aws-operand openshift/lws-operator presubmit Registry content changed
pull-ci-openshift-lws-operator-release-4.21-e2e-aws-operator openshift/lws-operator presubmit Registry content changed
pull-ci-openshift-lws-operator-release-4.21-e2e-aws-operand openshift/lws-operator presubmit Registry content changed
pull-ci-openshift-lws-operator-release-4.20-e2e-aws-operator openshift/lws-operator presubmit Registry content changed
pull-ci-openshift-lws-operator-release-4.20-e2e-aws-operand openshift/lws-operator presubmit Registry content changed
pull-ci-openshift-lws-operator-release-4.19-e2e-aws-operator openshift/lws-operator presubmit Registry content changed
pull-ci-openshift-lws-operator-release-4.19-e2e-aws-operand openshift/lws-operator presubmit Registry content changed
pull-ci-eclipse-che-che-server-main-v19-gitlab-no-pat-oauth-flow eclipse-che/che-server presubmit Registry content changed
pull-ci-eclipse-che-che-server-main-v19-gitlab-no-pat-oauth-flow-raw-devfile-url eclipse-che/che-server presubmit Registry content changed
pull-ci-eclipse-che-che-server-main-v19-gitlab-no-pat-oauth-flow-ssh-url eclipse-che/che-server presubmit Registry content changed
pull-ci-eclipse-che-che-server-main-v19-bitbucket-no-pat-oauth-flow eclipse-che/che-server presubmit Registry content changed
pull-ci-eclipse-che-che-server-main-v19-bitbucket-no-pat-oauth-flow-raw-devfile-url eclipse-che/che-server presubmit Registry content changed
pull-ci-eclipse-che-che-server-main-v19-bitbucket-no-pat-oauth-flow-ssh-url eclipse-che/che-server presubmit Registry content changed
pull-ci-eclipse-che-che-server-main-v19-github-no-pat-oauth-flow eclipse-che/che-server presubmit Registry content changed
pull-ci-eclipse-che-che-server-main-v19-github-no-pat-oauth-flow-raw-devfile-url eclipse-che/che-server presubmit Registry content changed
pull-ci-eclipse-che-che-server-main-v19-github-no-pat-oauth-flow-ssh-url eclipse-che/che-server presubmit Registry content changed
pull-ci-eclipse-che-che-server-main-v19-github-with-pat-setup-flow eclipse-che/che-server presubmit Registry content changed
pull-ci-eclipse-che-che-server-main-v19-gitlab-with-pat-setup-flow eclipse-che/che-server presubmit Registry content changed

A total of 31942 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

yunjiang29
yunjiang29 reviewed Mar 23, 2026
View reviewed changes
...nshift/openshift-tests-private/openshift-openshift-tests-private-release-4.22-periodics.yaml
@@ -625,7 +625,7 @@ periodics:
secret:
secretName: result-aggregator
- agent: kubernetes
cluster: build09
cluster: build10
Copy link
Contributor

@yunjiang29 yunjiang29 Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liweinan you can try to do a rebase to avoid such updates in your PR.

@patrickdillon
Copy link
Contributor

patrickdillon commented Mar 23, 2026

Rehearsal run shows that the credentials are not being found:

https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/75568/rehearse-75568-pull-ci-openshift-installer-main-e2e-aws-eusc-techpreview/2035055582121037824

@openshift-ci openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 23, 2026
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 23, 2026

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@neisw neisw Awaiting requested review from neisw

@xingxingxia xingxingxia Awaiting requested review from xingxingxia

1 more reviewer

@yunjiang29 yunjiang29 yunjiang29 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@liweinan @openshift-ci-robot @yunjiang29 @patrickdillon @tthvo @openshift-merge-robot