Sep 08 15:08:57.742752 ip-10-0-136-159 systemd[1]: Starting Kubernetes Kubelet... Sep 08 15:08:58.165000 ip-10-0-136-159 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 15:08:58.165000 ip-10-0-136-159 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 08 15:08:58.165000 ip-10-0-136-159 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 15:08:58.165000 ip-10-0-136-159 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 08 15:08:58.165000 ip-10-0-136-159 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 15:08:58.167814 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.167741 2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 08 15:08:58.171892 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171877 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 15:08:58.171929 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171893 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 15:08:58.171929 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171897 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 15:08:58.171929 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171900 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 15:08:58.171929 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171903 2572 feature_gate.go:328] unrecognized feature gate: Example Sep 08 15:08:58.171929 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171906 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 15:08:58.171929 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171909 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 15:08:58.171929 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171911 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 15:08:58.171929 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171914 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 15:08:58.171929 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171917 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 15:08:58.171929 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171919 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 15:08:58.171929 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171922 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 15:08:58.171929 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171924 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 15:08:58.171929 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171927 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 15:08:58.171929 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171929 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 15:08:58.171929 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171932 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 15:08:58.171929 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171935 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 15:08:58.171929 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171937 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171941 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171943 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171946 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171949 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171951 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171962 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171965 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171967 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171970 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171973 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171975 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171978 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171982 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171984 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171987 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171990 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171992 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171995 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171997 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 15:08:58.172330 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.171999 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 15:08:58.172824 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172002 2572 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 15:08:58.172824 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172004 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 15:08:58.172824 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172007 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 15:08:58.172824 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172009 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 15:08:58.172824 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172012 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 15:08:58.172824 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172014 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 15:08:58.172824 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172016 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 15:08:58.172824 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172019 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 15:08:58.172824 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172021 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 15:08:58.172824 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172025 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 15:08:58.172824 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172029 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 15:08:58.172824 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172031 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 15:08:58.172824 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172034 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 15:08:58.172824 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172038 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 15:08:58.172824 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172040 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 15:08:58.172824 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172043 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 15:08:58.172824 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172045 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 15:08:58.172824 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172048 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 15:08:58.172824 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172051 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172053 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172056 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172058 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172061 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172064 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172066 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172069 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172071 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172073 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172076 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172078 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172081 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172083 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172086 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172092 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172095 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172097 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172100 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172102 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 15:08:58.173277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172105 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 15:08:58.173771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172107 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 15:08:58.173771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172110 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 15:08:58.173771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172114 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 15:08:58.173771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172117 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 15:08:58.173771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172123 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 15:08:58.173771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172126 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 15:08:58.173771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172129 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 15:08:58.173771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172132 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 15:08:58.173771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172135 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 15:08:58.173771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172137 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 15:08:58.173771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172140 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 15:08:58.173771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172143 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 15:08:58.173771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172145 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 15:08:58.173771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172148 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 15:08:58.173771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172150 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 15:08:58.173771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172153 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 15:08:58.173771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172155 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 15:08:58.173771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172157 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 15:08:58.173771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172160 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 15:08:58.174264 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172163 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 15:08:58.174264 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172166 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 15:08:58.174264 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.172169 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 15:08:58.174680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174668 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 15:08:58.174680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174680 2572 feature_gate.go:328] unrecognized feature gate: Example Sep 08 15:08:58.174738 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174683 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 15:08:58.174738 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174687 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 15:08:58.174738 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174690 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 15:08:58.174738 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174693 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 15:08:58.174738 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174696 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 15:08:58.174738 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174699 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 15:08:58.174738 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174702 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 15:08:58.174738 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174704 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 15:08:58.174738 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174707 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 15:08:58.174738 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174710 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 15:08:58.174738 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174712 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 15:08:58.174738 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174716 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 15:08:58.174738 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174719 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 15:08:58.174738 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174722 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 15:08:58.174738 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174725 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 15:08:58.174738 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174728 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 15:08:58.174738 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174730 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 15:08:58.174738 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174733 2572 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 15:08:58.174738 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174736 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 15:08:58.175207 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174738 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 15:08:58.175207 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174741 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 15:08:58.175207 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174743 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 15:08:58.175207 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174747 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 15:08:58.175207 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174751 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 15:08:58.175207 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174754 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 15:08:58.175207 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174757 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 15:08:58.175207 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174759 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 15:08:58.175207 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174762 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 15:08:58.175207 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174764 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 15:08:58.175207 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174767 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 15:08:58.175207 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174769 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 15:08:58.175207 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174772 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 15:08:58.175207 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174776 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 15:08:58.175207 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174779 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 15:08:58.175207 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174782 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 15:08:58.175207 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174785 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 15:08:58.175207 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174789 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 15:08:58.175207 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174792 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174795 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174798 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174800 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174803 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174805 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174808 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174810 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174813 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174815 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174818 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174820 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174823 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174825 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174828 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174831 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174834 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174836 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174839 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174841 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 15:08:58.175689 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174844 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174846 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174849 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174852 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174855 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174858 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174861 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174863 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174866 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174868 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174870 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174873 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174876 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174878 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174881 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174883 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174886 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174888 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174891 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174893 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 15:08:58.176184 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174895 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174898 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174901 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174904 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174906 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174909 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174912 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174915 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174918 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174920 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174923 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174925 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174928 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174930 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174933 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174935 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174937 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174940 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174943 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.174945 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 15:08:58.176680 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175331 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175336 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175339 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175342 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175345 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175348 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175350 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175353 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175356 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175360 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175363 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175366 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175368 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175371 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175373 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175376 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175379 2572 feature_gate.go:328] unrecognized feature gate: Example Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175382 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175384 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175387 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 15:08:58.177170 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175390 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175392 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175395 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175397 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175400 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175402 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175405 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175407 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175409 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175412 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175414 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175417 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175419 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175421 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175424 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175426 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175429 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175437 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175439 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175442 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 15:08:58.177676 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175444 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 15:08:58.178169 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175447 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 15:08:58.178169 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175449 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 15:08:58.178169 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175452 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 15:08:58.178169 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175455 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 15:08:58.178169 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175457 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 15:08:58.178169 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175459 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 15:08:58.178169 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175462 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 15:08:58.178169 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175464 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 15:08:58.178169 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175467 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 15:08:58.178169 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175470 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 15:08:58.178169 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175472 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 15:08:58.178169 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175476 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 15:08:58.178169 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175479 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 15:08:58.178169 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175482 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 15:08:58.178169 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175485 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 15:08:58.178169 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175501 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 15:08:58.178169 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175505 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 15:08:58.178169 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175508 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 15:08:58.178169 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175510 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 15:08:58.178640 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175513 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 15:08:58.178640 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175515 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 15:08:58.178640 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175518 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 15:08:58.178640 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175520 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 15:08:58.178640 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175523 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 15:08:58.178640 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175525 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 15:08:58.178640 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175528 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 15:08:58.178640 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175531 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 15:08:58.178640 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175533 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 15:08:58.178640 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175537 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 15:08:58.178640 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175546 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 15:08:58.178640 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175549 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 15:08:58.178640 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175552 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 15:08:58.178640 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175555 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 15:08:58.178640 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175558 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 15:08:58.178640 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175561 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 15:08:58.178640 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175563 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 15:08:58.178640 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175566 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 15:08:58.178640 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175568 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175571 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175573 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175577 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175579 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175582 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175584 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175588 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175590 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175594 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175596 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175599 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175601 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175603 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175606 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175608 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175611 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175613 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175616 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175618 2572 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.175621 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 15:08:58.179103 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175702 2572 flags.go:64] FLAG: --address="0.0.0.0" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175750 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175756 2572 flags.go:64] FLAG: --anonymous-auth="true" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175760 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175765 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175768 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175773 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175776 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175780 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175782 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175786 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175789 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175792 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175795 2572 flags.go:64] FLAG: --cgroup-root="" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175805 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175808 2572 flags.go:64] FLAG: --client-ca-file="" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175811 2572 flags.go:64] FLAG: --cloud-provider="external" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175814 2572 flags.go:64] FLAG: --cluster-dns="[]" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175819 2572 flags.go:64] FLAG: --cluster-domain="" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175822 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175825 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175828 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175831 2572 flags.go:64] FLAG: --container-log-max-files="5" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175835 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 08 15:08:58.179651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175838 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175841 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175844 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175847 2572 flags.go:64] FLAG: --contention-profiling="false" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175850 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175853 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175857 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175859 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175863 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175866 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175869 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175872 2572 flags.go:64] FLAG: --enable-load-reader="false" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175875 2572 flags.go:64] FLAG: --enable-server="true" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175878 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175883 2572 flags.go:64] FLAG: --event-burst="100" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175886 2572 flags.go:64] FLAG: --event-qps="50" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175889 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175891 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175895 2572 flags.go:64] FLAG: --eviction-hard="" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175898 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175901 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175904 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175908 2572 flags.go:64] FLAG: --eviction-soft="" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175911 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175914 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 08 15:08:58.180236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175917 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175920 2572 flags.go:64] FLAG: --experimental-mounter-path="" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175923 2572 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175926 2572 flags.go:64] FLAG: --fail-swap-on="true" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175928 2572 flags.go:64] FLAG: --feature-gates="" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175932 2572 flags.go:64] FLAG: --file-check-frequency="20s" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175935 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175938 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175940 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175944 2572 flags.go:64] FLAG: --healthz-port="10248" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175946 2572 flags.go:64] FLAG: --help="false" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175949 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-136-159.ec2.internal" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175952 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175955 2572 flags.go:64] FLAG: --http-check-frequency="20s" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175958 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175961 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175964 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175967 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175970 2572 flags.go:64] FLAG: --image-service-endpoint="" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175973 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175976 2572 flags.go:64] FLAG: --kube-api-burst="100" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175981 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175984 2572 flags.go:64] FLAG: --kube-api-qps="50" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175987 2572 flags.go:64] FLAG: --kube-reserved="" Sep 08 15:08:58.180863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175990 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175993 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175996 2572 flags.go:64] FLAG: --kubelet-cgroups="" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.175998 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176001 2572 flags.go:64] FLAG: --lock-file="" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176006 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176009 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176012 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176017 2572 flags.go:64] FLAG: --log-json-split-stream="false" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176020 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176022 2572 flags.go:64] FLAG: --log-text-split-stream="false" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176025 2572 flags.go:64] FLAG: --logging-format="text" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176028 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176031 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176034 2572 flags.go:64] FLAG: --manifest-url="" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176036 2572 flags.go:64] FLAG: --manifest-url-header="" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176040 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176043 2572 flags.go:64] FLAG: --max-open-files="1000000" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176047 2572 flags.go:64] FLAG: --max-pods="110" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176050 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176053 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176055 2572 flags.go:64] FLAG: --memory-manager-policy="None" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176058 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176061 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176064 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 08 15:08:58.181442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176067 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176073 2572 flags.go:64] FLAG: --node-status-max-images="50" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176076 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176079 2572 flags.go:64] FLAG: --oom-score-adj="-999" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176083 2572 flags.go:64] FLAG: --pod-cidr="" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176086 2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176091 2572 flags.go:64] FLAG: --pod-manifest-path="" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176094 2572 flags.go:64] FLAG: --pod-max-pids="-1" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176097 2572 flags.go:64] FLAG: --pods-per-core="0" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176100 2572 flags.go:64] FLAG: --port="10250" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176103 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176105 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-008ecebbb40e3e7e0" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176110 2572 flags.go:64] FLAG: --qos-reserved="" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176113 2572 flags.go:64] FLAG: --read-only-port="10255" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176116 2572 flags.go:64] FLAG: --register-node="true" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176118 2572 flags.go:64] FLAG: --register-with-taints="" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176122 2572 flags.go:64] FLAG: --registry-burst="10" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176125 2572 flags.go:64] FLAG: --registry-qps="5" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176127 2572 flags.go:64] FLAG: --reserved-cpus="" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176130 2572 flags.go:64] FLAG: --reserved-memory="" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176134 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176137 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176139 2572 flags.go:64] FLAG: --rotate-certificates="false" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176142 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176145 2572 flags.go:64] FLAG: --runonce="false" Sep 08 15:08:58.182063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176148 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176151 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176154 2572 flags.go:64] FLAG: --seccomp-default="false" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176157 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176159 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176162 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176165 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176168 2572 flags.go:64] FLAG: --storage-driver-password="root" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176171 2572 flags.go:64] FLAG: --storage-driver-secure="false" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176174 2572 flags.go:64] FLAG: --storage-driver-table="stats" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176176 2572 flags.go:64] FLAG: --storage-driver-user="root" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176179 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176183 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176187 2572 flags.go:64] FLAG: --system-cgroups="" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176190 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176195 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176198 2572 flags.go:64] FLAG: --tls-cert-file="" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176201 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176205 2572 flags.go:64] FLAG: --tls-min-version="" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176209 2572 flags.go:64] FLAG: --tls-private-key-file="" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176212 2572 flags.go:64] FLAG: --topology-manager-policy="none" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176215 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176218 2572 flags.go:64] FLAG: --topology-manager-scope="container" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176221 2572 flags.go:64] FLAG: --v="2" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176224 2572 flags.go:64] FLAG: --version="false" Sep 08 15:08:58.182677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176228 2572 flags.go:64] FLAG: --vmodule="" Sep 08 15:08:58.183311 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176232 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 08 15:08:58.183311 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.176235 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 08 15:08:58.183311 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176365 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 15:08:58.183311 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176369 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 15:08:58.183311 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176372 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 15:08:58.183311 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176375 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 15:08:58.183311 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176378 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 15:08:58.183311 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176381 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 15:08:58.183311 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176384 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 15:08:58.183311 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176387 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 15:08:58.183311 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176390 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 15:08:58.183311 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176392 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 15:08:58.183311 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176395 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 15:08:58.183311 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176398 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 15:08:58.183311 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176401 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 15:08:58.183311 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176404 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 15:08:58.183311 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176408 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 15:08:58.183311 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176412 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 15:08:58.183311 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176416 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176418 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176421 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176424 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176427 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176430 2572 feature_gate.go:328] unrecognized feature gate: Example Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176432 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176435 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176438 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176441 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176443 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176445 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176448 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176450 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176453 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176455 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176458 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176460 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176463 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176465 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 15:08:58.183901 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176468 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176470 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176473 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176476 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176478 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176481 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176483 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176486 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176503 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176506 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176508 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176511 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176515 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176518 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176520 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176523 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176526 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176528 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176531 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176535 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 15:08:58.184385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176537 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176540 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176543 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176545 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176548 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176551 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176553 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176555 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176558 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176560 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176563 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176565 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176568 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176571 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176574 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176586 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176589 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176592 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176595 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176598 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 15:08:58.184894 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176601 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176603 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176607 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176609 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176613 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176616 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176619 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176622 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176625 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176627 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176630 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176634 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176637 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176639 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176642 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176644 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176647 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176650 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176652 2572 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176654 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 15:08:58.185628 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176657 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 15:08:58.186523 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176660 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 15:08:58.186523 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.176663 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 15:08:58.186523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.177577 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 15:08:58.186523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.185078 2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 08 15:08:58.186523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.185093 2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 08 15:08:58.186523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.185110 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 08 15:08:58.186523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.185119 2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 08 15:08:58.186523 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185183 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 15:08:58.186523 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185188 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 15:08:58.186523 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185192 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 15:08:58.186523 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185195 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 15:08:58.186523 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185199 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 15:08:58.186523 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185202 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 15:08:58.186523 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185205 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 15:08:58.187119 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185208 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 15:08:58.187119 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185211 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 15:08:58.187119 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185213 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 15:08:58.187119 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185216 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 15:08:58.187119 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185220 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 15:08:58.187119 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185230 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 15:08:58.187119 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185233 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 15:08:58.187119 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185236 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 15:08:58.187119 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185239 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 15:08:58.187119 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185241 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 15:08:58.187119 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185244 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 15:08:58.187119 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185246 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 15:08:58.187119 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185249 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 15:08:58.187119 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185252 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 15:08:58.187119 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185254 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 15:08:58.187119 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185257 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 15:08:58.187119 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185260 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 15:08:58.187119 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185264 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 15:08:58.187119 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185267 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185270 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185272 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185275 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185279 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185281 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185284 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185287 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185290 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185292 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185295 2572 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185298 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185300 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185306 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185310 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185314 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185316 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185319 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185321 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185324 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 15:08:58.188058 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185332 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185334 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185344 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185347 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185349 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185352 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185355 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185357 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185359 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185362 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185365 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185367 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185370 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185372 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185375 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185378 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185381 2572 feature_gate.go:328] unrecognized feature gate: Example Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185384 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185387 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185389 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 15:08:58.188580 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185392 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185394 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185397 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185399 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185402 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185404 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185407 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185409 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185411 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185414 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185416 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185419 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185421 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185429 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185432 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185435 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185438 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185440 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185442 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185446 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 15:08:58.189066 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185448 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 15:08:58.189577 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185451 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 15:08:58.189577 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185454 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 15:08:58.189577 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185456 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 15:08:58.189577 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185459 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 15:08:58.189577 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185461 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 15:08:58.189577 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185464 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 15:08:58.189577 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185466 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 15:08:58.189577 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185469 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 15:08:58.189577 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185471 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 15:08:58.189577 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185474 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 15:08:58.189577 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185476 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 15:08:58.189577 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185479 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 15:08:58.189577 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185481 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 15:08:58.189577 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.185486 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185668 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185673 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185676 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185678 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185681 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185683 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185686 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185688 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185691 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185693 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185695 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185705 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185708 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185710 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185713 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185715 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185717 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185720 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185723 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 15:08:58.189943 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185725 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 15:08:58.190410 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185727 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 15:08:58.190410 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185730 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 15:08:58.190410 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185732 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 15:08:58.190410 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185735 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 15:08:58.190410 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185738 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 15:08:58.190410 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185742 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 15:08:58.190410 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185744 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 15:08:58.190410 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185747 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 15:08:58.190410 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185749 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 15:08:58.190410 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185752 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 15:08:58.190410 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185754 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 15:08:58.190410 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185756 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 15:08:58.190410 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185759 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 15:08:58.190410 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185761 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 15:08:58.190410 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185764 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 15:08:58.190410 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185766 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 15:08:58.190410 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185768 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 15:08:58.190410 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185771 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 15:08:58.190410 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185773 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185776 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185778 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185781 2572 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185783 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185786 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185794 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185796 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185799 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185801 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185804 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185806 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185809 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185811 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185815 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185819 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185821 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185824 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185826 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185829 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 15:08:58.190888 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185831 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185834 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185836 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185839 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185841 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185843 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185846 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185848 2572 feature_gate.go:328] unrecognized feature gate: Example Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185851 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185853 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185856 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185858 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185862 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185865 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185868 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185871 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185873 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185876 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185878 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185887 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 15:08:58.191371 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185889 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185892 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185894 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185897 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185899 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185902 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185904 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185906 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185909 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185911 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185914 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185916 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185918 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185921 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185923 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185925 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185928 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185930 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185933 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185935 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 15:08:58.191881 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:58.185937 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 15:08:58.192363 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.185942 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 15:08:58.192363 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.186047 2572 server.go:962] "Client rotation is on, will bootstrap in background" Sep 08 15:08:58.192363 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.188517 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 08 15:08:58.192363 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.189602 2572 server.go:1019] "Starting client certificate rotation" Sep 08 15:08:58.192363 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.189724 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 15:08:58.192363 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.189754 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 15:08:58.192658 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.192644 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 15:08:58.196726 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.196682 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 15:08:58.207686 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.207669 2572 log.go:25] "Validated CRI v1 runtime API" Sep 08 15:08:58.211183 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.211166 2572 log.go:25] "Validated CRI v1 image API" Sep 08 15:08:58.212107 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.212083 2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 08 15:08:58.215484 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.215467 2572 fs.go:135] Filesystem UUIDs: map[240e85dc-a9b5-4d68-8b55-50da6ca0ca35:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cf1ee93b-b4c3-415a-a2b0-f054d1af6d1f:/dev/nvme0n1p3] Sep 08 15:08:58.215575 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.215483 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 08 15:08:58.221221 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.221117 2572 manager.go:216] Machine: {Timestamp:2026-09-08 15:08:58.219764666 +0000 UTC m=+0.371943237 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100952 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f557ab2ae50f3f32b6822ecb0b9ae SystemUUID:ec2f557a-b2ae-50f3-f32b-6822ecb0b9ae BootID:ba6c1f7a-d5d9-4406-af1c-5c97a9f48778 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:af:b4:a8:ea:31 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:af:b4:a8:ea:31 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:03:27:c5:d0:98 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 08 15:08:58.221221 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.221216 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 08 15:08:58.221350 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.221278 2572 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 08 15:08:58.221643 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.221627 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 15:08:58.224855 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.224833 2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 08 15:08:58.224990 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.224859 2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-159.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 08 15:08:58.226393 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.226383 2572 topology_manager.go:138] "Creating topology manager with none policy" Sep 08 15:08:58.226426 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.226396 2572 container_manager_linux.go:309] "Creating device plugin manager" Sep 08 15:08:58.226426 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.226403 2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 15:08:58.226426 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.226416 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 15:08:58.228311 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.228301 2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 08 15:08:58.232992 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.232981 2572 state_mem.go:36] "Initialized new in-memory state store" Sep 08 15:08:58.233107 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.233098 2572 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 08 15:08:58.233145 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.233126 2572 kubelet.go:486] "Attempting to sync node with API server" Sep 08 15:08:58.233145 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.233143 2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 08 15:08:58.233214 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.233157 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 08 15:08:58.233214 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.233167 2572 kubelet.go:398] "Adding apiserver pod source" Sep 08 15:08:58.233214 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.233174 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 08 15:08:58.233577 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.233557 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 15:08:58.233676 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.233586 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 15:08:58.236103 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.236078 2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 08 15:08:58.238847 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.238833 2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 08 15:08:58.238910 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.238878 2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 08 15:08:58.239066 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.239054 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 08 15:08:58.239104 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.239078 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 08 15:08:58.239104 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.239084 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 08 15:08:58.239104 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.239091 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 08 15:08:58.239104 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.239100 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 08 15:08:58.239104 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.239106 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 08 15:08:58.239689 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.239679 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 08 15:08:58.239689 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.239689 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 08 15:08:58.239745 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.239698 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 08 15:08:58.239745 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.239703 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 08 15:08:58.239745 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.239716 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 08 15:08:58.239745 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.239735 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 08 15:08:58.240632 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.240621 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 08 15:08:58.240662 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.240635 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 08 15:08:58.245097 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.245075 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-159.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 08 15:08:58.245255 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.245233 2572 server.go:1268] "Started kubelet" Sep 08 15:08:58.245902 ip-10-0-136-159 systemd[1]: Started Kubernetes Kubelet. Sep 08 15:08:58.246036 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.245976 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-159.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 08 15:08:58.246036 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.245975 2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 08 15:08:58.246132 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.245976 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 08 15:08:58.246254 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.246222 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 08 15:08:58.246345 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.246273 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 08 15:08:58.247973 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.247958 2572 server.go:310] "Adding debug handlers to kubelet server" Sep 08 15:08:58.248784 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.248766 2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 08 15:08:58.253774 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.253758 2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 08 15:08:58.253924 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.253911 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 08 15:08:58.254032 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.252883 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-159.ec2.internal.18d3610201db3b52 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-159.ec2.internal,UID:ip-10-0-136-159.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-159.ec2.internal,},FirstTimestamp:2026-09-08 15:08:58.24519253 +0000 UTC m=+0.397371101,LastTimestamp:2026-09-08 15:08:58.24519253 +0000 UTC m=+0.397371101,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-159.ec2.internal,}" Sep 08 15:08:58.254666 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.254652 2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 08 15:08:58.254792 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.254770 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-159.ec2.internal\" not found" Sep 08 15:08:58.255445 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.255429 2572 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 08 15:08:58.255445 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.255447 2572 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 08 15:08:58.255589 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.255429 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 08 15:08:58.255589 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.255536 2572 reconstruct.go:97] "Volume reconstruction finished" Sep 08 15:08:58.255589 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.255542 2572 reconciler.go:29] "Reconciler: start to sync state" Sep 08 15:08:58.255854 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.255831 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 08 15:08:58.255891 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.255858 2572 factory.go:55] Registering systemd factory Sep 08 15:08:58.255891 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.255868 2572 factory.go:223] Registration of the systemd container factory successfully Sep 08 15:08:58.256071 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.256059 2572 factory.go:153] Registering CRI-O factory Sep 08 15:08:58.256110 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.256073 2572 factory.go:223] Registration of the crio container factory successfully Sep 08 15:08:58.256110 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.256085 2572 factory.go:103] Registering Raw factory Sep 08 15:08:58.256110 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.256109 2572 manager.go:1196] Started watching for new ooms in manager Sep 08 15:08:58.256486 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.256469 2572 manager.go:349] Starting recovery of all containers Sep 08 15:08:58.264194 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.264049 2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-159.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 08 15:08:58.264194 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.264095 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 08 15:08:58.267647 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.267629 2572 manager.go:354] Recovery completed Sep 08 15:08:58.272149 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.272136 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 15:08:58.274147 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.274121 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasSufficientMemory" Sep 08 15:08:58.274188 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.274159 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 15:08:58.274188 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.274170 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasSufficientPID" Sep 08 15:08:58.274734 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.274711 2572 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 08 15:08:58.274734 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.274724 2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 08 15:08:58.274820 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.274740 2572 state_mem.go:36] "Initialized new in-memory state store" Sep 08 15:08:58.274865 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.274725 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 15:08:58.275526 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.275512 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasSufficientMemory" Sep 08 15:08:58.275574 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.275539 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 15:08:58.275574 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.275552 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasSufficientPID" Sep 08 15:08:58.276183 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.276115 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-159.ec2.internal.18d3610203950d3a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-159.ec2.internal,UID:ip-10-0-136-159.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-159.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-159.ec2.internal,},FirstTimestamp:2026-09-08 15:08:58.274147642 +0000 UTC m=+0.426326213,LastTimestamp:2026-09-08 15:08:58.274147642 +0000 UTC m=+0.426326213,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-159.ec2.internal,}" Sep 08 15:08:58.276874 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.276863 2572 policy_none.go:49] "None policy: Start" Sep 08 15:08:58.276937 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.276878 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 08 15:08:58.276937 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.276887 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 08 15:08:58.278821 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.278811 2572 policy_none.go:47] "Start" Sep 08 15:08:58.286323 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.286307 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kfqhw" Sep 08 15:08:58.289227 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.289165 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-159.ec2.internal.18d3610203955067 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-159.ec2.internal,UID:ip-10-0-136-159.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-159.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-159.ec2.internal,},FirstTimestamp:2026-09-08 15:08:58.274164839 +0000 UTC m=+0.426343410,LastTimestamp:2026-09-08 15:08:58.274164839 +0000 UTC m=+0.426343410,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-159.ec2.internal,}" Sep 08 15:08:58.293008 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.292992 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kfqhw" Sep 08 15:08:58.316249 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.316232 2572 manager.go:337] "Starting Device Plugin manager" Sep 08 15:08:58.316341 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.316275 2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 08 15:08:58.316341 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.316288 2572 server.go:87] "Starting device plugin registration server" Sep 08 15:08:58.316699 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.316581 2572 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 08 15:08:58.316699 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.316592 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 08 15:08:58.316835 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.316719 2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 08 15:08:58.316835 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.316787 2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 08 15:08:58.316835 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.316796 2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 08 15:08:58.317247 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.317217 2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 08 15:08:58.317314 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.317293 2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-159.ec2.internal\" not found" Sep 08 15:08:58.363775 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.363747 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 08 15:08:58.365005 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.364987 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 08 15:08:58.365076 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.365029 2572 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 08 15:08:58.365076 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.365046 2572 kubelet.go:2448] "Starting kubelet main sync loop" Sep 08 15:08:58.365169 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.365080 2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 08 15:08:58.367974 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.367951 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 15:08:58.417236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.417171 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 15:08:58.417939 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.417920 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasSufficientMemory" Sep 08 15:08:58.418005 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.417946 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 15:08:58.418005 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.417956 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasSufficientPID" Sep 08 15:08:58.418005 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.417976 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-159.ec2.internal" Sep 08 15:08:58.425742 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.425727 2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-159.ec2.internal" Sep 08 15:08:58.425813 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.425750 2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-159.ec2.internal\": node \"ip-10-0-136-159.ec2.internal\" not found" Sep 08 15:08:58.465148 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.465116 2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-159.ec2.internal"] Sep 08 15:08:58.465228 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.465198 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 15:08:58.466445 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.466430 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasSufficientMemory" Sep 08 15:08:58.466533 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.466456 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 15:08:58.466533 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.466471 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasSufficientPID" Sep 08 15:08:58.468772 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.468760 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 15:08:58.468892 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.468878 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal" Sep 08 15:08:58.468927 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.468905 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 15:08:58.469477 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.469460 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasSufficientMemory" Sep 08 15:08:58.469581 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.469509 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 15:08:58.469581 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.469526 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasSufficientPID" Sep 08 15:08:58.469581 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.469531 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasSufficientMemory" Sep 08 15:08:58.469581 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.469556 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 15:08:58.469581 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.469571 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasSufficientPID" Sep 08 15:08:58.471704 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.471689 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-159.ec2.internal" Sep 08 15:08:58.471786 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.471718 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 15:08:58.472480 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.472464 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasSufficientMemory" Sep 08 15:08:58.472552 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.472514 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 15:08:58.472552 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.472535 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasSufficientPID" Sep 08 15:08:58.478332 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.478318 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-159.ec2.internal\" not found" Sep 08 15:08:58.502030 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.502007 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-159.ec2.internal\" not found" node="ip-10-0-136-159.ec2.internal" Sep 08 15:08:58.506183 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.506165 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-159.ec2.internal\" not found" node="ip-10-0-136-159.ec2.internal" Sep 08 15:08:58.556846 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.556825 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e79f6357314a98cc42b005b66beb364-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal\" (UID: \"1e79f6357314a98cc42b005b66beb364\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal" Sep 08 15:08:58.556931 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.556856 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f326de9abe5883a85f6af224dafa6b15-config\") pod \"kube-apiserver-proxy-ip-10-0-136-159.ec2.internal\" (UID: \"f326de9abe5883a85f6af224dafa6b15\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-159.ec2.internal" Sep 08 15:08:58.556931 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.556880 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1e79f6357314a98cc42b005b66beb364-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal\" (UID: \"1e79f6357314a98cc42b005b66beb364\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal" Sep 08 15:08:58.579272 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.579248 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-159.ec2.internal\" not found" Sep 08 15:08:58.657251 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.657228 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1e79f6357314a98cc42b005b66beb364-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal\" (UID: \"1e79f6357314a98cc42b005b66beb364\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal" Sep 08 15:08:58.657341 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.657253 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e79f6357314a98cc42b005b66beb364-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal\" (UID: \"1e79f6357314a98cc42b005b66beb364\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal" Sep 08 15:08:58.657341 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.657271 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f326de9abe5883a85f6af224dafa6b15-config\") pod \"kube-apiserver-proxy-ip-10-0-136-159.ec2.internal\" (UID: \"f326de9abe5883a85f6af224dafa6b15\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-159.ec2.internal" Sep 08 15:08:58.657341 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.657308 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f326de9abe5883a85f6af224dafa6b15-config\") pod \"kube-apiserver-proxy-ip-10-0-136-159.ec2.internal\" (UID: \"f326de9abe5883a85f6af224dafa6b15\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-159.ec2.internal" Sep 08 15:08:58.657341 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.657335 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1e79f6357314a98cc42b005b66beb364-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal\" (UID: \"1e79f6357314a98cc42b005b66beb364\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal" Sep 08 15:08:58.657513 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.657341 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e79f6357314a98cc42b005b66beb364-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal\" (UID: \"1e79f6357314a98cc42b005b66beb364\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal" Sep 08 15:08:58.680390 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.680342 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-159.ec2.internal\" not found" Sep 08 15:08:58.781116 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.781096 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-159.ec2.internal\" not found" Sep 08 15:08:58.805256 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.805239 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal" Sep 08 15:08:58.808588 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:58.808573 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-159.ec2.internal" Sep 08 15:08:58.881810 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.881789 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-159.ec2.internal\" not found" Sep 08 15:08:58.982333 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:58.982280 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-159.ec2.internal\" not found" Sep 08 15:08:59.082840 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:59.082805 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-159.ec2.internal\" not found" Sep 08 15:08:59.183384 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:59.183351 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-159.ec2.internal\" not found" Sep 08 15:08:59.189539 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:59.189521 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 08 15:08:59.189681 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:59.189659 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 15:08:59.246839 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:59.246775 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 15:08:59.284470 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:59.284448 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-159.ec2.internal\" not found" Sep 08 15:08:59.295614 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:59.295586 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 15:03:58 +0000 UTC" deadline="2028-05-28 00:09:00.508844422 +0000 UTC" Sep 08 15:08:59.295614 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:59.295612 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15057h0m1.213235495s" Sep 08 15:08:59.385217 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:59.385196 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-159.ec2.internal\" not found" Sep 08 15:08:59.466778 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:59.466759 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 15:08:59.467995 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:59.467980 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasSufficientMemory" Sep 08 15:08:59.468065 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:59.468008 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 15:08:59.468065 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:59.468019 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeHasSufficientPID" Sep 08 15:08:59.485623 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:08:59.485600 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-159.ec2.internal\" not found" Sep 08 15:08:59.491403 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:59.491386 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 15:08:59.536724 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:59.536680 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 15:08:59.555445 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:59.555424 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal" Sep 08 15:08:59.569967 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:59.569954 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 15:08:59.570887 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:59.570875 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-159.ec2.internal" Sep 08 15:08:59.580505 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:59.580474 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 15:08:59.603070 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:59.603037 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e79f6357314a98cc42b005b66beb364.slice/crio-e11ce753438fae6a7628d70ace482d7febcdca3d4c6ed0b96a696759a7958e75 WatchSource:0}: Error finding container e11ce753438fae6a7628d70ace482d7febcdca3d4c6ed0b96a696759a7958e75: Status 404 returned error can't find the container with id e11ce753438fae6a7628d70ace482d7febcdca3d4c6ed0b96a696759a7958e75 Sep 08 15:08:59.604686 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:08:59.604665 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf326de9abe5883a85f6af224dafa6b15.slice/crio-5efe5b9c907c4091d3d62e9a88362a3fa35bb606437bf0f94ea9cf7d39bfe0a7 WatchSource:0}: Error finding container 5efe5b9c907c4091d3d62e9a88362a3fa35bb606437bf0f94ea9cf7d39bfe0a7: Status 404 returned error can't find the container with id 5efe5b9c907c4091d3d62e9a88362a3fa35bb606437bf0f94ea9cf7d39bfe0a7 Sep 08 15:08:59.608842 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:08:59.608825 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 15:09:00.109365 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.109336 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 15:09:00.234278 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.233979 2572 apiserver.go:52] "Watching apiserver" Sep 08 15:09:00.241739 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.241702 2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 08 15:09:00.244027 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.243998 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-zxbnk","openshift-image-registry/node-ca-wjg4s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal","openshift-multus/multus-additional-cni-plugins-2mhvv","openshift-multus/multus-pxfh4","openshift-network-diagnostics/network-check-target-mpw6h","kube-system/global-pull-secret-syncer-9b44p","kube-system/kube-apiserver-proxy-ip-10-0-136-159.ec2.internal","openshift-cluster-node-tuning-operator/tuned-mmfzq","openshift-multus/network-metrics-daemon-n5x8x","openshift-network-operator/iptables-alerter-57pdq","openshift-ovn-kubernetes/ovnkube-node-vkmfv","kube-system/konnectivity-agent-j97ml","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4"] Sep 08 15:09:00.246673 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.246653 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zxbnk" Sep 08 15:09:00.248998 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.248976 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wjg4s" Sep 08 15:09:00.249337 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.249314 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cbxvm\"" Sep 08 15:09:00.249880 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.249864 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 08 15:09:00.249880 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.249873 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 08 15:09:00.251561 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.251374 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:00.251561 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:00.251432 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b44p" podUID="f755fdd9-db6c-43ec-bd91-5d0d861aae73" Sep 08 15:09:00.252559 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.252531 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wkfsh\"" Sep 08 15:09:00.253189 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.253157 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 08 15:09:00.253189 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.253188 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 08 15:09:00.253189 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.253156 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 08 15:09:00.254098 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.254077 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.254189 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.254134 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 08 15:09:00.256914 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.256669 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.259078 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.259059 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:00.259185 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:00.259126 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpw6h" podUID="ca0287ff-6de3-46ba-b97f-259d5477f07c" Sep 08 15:09:00.260640 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.260338 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7trxz\"" Sep 08 15:09:00.260640 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.260408 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 08 15:09:00.260640 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.260428 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 08 15:09:00.261539 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.261241 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 08 15:09:00.261539 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.261308 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 08 15:09:00.261539 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.261318 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 08 15:09:00.261539 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.261414 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6c75w\"" Sep 08 15:09:00.261942 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.261920 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.263127 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.262967 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 08 15:09:00.265854 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.265658 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 08 15:09:00.265854 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.265671 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cq9bf\"" Sep 08 15:09:00.265854 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.265760 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 08 15:09:00.266050 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.265917 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:00.266050 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:00.265978 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:09:00.267329 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267310 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-system-cni-dir\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.267423 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267342 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-host-var-lib-cni-multus\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.267423 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267368 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-host-var-lib-kubelet\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.267423 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267392 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/90fade35-fbac-48dd-834c-919ef54da378-serviceca\") pod \"node-ca-wjg4s\" (UID: \"90fade35-fbac-48dd-834c-919ef54da378\") " pod="openshift-image-registry/node-ca-wjg4s" Sep 08 15:09:00.267542 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267436 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f755fdd9-db6c-43ec-bd91-5d0d861aae73-dbus\") pod \"global-pull-secret-syncer-9b44p\" (UID: \"f755fdd9-db6c-43ec-bd91-5d0d861aae73\") " pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:00.267542 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267476 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-cni-binary-copy\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.267542 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267496 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-multus-conf-dir\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.267542 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267516 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-system-cni-dir\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.267542 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267537 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.267779 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267589 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z22cp\" (UniqueName: \"kubernetes.io/projected/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-kube-api-access-z22cp\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.267779 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267617 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-multus-cni-dir\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.267779 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267640 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-host-run-k8s-cni-cncf-io\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.267779 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267660 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-host-var-lib-cni-bin\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.267779 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267675 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-host-run-multus-certs\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.267779 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267720 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.267779 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267736 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-host-run-netns\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.267779 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267752 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-hostroot\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.268136 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267785 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grdw2\" (UniqueName: \"kubernetes.io/projected/1dfce943-0703-4e7d-b479-a7616aa67993-kube-api-access-grdw2\") pod \"node-resolver-zxbnk\" (UID: \"1dfce943-0703-4e7d-b479-a7616aa67993\") " pod="openshift-dns/node-resolver-zxbnk" Sep 08 15:09:00.268136 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267811 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nsrq\" (UniqueName: \"kubernetes.io/projected/90fade35-fbac-48dd-834c-919ef54da378-kube-api-access-5nsrq\") pod \"node-ca-wjg4s\" (UID: \"90fade35-fbac-48dd-834c-919ef54da378\") " pod="openshift-image-registry/node-ca-wjg4s" Sep 08 15:09:00.268136 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267835 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-cnibin\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.268136 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267860 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.268136 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267883 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7662e1ba-5c64-4348-844c-1f5fd285ce6f-multus-daemon-config\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.268136 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267909 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7662e1ba-5c64-4348-844c-1f5fd285ce6f-cni-binary-copy\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.268136 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267932 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-multus-socket-dir-parent\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.268136 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267955 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqvp9\" (UniqueName: \"kubernetes.io/projected/7662e1ba-5c64-4348-844c-1f5fd285ce6f-kube-api-access-nqvp9\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.268136 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.267981 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-cnibin\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.268136 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.268004 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1dfce943-0703-4e7d-b479-a7616aa67993-hosts-file\") pod \"node-resolver-zxbnk\" (UID: \"1dfce943-0703-4e7d-b479-a7616aa67993\") " pod="openshift-dns/node-resolver-zxbnk" Sep 08 15:09:00.268136 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.268033 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret\") pod \"global-pull-secret-syncer-9b44p\" (UID: \"f755fdd9-db6c-43ec-bd91-5d0d861aae73\") " pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:00.268136 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.268070 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-etc-kubernetes\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.268136 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.268093 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dfce943-0703-4e7d-b479-a7616aa67993-tmp-dir\") pod \"node-resolver-zxbnk\" (UID: \"1dfce943-0703-4e7d-b479-a7616aa67993\") " pod="openshift-dns/node-resolver-zxbnk" Sep 08 15:09:00.268136 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.268118 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90fade35-fbac-48dd-834c-919ef54da378-host\") pod \"node-ca-wjg4s\" (UID: \"90fade35-fbac-48dd-834c-919ef54da378\") " pod="openshift-image-registry/node-ca-wjg4s" Sep 08 15:09:00.268136 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.268141 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f755fdd9-db6c-43ec-bd91-5d0d861aae73-kubelet-config\") pod \"global-pull-secret-syncer-9b44p\" (UID: \"f755fdd9-db6c-43ec-bd91-5d0d861aae73\") " pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:00.268817 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.268282 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-57pdq" Sep 08 15:09:00.270985 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.270964 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 08 15:09:00.271123 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.271085 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 08 15:09:00.271492 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.271342 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.271492 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.271367 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8xm5w\"" Sep 08 15:09:00.271492 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.271460 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 08 15:09:00.274877 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.273787 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j97ml" Sep 08 15:09:00.274877 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.274557 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 08 15:09:00.275038 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.274962 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 08 15:09:00.275098 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.275055 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 08 15:09:00.275251 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.275231 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 15:09:00.275330 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.275256 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 08 15:09:00.275424 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.275402 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-shrg8\"" Sep 08 15:09:00.275510 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.275494 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 08 15:09:00.276178 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.276018 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 08 15:09:00.276178 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.276051 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gs57x\"" Sep 08 15:09:00.276178 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.276073 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 08 15:09:00.276360 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.276279 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.276817 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.276799 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 08 15:09:00.279428 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.279404 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 08 15:09:00.279428 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.279419 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 08 15:09:00.279610 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.279472 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 08 15:09:00.280194 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.280058 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-thd27\"" Sep 08 15:09:00.320542 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.320489 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5xzwc" Sep 08 15:09:00.329812 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.329791 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5xzwc" Sep 08 15:09:00.357025 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.357002 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 08 15:09:00.368354 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368329 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-run-ovn\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.368442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368421 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4293ce5-30ea-4210-9b84-ec6539d992cb-ovnkube-config\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.368484 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368445 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4293ce5-30ea-4210-9b84-ec6539d992cb-ovn-node-metrics-cert\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.368484 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368461 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-etc-modprobe-d\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.368600 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368516 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0bfdcd24-b0c7-444b-af1a-7d9370b714b4-host-slash\") pod \"iptables-alerter-57pdq\" (UID: \"0bfdcd24-b0c7-444b-af1a-7d9370b714b4\") " pod="openshift-network-operator/iptables-alerter-57pdq" Sep 08 15:09:00.368600 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368550 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-system-cni-dir\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.368600 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368569 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z22cp\" (UniqueName: \"kubernetes.io/projected/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-kube-api-access-z22cp\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.368600 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368591 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-multus-cni-dir\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.368801 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368624 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-host-run-k8s-cni-cncf-io\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.368801 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368650 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4293ce5-30ea-4210-9b84-ec6539d992cb-env-overrides\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.368801 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368659 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-system-cni-dir\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.368801 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368676 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nsrq\" (UniqueName: \"kubernetes.io/projected/90fade35-fbac-48dd-834c-919ef54da378-kube-api-access-5nsrq\") pod \"node-ca-wjg4s\" (UID: \"90fade35-fbac-48dd-834c-919ef54da378\") " pod="openshift-image-registry/node-ca-wjg4s" Sep 08 15:09:00.368801 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368717 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-systemd-units\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.368801 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368746 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-slash\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.368801 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368773 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-run-netns\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.369131 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368804 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-cnibin\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.369131 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368830 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.369131 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368857 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4293ce5-30ea-4210-9b84-ec6539d992cb-ovnkube-script-lib\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.369131 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368897 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-multus-cni-dir\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.369131 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368907 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-host-run-k8s-cni-cncf-io\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.369131 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368931 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-cnibin\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.369131 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368958 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-etc-sysconfig\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.369131 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.368986 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-etc-systemd\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.369131 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369010 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-host\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.369131 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369051 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/257cd212-bcaa-405f-8682-6c197ca64024-etc-tuned\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.369131 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369098 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0bfdcd24-b0c7-444b-af1a-7d9370b714b4-iptables-alerter-script\") pod \"iptables-alerter-57pdq\" (UID: \"0bfdcd24-b0c7-444b-af1a-7d9370b714b4\") " pod="openshift-network-operator/iptables-alerter-57pdq" Sep 08 15:09:00.369688 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369146 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-multus-socket-dir-parent\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.369688 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369194 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqvp9\" (UniqueName: \"kubernetes.io/projected/7662e1ba-5c64-4348-844c-1f5fd285ce6f-kube-api-access-nqvp9\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.369688 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369272 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-multus-socket-dir-parent\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.369688 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369308 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-etc-sysctl-d\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.369688 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369342 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-cnibin\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.369688 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369369 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret\") pod \"global-pull-secret-syncer-9b44p\" (UID: \"f755fdd9-db6c-43ec-bd91-5d0d861aae73\") " pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:00.369688 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369394 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs\") pod \"network-metrics-daemon-n5x8x\" (UID: \"1dafedb8-50fc-4eff-a5a8-ee1be3e3f751\") " pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:00.369688 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369422 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqcg7\" (UniqueName: \"kubernetes.io/projected/0bfdcd24-b0c7-444b-af1a-7d9370b714b4-kube-api-access-jqcg7\") pod \"iptables-alerter-57pdq\" (UID: \"0bfdcd24-b0c7-444b-af1a-7d9370b714b4\") " pod="openshift-network-operator/iptables-alerter-57pdq" Sep 08 15:09:00.369688 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369446 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-cnibin\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.369688 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369448 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-etc-kubernetes\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.369688 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369518 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90fade35-fbac-48dd-834c-919ef54da378-host\") pod \"node-ca-wjg4s\" (UID: \"90fade35-fbac-48dd-834c-919ef54da378\") " pod="openshift-image-registry/node-ca-wjg4s" Sep 08 15:09:00.369688 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:00.369518 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 15:09:00.369688 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369566 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90fade35-fbac-48dd-834c-919ef54da378-host\") pod \"node-ca-wjg4s\" (UID: \"90fade35-fbac-48dd-834c-919ef54da378\") " pod="openshift-image-registry/node-ca-wjg4s" Sep 08 15:09:00.369688 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:00.369604 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret podName:f755fdd9-db6c-43ec-bd91-5d0d861aae73 nodeName:}" failed. No retries permitted until 2026-09-08 15:09:00.869580605 +0000 UTC m=+3.021759166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret") pod "global-pull-secret-syncer-9b44p" (UID: "f755fdd9-db6c-43ec-bd91-5d0d861aae73") : object "kube-system"/"original-pull-secret" not registered Sep 08 15:09:00.369688 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369614 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-etc-kubernetes\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.369688 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369643 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f755fdd9-db6c-43ec-bd91-5d0d861aae73-kubelet-config\") pod \"global-pull-secret-syncer-9b44p\" (UID: \"f755fdd9-db6c-43ec-bd91-5d0d861aae73\") " pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:00.369688 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369578 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f755fdd9-db6c-43ec-bd91-5d0d861aae73-kubelet-config\") pod \"global-pull-secret-syncer-9b44p\" (UID: \"f755fdd9-db6c-43ec-bd91-5d0d861aae73\") " pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:00.370461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369695 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk4gf\" (UniqueName: \"kubernetes.io/projected/e4293ce5-30ea-4210-9b84-ec6539d992cb-kube-api-access-tk4gf\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.370461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369724 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgfv7\" (UniqueName: \"kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7\") pod \"network-check-target-mpw6h\" (UID: \"ca0287ff-6de3-46ba-b97f-259d5477f07c\") " pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:00.370461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369751 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-var-lib-kubelet\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.370461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369776 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c83abab5-7e60-48e3-b075-bbd45a2a68ed-agent-certs\") pod \"konnectivity-agent-j97ml\" (UID: \"c83abab5-7e60-48e3-b075-bbd45a2a68ed\") " pod="kube-system/konnectivity-agent-j97ml" Sep 08 15:09:00.370461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369806 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-system-cni-dir\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.370461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369870 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-host-var-lib-cni-multus\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.370461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369903 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-kubelet\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.370461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369929 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-cni-netd\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.370461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369947 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-cni-binary-copy\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.370461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369956 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.370461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369962 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-multus-conf-dir\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.370461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369986 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-multus-conf-dir\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.370461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.369998 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-system-cni-dir\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.370461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370015 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.370461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370079 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-host-var-lib-cni-multus\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.370461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370113 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-host-var-lib-cni-bin\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.370461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370154 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-host-run-multus-certs\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.371220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370204 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-host-var-lib-cni-bin\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.371220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370254 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grdw2\" (UniqueName: \"kubernetes.io/projected/1dfce943-0703-4e7d-b479-a7616aa67993-kube-api-access-grdw2\") pod \"node-resolver-zxbnk\" (UID: \"1dfce943-0703-4e7d-b479-a7616aa67993\") " pod="openshift-dns/node-resolver-zxbnk" Sep 08 15:09:00.371220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370270 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-host-run-multus-certs\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.371220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370356 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.371220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370389 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-host-run-netns\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.371220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370416 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-hostroot\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.371220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370445 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.371220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370469 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/257cd212-bcaa-405f-8682-6c197ca64024-tmp\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.371220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370481 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-cni-binary-copy\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.371220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370495 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7662e1ba-5c64-4348-844c-1f5fd285ce6f-multus-daemon-config\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.371220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370543 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s95s4\" (UniqueName: \"kubernetes.io/projected/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-kube-api-access-s95s4\") pod \"network-metrics-daemon-n5x8x\" (UID: \"1dafedb8-50fc-4eff-a5a8-ee1be3e3f751\") " pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:00.371220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370570 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-etc-openvswitch\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.371220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370598 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-host-run-netns\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.371220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370602 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-log-socket\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.371220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370608 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.371220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370652 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c83abab5-7e60-48e3-b075-bbd45a2a68ed-konnectivity-ca\") pod \"konnectivity-agent-j97ml\" (UID: \"c83abab5-7e60-48e3-b075-bbd45a2a68ed\") " pod="kube-system/konnectivity-agent-j97ml" Sep 08 15:09:00.371220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370698 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-hostroot\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.372002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370706 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddc06cac-5923-47a1-81c1-eef7c8aefb24-sys-fs\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.372002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370761 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7662e1ba-5c64-4348-844c-1f5fd285ce6f-cni-binary-copy\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.372002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370804 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-etc-kubernetes\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.372002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370902 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-etc-sysctl-conf\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.372002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370915 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.372002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370936 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-lib-modules\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.372002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370964 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pqh5\" (UniqueName: \"kubernetes.io/projected/257cd212-bcaa-405f-8682-6c197ca64024-kube-api-access-8pqh5\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.372002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.370999 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddc06cac-5923-47a1-81c1-eef7c8aefb24-socket-dir\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.372002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371032 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddc06cac-5923-47a1-81c1-eef7c8aefb24-device-dir\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.372002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371067 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1dfce943-0703-4e7d-b479-a7616aa67993-hosts-file\") pod \"node-resolver-zxbnk\" (UID: \"1dfce943-0703-4e7d-b479-a7616aa67993\") " pod="openshift-dns/node-resolver-zxbnk" Sep 08 15:09:00.372002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371099 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-var-lib-openvswitch\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.372002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371107 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7662e1ba-5c64-4348-844c-1f5fd285ce6f-multus-daemon-config\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.372002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371124 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-cni-bin\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.372002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371169 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1dfce943-0703-4e7d-b479-a7616aa67993-hosts-file\") pod \"node-resolver-zxbnk\" (UID: \"1dfce943-0703-4e7d-b479-a7616aa67993\") " pod="openshift-dns/node-resolver-zxbnk" Sep 08 15:09:00.372002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371169 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-run\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.372002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371200 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dfce943-0703-4e7d-b479-a7616aa67993-tmp-dir\") pod \"node-resolver-zxbnk\" (UID: \"1dfce943-0703-4e7d-b479-a7616aa67993\") " pod="openshift-dns/node-resolver-zxbnk" Sep 08 15:09:00.372002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371226 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-run-systemd\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.372523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371240 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7662e1ba-5c64-4348-844c-1f5fd285ce6f-cni-binary-copy\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.372523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371250 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-run-openvswitch\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.372523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371276 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-sys\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.372523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371300 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddc06cac-5923-47a1-81c1-eef7c8aefb24-kubelet-dir\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.372523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371340 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddc06cac-5923-47a1-81c1-eef7c8aefb24-etc-selinux\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.372523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371372 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttblt\" (UniqueName: \"kubernetes.io/projected/ddc06cac-5923-47a1-81c1-eef7c8aefb24-kube-api-access-ttblt\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.372523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371407 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-host-var-lib-kubelet\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.372523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371436 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/90fade35-fbac-48dd-834c-919ef54da378-serviceca\") pod \"node-ca-wjg4s\" (UID: \"90fade35-fbac-48dd-834c-919ef54da378\") " pod="openshift-image-registry/node-ca-wjg4s" Sep 08 15:09:00.372523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371458 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7662e1ba-5c64-4348-844c-1f5fd285ce6f-host-var-lib-kubelet\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.372523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371461 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dfce943-0703-4e7d-b479-a7616aa67993-tmp-dir\") pod \"node-resolver-zxbnk\" (UID: \"1dfce943-0703-4e7d-b479-a7616aa67993\") " pod="openshift-dns/node-resolver-zxbnk" Sep 08 15:09:00.372523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371463 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f755fdd9-db6c-43ec-bd91-5d0d861aae73-dbus\") pod \"global-pull-secret-syncer-9b44p\" (UID: \"f755fdd9-db6c-43ec-bd91-5d0d861aae73\") " pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:00.372523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371530 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-node-log\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.372523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371549 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.372523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371548 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f755fdd9-db6c-43ec-bd91-5d0d861aae73-dbus\") pod \"global-pull-secret-syncer-9b44p\" (UID: \"f755fdd9-db6c-43ec-bd91-5d0d861aae73\") " pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:00.372523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371568 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddc06cac-5923-47a1-81c1-eef7c8aefb24-registration-dir\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.372523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.371894 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/90fade35-fbac-48dd-834c-919ef54da378-serviceca\") pod \"node-ca-wjg4s\" (UID: \"90fade35-fbac-48dd-834c-919ef54da378\") " pod="openshift-image-registry/node-ca-wjg4s" Sep 08 15:09:00.373291 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.373044 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal" event={"ID":"1e79f6357314a98cc42b005b66beb364","Type":"ContainerStarted","Data":"e11ce753438fae6a7628d70ace482d7febcdca3d4c6ed0b96a696759a7958e75"} Sep 08 15:09:00.374680 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.374654 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-159.ec2.internal" event={"ID":"f326de9abe5883a85f6af224dafa6b15","Type":"ContainerStarted","Data":"5efe5b9c907c4091d3d62e9a88362a3fa35bb606437bf0f94ea9cf7d39bfe0a7"} Sep 08 15:09:00.377198 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.377173 2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 08 15:09:00.384154 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.383725 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqvp9\" (UniqueName: \"kubernetes.io/projected/7662e1ba-5c64-4348-844c-1f5fd285ce6f-kube-api-access-nqvp9\") pod \"multus-pxfh4\" (UID: \"7662e1ba-5c64-4348-844c-1f5fd285ce6f\") " pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.384154 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.383732 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z22cp\" (UniqueName: \"kubernetes.io/projected/4d45089b-2c7d-45d7-883b-417a0bcb9c1b-kube-api-access-z22cp\") pod \"multus-additional-cni-plugins-2mhvv\" (UID: \"4d45089b-2c7d-45d7-883b-417a0bcb9c1b\") " pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.384154 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.384107 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nsrq\" (UniqueName: \"kubernetes.io/projected/90fade35-fbac-48dd-834c-919ef54da378-kube-api-access-5nsrq\") pod \"node-ca-wjg4s\" (UID: \"90fade35-fbac-48dd-834c-919ef54da378\") " pod="openshift-image-registry/node-ca-wjg4s" Sep 08 15:09:00.387954 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.387929 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grdw2\" (UniqueName: \"kubernetes.io/projected/1dfce943-0703-4e7d-b479-a7616aa67993-kube-api-access-grdw2\") pod \"node-resolver-zxbnk\" (UID: \"1dfce943-0703-4e7d-b479-a7616aa67993\") " pod="openshift-dns/node-resolver-zxbnk" Sep 08 15:09:00.472691 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.472653 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4gf\" (UniqueName: \"kubernetes.io/projected/e4293ce5-30ea-4210-9b84-ec6539d992cb-kube-api-access-tk4gf\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.472876 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.472695 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfv7\" (UniqueName: \"kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7\") pod \"network-check-target-mpw6h\" (UID: \"ca0287ff-6de3-46ba-b97f-259d5477f07c\") " pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:00.472953 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.472895 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-var-lib-kubelet\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.472953 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.472926 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c83abab5-7e60-48e3-b075-bbd45a2a68ed-agent-certs\") pod \"konnectivity-agent-j97ml\" (UID: \"c83abab5-7e60-48e3-b075-bbd45a2a68ed\") " pod="kube-system/konnectivity-agent-j97ml" Sep 08 15:09:00.473060 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.472954 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-kubelet\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.473060 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.472980 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-cni-netd\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.473060 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473020 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.473060 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473046 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/257cd212-bcaa-405f-8682-6c197ca64024-tmp\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.473273 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473072 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s95s4\" (UniqueName: \"kubernetes.io/projected/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-kube-api-access-s95s4\") pod \"network-metrics-daemon-n5x8x\" (UID: \"1dafedb8-50fc-4eff-a5a8-ee1be3e3f751\") " pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:00.473273 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473099 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-etc-openvswitch\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.473273 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473123 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-log-socket\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.473273 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473149 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c83abab5-7e60-48e3-b075-bbd45a2a68ed-konnectivity-ca\") pod \"konnectivity-agent-j97ml\" (UID: \"c83abab5-7e60-48e3-b075-bbd45a2a68ed\") " pod="kube-system/konnectivity-agent-j97ml" Sep 08 15:09:00.473273 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473174 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddc06cac-5923-47a1-81c1-eef7c8aefb24-sys-fs\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.473273 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473197 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.473273 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473214 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-etc-kubernetes\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.473661 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473286 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-etc-kubernetes\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.473661 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473299 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-etc-sysctl-conf\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.473661 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473336 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-log-socket\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.473661 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473206 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-var-lib-kubelet\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.473661 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473464 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-etc-openvswitch\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.473661 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473582 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-kubelet\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.473661 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473610 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddc06cac-5923-47a1-81c1-eef7c8aefb24-sys-fs\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.473661 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473618 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-etc-sysctl-conf\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.473661 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473654 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-cni-netd\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.473661 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473644 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-lib-modules\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.474169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473681 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pqh5\" (UniqueName: \"kubernetes.io/projected/257cd212-bcaa-405f-8682-6c197ca64024-kube-api-access-8pqh5\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.474169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473697 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddc06cac-5923-47a1-81c1-eef7c8aefb24-socket-dir\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.474169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473726 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddc06cac-5923-47a1-81c1-eef7c8aefb24-device-dir\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.474169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473742 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-lib-modules\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.474169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473778 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddc06cac-5923-47a1-81c1-eef7c8aefb24-device-dir\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.474169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473828 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddc06cac-5923-47a1-81c1-eef7c8aefb24-socket-dir\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.474169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473887 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-var-lib-openvswitch\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.474169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473897 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c83abab5-7e60-48e3-b075-bbd45a2a68ed-konnectivity-ca\") pod \"konnectivity-agent-j97ml\" (UID: \"c83abab5-7e60-48e3-b075-bbd45a2a68ed\") " pod="kube-system/konnectivity-agent-j97ml" Sep 08 15:09:00.474169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473929 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-cni-bin\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.474169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473953 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-run\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.474169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473958 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-var-lib-openvswitch\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.474169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473981 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-run-systemd\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.474169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.473986 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-cni-bin\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.474169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474007 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-run-openvswitch\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.474169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474020 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-run-systemd\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.474169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474019 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-run\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.474169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474030 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-sys\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.475084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474055 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddc06cac-5923-47a1-81c1-eef7c8aefb24-kubelet-dir\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.475084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474058 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-run-openvswitch\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.475084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474080 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddc06cac-5923-47a1-81c1-eef7c8aefb24-etc-selinux\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.475084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474107 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttblt\" (UniqueName: \"kubernetes.io/projected/ddc06cac-5923-47a1-81c1-eef7c8aefb24-kube-api-access-ttblt\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.475084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474117 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-sys\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.475084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474133 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddc06cac-5923-47a1-81c1-eef7c8aefb24-kubelet-dir\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.475084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474133 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-node-log\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.475084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474162 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.475084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474174 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-node-log\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.475084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474180 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddc06cac-5923-47a1-81c1-eef7c8aefb24-registration-dir\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.475084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474202 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddc06cac-5923-47a1-81c1-eef7c8aefb24-etc-selinux\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.475084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474208 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-run-ovn\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.475084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474206 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddc06cac-5923-47a1-81c1-eef7c8aefb24-registration-dir\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.475084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474224 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.475084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474234 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4293ce5-30ea-4210-9b84-ec6539d992cb-ovnkube-config\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.475084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474250 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-run-ovn\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.475084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474260 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4293ce5-30ea-4210-9b84-ec6539d992cb-ovn-node-metrics-cert\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.475976 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474303 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-etc-modprobe-d\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.475976 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474327 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0bfdcd24-b0c7-444b-af1a-7d9370b714b4-host-slash\") pod \"iptables-alerter-57pdq\" (UID: \"0bfdcd24-b0c7-444b-af1a-7d9370b714b4\") " pod="openshift-network-operator/iptables-alerter-57pdq" Sep 08 15:09:00.475976 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474356 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4293ce5-30ea-4210-9b84-ec6539d992cb-env-overrides\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.475976 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474374 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-systemd-units\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.475976 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474388 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-slash\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.475976 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474403 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-run-netns\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.475976 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474419 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4293ce5-30ea-4210-9b84-ec6539d992cb-ovnkube-script-lib\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.475976 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474458 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-etc-sysconfig\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.475976 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474483 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-etc-systemd\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.475976 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474508 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-host\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.475976 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474532 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/257cd212-bcaa-405f-8682-6c197ca64024-etc-tuned\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.475976 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474577 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0bfdcd24-b0c7-444b-af1a-7d9370b714b4-iptables-alerter-script\") pod \"iptables-alerter-57pdq\" (UID: \"0bfdcd24-b0c7-444b-af1a-7d9370b714b4\") " pod="openshift-network-operator/iptables-alerter-57pdq" Sep 08 15:09:00.475976 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474607 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-etc-sysctl-d\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.475976 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474648 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs\") pod \"network-metrics-daemon-n5x8x\" (UID: \"1dafedb8-50fc-4eff-a5a8-ee1be3e3f751\") " pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:00.475976 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474671 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqcg7\" (UniqueName: \"kubernetes.io/projected/0bfdcd24-b0c7-444b-af1a-7d9370b714b4-kube-api-access-jqcg7\") pod \"iptables-alerter-57pdq\" (UID: \"0bfdcd24-b0c7-444b-af1a-7d9370b714b4\") " pod="openshift-network-operator/iptables-alerter-57pdq" Sep 08 15:09:00.475976 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474792 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4293ce5-30ea-4210-9b84-ec6539d992cb-ovnkube-config\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.475976 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474874 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4293ce5-30ea-4210-9b84-ec6539d992cb-env-overrides\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.476676 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474895 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-etc-modprobe-d\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.476676 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474945 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-slash\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.476676 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474945 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0bfdcd24-b0c7-444b-af1a-7d9370b714b4-host-slash\") pod \"iptables-alerter-57pdq\" (UID: \"0bfdcd24-b0c7-444b-af1a-7d9370b714b4\") " pod="openshift-network-operator/iptables-alerter-57pdq" Sep 08 15:09:00.476676 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474980 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-systemd-units\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.476676 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.474988 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4293ce5-30ea-4210-9b84-ec6539d992cb-host-run-netns\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.476676 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.475037 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-host\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.476676 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.475276 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-etc-sysconfig\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.476676 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.475378 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-etc-sysctl-d\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.476676 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.475527 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4293ce5-30ea-4210-9b84-ec6539d992cb-ovnkube-script-lib\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.476676 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.475622 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/257cd212-bcaa-405f-8682-6c197ca64024-etc-systemd\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.476676 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:00.475706 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 15:09:00.476676 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:00.475777 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs podName:1dafedb8-50fc-4eff-a5a8-ee1be3e3f751 nodeName:}" failed. No retries permitted until 2026-09-08 15:09:00.975739786 +0000 UTC m=+3.127918350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs") pod "network-metrics-daemon-n5x8x" (UID: "1dafedb8-50fc-4eff-a5a8-ee1be3e3f751") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 15:09:00.476676 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.475865 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0bfdcd24-b0c7-444b-af1a-7d9370b714b4-iptables-alerter-script\") pod \"iptables-alerter-57pdq\" (UID: \"0bfdcd24-b0c7-444b-af1a-7d9370b714b4\") " pod="openshift-network-operator/iptables-alerter-57pdq" Sep 08 15:09:00.484705 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.480016 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4293ce5-30ea-4210-9b84-ec6539d992cb-ovn-node-metrics-cert\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.484705 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.480076 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c83abab5-7e60-48e3-b075-bbd45a2a68ed-agent-certs\") pod \"konnectivity-agent-j97ml\" (UID: \"c83abab5-7e60-48e3-b075-bbd45a2a68ed\") " pod="kube-system/konnectivity-agent-j97ml" Sep 08 15:09:00.484705 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:00.480187 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 15:09:00.484705 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:00.484686 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 15:09:00.484705 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:00.484704 2572 projected.go:196] Error preparing data for projected volume kube-api-access-hgfv7 for pod openshift-network-diagnostics/network-check-target-mpw6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 15:09:00.485022 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.480192 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/257cd212-bcaa-405f-8682-6c197ca64024-tmp\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.485022 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.481595 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/257cd212-bcaa-405f-8682-6c197ca64024-etc-tuned\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.485022 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:00.484770 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7 podName:ca0287ff-6de3-46ba-b97f-259d5477f07c nodeName:}" failed. No retries permitted until 2026-09-08 15:09:00.984750915 +0000 UTC m=+3.136929490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hgfv7" (UniqueName: "kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7") pod "network-check-target-mpw6h" (UID: "ca0287ff-6de3-46ba-b97f-259d5477f07c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 15:09:00.485022 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.484273 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk4gf\" (UniqueName: \"kubernetes.io/projected/e4293ce5-30ea-4210-9b84-ec6539d992cb-kube-api-access-tk4gf\") pod \"ovnkube-node-vkmfv\" (UID: \"e4293ce5-30ea-4210-9b84-ec6539d992cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.486291 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.486219 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s95s4\" (UniqueName: \"kubernetes.io/projected/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-kube-api-access-s95s4\") pod \"network-metrics-daemon-n5x8x\" (UID: \"1dafedb8-50fc-4eff-a5a8-ee1be3e3f751\") " pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:00.486791 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.486761 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttblt\" (UniqueName: \"kubernetes.io/projected/ddc06cac-5923-47a1-81c1-eef7c8aefb24-kube-api-access-ttblt\") pod \"aws-ebs-csi-driver-node-swqb4\" (UID: \"ddc06cac-5923-47a1-81c1-eef7c8aefb24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.487111 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.487089 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pqh5\" (UniqueName: \"kubernetes.io/projected/257cd212-bcaa-405f-8682-6c197ca64024-kube-api-access-8pqh5\") pod \"tuned-mmfzq\" (UID: \"257cd212-bcaa-405f-8682-6c197ca64024\") " pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.487192 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.487146 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqcg7\" (UniqueName: \"kubernetes.io/projected/0bfdcd24-b0c7-444b-af1a-7d9370b714b4-kube-api-access-jqcg7\") pod \"iptables-alerter-57pdq\" (UID: \"0bfdcd24-b0c7-444b-af1a-7d9370b714b4\") " pod="openshift-network-operator/iptables-alerter-57pdq" Sep 08 15:09:00.558776 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.558753 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zxbnk" Sep 08 15:09:00.568143 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.568113 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wjg4s" Sep 08 15:09:00.571309 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:09:00.571279 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dfce943_0703_4e7d_b479_a7616aa67993.slice/crio-e28955ed0699fc07103620c5474c579ff3d03a43b3e031e47a3e218cd6e5594d WatchSource:0}: Error finding container e28955ed0699fc07103620c5474c579ff3d03a43b3e031e47a3e218cd6e5594d: Status 404 returned error can't find the container with id e28955ed0699fc07103620c5474c579ff3d03a43b3e031e47a3e218cd6e5594d Sep 08 15:09:00.577359 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.577335 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2mhvv" Sep 08 15:09:00.577932 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:09:00.577899 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fade35_fbac_48dd_834c_919ef54da378.slice/crio-76ce9f62daa1aba8b9a1dcb158c86dbcaf94e241de2d313b7fcd935933e30c77 WatchSource:0}: Error finding container 76ce9f62daa1aba8b9a1dcb158c86dbcaf94e241de2d313b7fcd935933e30c77: Status 404 returned error can't find the container with id 76ce9f62daa1aba8b9a1dcb158c86dbcaf94e241de2d313b7fcd935933e30c77 Sep 08 15:09:00.584728 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.584560 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pxfh4" Sep 08 15:09:00.587315 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:09:00.587204 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d45089b_2c7d_45d7_883b_417a0bcb9c1b.slice/crio-cf416c6bb1cbec2ddd832c9ce5f1d0b784e8dd65d897f8ad04f2cd0493f1b3b6 WatchSource:0}: Error finding container cf416c6bb1cbec2ddd832c9ce5f1d0b784e8dd65d897f8ad04f2cd0493f1b3b6: Status 404 returned error can't find the container with id cf416c6bb1cbec2ddd832c9ce5f1d0b784e8dd65d897f8ad04f2cd0493f1b3b6 Sep 08 15:09:00.593014 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.592836 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" Sep 08 15:09:00.600045 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.600025 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-57pdq" Sep 08 15:09:00.607255 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.607229 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:00.614338 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.614318 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j97ml" Sep 08 15:09:00.620828 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.620674 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" Sep 08 15:09:00.877925 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.877888 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret\") pod \"global-pull-secret-syncer-9b44p\" (UID: \"f755fdd9-db6c-43ec-bd91-5d0d861aae73\") " pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:00.878112 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:00.878075 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 15:09:00.878176 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:00.878147 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret podName:f755fdd9-db6c-43ec-bd91-5d0d861aae73 nodeName:}" failed. No retries permitted until 2026-09-08 15:09:01.878129595 +0000 UTC m=+4.030308177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret") pod "global-pull-secret-syncer-9b44p" (UID: "f755fdd9-db6c-43ec-bd91-5d0d861aae73") : object "kube-system"/"original-pull-secret" not registered Sep 08 15:09:00.978880 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:00.978847 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs\") pod \"network-metrics-daemon-n5x8x\" (UID: \"1dafedb8-50fc-4eff-a5a8-ee1be3e3f751\") " pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:00.979034 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:00.978992 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 15:09:00.979096 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:00.979064 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs podName:1dafedb8-50fc-4eff-a5a8-ee1be3e3f751 nodeName:}" failed. No retries permitted until 2026-09-08 15:09:01.979037876 +0000 UTC m=+4.131216443 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs") pod "network-metrics-daemon-n5x8x" (UID: "1dafedb8-50fc-4eff-a5a8-ee1be3e3f751") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 15:09:01.079927 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:01.079881 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfv7\" (UniqueName: \"kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7\") pod \"network-check-target-mpw6h\" (UID: \"ca0287ff-6de3-46ba-b97f-259d5477f07c\") " pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:01.080071 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:01.080055 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 15:09:01.080071 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:01.080071 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 15:09:01.080144 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:01.080081 2572 projected.go:196] Error preparing data for projected volume kube-api-access-hgfv7 for pod openshift-network-diagnostics/network-check-target-mpw6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 15:09:01.080144 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:01.080132 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7 podName:ca0287ff-6de3-46ba-b97f-259d5477f07c nodeName:}" failed. No retries permitted until 2026-09-08 15:09:02.080117554 +0000 UTC m=+4.232296113 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hgfv7" (UniqueName: "kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7") pod "network-check-target-mpw6h" (UID: "ca0287ff-6de3-46ba-b97f-259d5477f07c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 15:09:01.333945 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:01.333896 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 15:04:00 +0000 UTC" deadline="2028-05-04 09:42:10.134934113 +0000 UTC" Sep 08 15:09:01.333945 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:01.333944 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14490h33m8.800995583s" Sep 08 15:09:01.380724 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:01.378775 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mhvv" event={"ID":"4d45089b-2c7d-45d7-883b-417a0bcb9c1b","Type":"ContainerStarted","Data":"cf416c6bb1cbec2ddd832c9ce5f1d0b784e8dd65d897f8ad04f2cd0493f1b3b6"} Sep 08 15:09:01.381970 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:01.381932 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wjg4s" event={"ID":"90fade35-fbac-48dd-834c-919ef54da378","Type":"ContainerStarted","Data":"76ce9f62daa1aba8b9a1dcb158c86dbcaf94e241de2d313b7fcd935933e30c77"} Sep 08 15:09:01.383218 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:01.383191 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zxbnk" event={"ID":"1dfce943-0703-4e7d-b479-a7616aa67993","Type":"ContainerStarted","Data":"e28955ed0699fc07103620c5474c579ff3d03a43b3e031e47a3e218cd6e5594d"} Sep 08 15:09:01.885677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:01.885634 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret\") pod \"global-pull-secret-syncer-9b44p\" (UID: \"f755fdd9-db6c-43ec-bd91-5d0d861aae73\") " pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:01.885859 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:01.885848 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 15:09:01.885943 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:01.885906 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret podName:f755fdd9-db6c-43ec-bd91-5d0d861aae73 nodeName:}" failed. No retries permitted until 2026-09-08 15:09:03.885892014 +0000 UTC m=+6.038070573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret") pod "global-pull-secret-syncer-9b44p" (UID: "f755fdd9-db6c-43ec-bd91-5d0d861aae73") : object "kube-system"/"original-pull-secret" not registered Sep 08 15:09:01.904266 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:09:01.904238 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc06cac_5923_47a1_81c1_eef7c8aefb24.slice/crio-d0629a74d95827e1f3d8fbd37793f7e81c099979d092370c4abd6d8bfeb4b0c3 WatchSource:0}: Error finding container d0629a74d95827e1f3d8fbd37793f7e81c099979d092370c4abd6d8bfeb4b0c3: Status 404 returned error can't find the container with id d0629a74d95827e1f3d8fbd37793f7e81c099979d092370c4abd6d8bfeb4b0c3 Sep 08 15:09:01.921390 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:09:01.921289 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7662e1ba_5c64_4348_844c_1f5fd285ce6f.slice/crio-d60a00d47eb7b0a640458d6eaf662ec8e85d637de2f3394b777aaee3031c41c2 WatchSource:0}: Error finding container d60a00d47eb7b0a640458d6eaf662ec8e85d637de2f3394b777aaee3031c41c2: Status 404 returned error can't find the container with id d60a00d47eb7b0a640458d6eaf662ec8e85d637de2f3394b777aaee3031c41c2 Sep 08 15:09:01.925079 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:09:01.925054 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83abab5_7e60_48e3_b075_bbd45a2a68ed.slice/crio-fd205dc97214c9aad68c5c4be6f0073b5e9ba32789aa76a65893c73858bccaf9 WatchSource:0}: Error finding container fd205dc97214c9aad68c5c4be6f0073b5e9ba32789aa76a65893c73858bccaf9: Status 404 returned error can't find the container with id fd205dc97214c9aad68c5c4be6f0073b5e9ba32789aa76a65893c73858bccaf9 Sep 08 15:09:01.928326 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:09:01.928299 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4293ce5_30ea_4210_9b84_ec6539d992cb.slice/crio-f0b15c63f649ba39df19e4f027eca0ec929f44afb2a08907289e463e31fc4744 WatchSource:0}: Error finding container f0b15c63f649ba39df19e4f027eca0ec929f44afb2a08907289e463e31fc4744: Status 404 returned error can't find the container with id f0b15c63f649ba39df19e4f027eca0ec929f44afb2a08907289e463e31fc4744 Sep 08 15:09:01.930305 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:09:01.930279 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257cd212_bcaa_405f_8682_6c197ca64024.slice/crio-7872351d0371dec2994e3af8f2ecf0e8f4a075e74a70bc0af3b7a438c1864b25 WatchSource:0}: Error finding container 7872351d0371dec2994e3af8f2ecf0e8f4a075e74a70bc0af3b7a438c1864b25: Status 404 returned error can't find the container with id 7872351d0371dec2994e3af8f2ecf0e8f4a075e74a70bc0af3b7a438c1864b25 Sep 08 15:09:01.946977 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:09:01.946949 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bfdcd24_b0c7_444b_af1a_7d9370b714b4.slice/crio-19947586f8d7af5184ad9162e911dd04a06933aaf355b66fa6f1b4546335f6bd WatchSource:0}: Error finding container 19947586f8d7af5184ad9162e911dd04a06933aaf355b66fa6f1b4546335f6bd: Status 404 returned error can't find the container with id 19947586f8d7af5184ad9162e911dd04a06933aaf355b66fa6f1b4546335f6bd Sep 08 15:09:01.986683 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:01.986654 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs\") pod \"network-metrics-daemon-n5x8x\" (UID: \"1dafedb8-50fc-4eff-a5a8-ee1be3e3f751\") " pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:01.986821 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:01.986806 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 15:09:01.986895 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:01.986865 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs podName:1dafedb8-50fc-4eff-a5a8-ee1be3e3f751 nodeName:}" failed. No retries permitted until 2026-09-08 15:09:03.986846215 +0000 UTC m=+6.139024777 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs") pod "network-metrics-daemon-n5x8x" (UID: "1dafedb8-50fc-4eff-a5a8-ee1be3e3f751") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 15:09:02.087776 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:02.087745 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfv7\" (UniqueName: \"kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7\") pod \"network-check-target-mpw6h\" (UID: \"ca0287ff-6de3-46ba-b97f-259d5477f07c\") " pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:02.087919 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:02.087904 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 15:09:02.087980 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:02.087923 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 15:09:02.087980 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:02.087936 2572 projected.go:196] Error preparing data for projected volume kube-api-access-hgfv7 for pod openshift-network-diagnostics/network-check-target-mpw6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 15:09:02.088088 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:02.087990 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7 podName:ca0287ff-6de3-46ba-b97f-259d5477f07c nodeName:}" failed. No retries permitted until 2026-09-08 15:09:04.087971722 +0000 UTC m=+6.240150284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hgfv7" (UniqueName: "kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7") pod "network-check-target-mpw6h" (UID: "ca0287ff-6de3-46ba-b97f-259d5477f07c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 15:09:02.334739 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:02.334681 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 15:04:00 +0000 UTC" deadline="2028-02-09 19:02:40.467141323 +0000 UTC" Sep 08 15:09:02.334739 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:02.334737 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12459h53m38.132408536s" Sep 08 15:09:02.366382 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:02.365622 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:02.366382 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:02.365757 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b44p" podUID="f755fdd9-db6c-43ec-bd91-5d0d861aae73" Sep 08 15:09:02.366382 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:02.366293 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:02.366677 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:02.366446 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:09:02.366677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:02.366552 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:02.366677 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:02.366646 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpw6h" podUID="ca0287ff-6de3-46ba-b97f-259d5477f07c" Sep 08 15:09:02.388239 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:02.388176 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-57pdq" event={"ID":"0bfdcd24-b0c7-444b-af1a-7d9370b714b4","Type":"ContainerStarted","Data":"19947586f8d7af5184ad9162e911dd04a06933aaf355b66fa6f1b4546335f6bd"} Sep 08 15:09:02.389706 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:02.389677 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" event={"ID":"257cd212-bcaa-405f-8682-6c197ca64024","Type":"ContainerStarted","Data":"7872351d0371dec2994e3af8f2ecf0e8f4a075e74a70bc0af3b7a438c1864b25"} Sep 08 15:09:02.391267 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:02.391231 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pxfh4" event={"ID":"7662e1ba-5c64-4348-844c-1f5fd285ce6f","Type":"ContainerStarted","Data":"d60a00d47eb7b0a640458d6eaf662ec8e85d637de2f3394b777aaee3031c41c2"} Sep 08 15:09:02.394217 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:02.394183 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j97ml" event={"ID":"c83abab5-7e60-48e3-b075-bbd45a2a68ed","Type":"ContainerStarted","Data":"fd205dc97214c9aad68c5c4be6f0073b5e9ba32789aa76a65893c73858bccaf9"} Sep 08 15:09:02.395661 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:02.395632 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" event={"ID":"e4293ce5-30ea-4210-9b84-ec6539d992cb","Type":"ContainerStarted","Data":"f0b15c63f649ba39df19e4f027eca0ec929f44afb2a08907289e463e31fc4744"} Sep 08 15:09:02.397910 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:02.397885 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-159.ec2.internal" event={"ID":"f326de9abe5883a85f6af224dafa6b15","Type":"ContainerStarted","Data":"a0ddfd7f03bd7c21714bb5def4a4dad1dfc380be7cb56a68b2380f00f37c5380"} Sep 08 15:09:02.400884 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:02.400862 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" event={"ID":"ddc06cac-5923-47a1-81c1-eef7c8aefb24","Type":"ContainerStarted","Data":"d0629a74d95827e1f3d8fbd37793f7e81c099979d092370c4abd6d8bfeb4b0c3"} Sep 08 15:09:02.436797 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:02.436744 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-159.ec2.internal" podStartSLOduration=2.436728605 podStartE2EDuration="2.436728605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:09:02.436225371 +0000 UTC m=+4.588403958" watchObservedRunningTime="2026-09-08 15:09:02.436728605 +0000 UTC m=+4.588907189" Sep 08 15:09:03.410981 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:03.410928 2572 generic.go:356] "Generic (PLEG): container finished" podID="1e79f6357314a98cc42b005b66beb364" containerID="1f97126aedafc0645dbce9a1c0d67167e5643c8daaef0ef3bed78be32e3ffced" exitCode=0 Sep 08 15:09:03.411701 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:03.411632 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal" event={"ID":"1e79f6357314a98cc42b005b66beb364","Type":"ContainerDied","Data":"1f97126aedafc0645dbce9a1c0d67167e5643c8daaef0ef3bed78be32e3ffced"} Sep 08 15:09:03.904128 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:03.904092 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret\") pod \"global-pull-secret-syncer-9b44p\" (UID: \"f755fdd9-db6c-43ec-bd91-5d0d861aae73\") " pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:03.904313 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:03.904270 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 15:09:03.904382 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:03.904329 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret podName:f755fdd9-db6c-43ec-bd91-5d0d861aae73 nodeName:}" failed. No retries permitted until 2026-09-08 15:09:07.904310767 +0000 UTC m=+10.056489328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret") pod "global-pull-secret-syncer-9b44p" (UID: "f755fdd9-db6c-43ec-bd91-5d0d861aae73") : object "kube-system"/"original-pull-secret" not registered Sep 08 15:09:04.004803 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:04.004765 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs\") pod \"network-metrics-daemon-n5x8x\" (UID: \"1dafedb8-50fc-4eff-a5a8-ee1be3e3f751\") " pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:04.005036 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:04.004987 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 15:09:04.005119 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:04.005053 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs podName:1dafedb8-50fc-4eff-a5a8-ee1be3e3f751 nodeName:}" failed. No retries permitted until 2026-09-08 15:09:08.005031449 +0000 UTC m=+10.157210016 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs") pod "network-metrics-daemon-n5x8x" (UID: "1dafedb8-50fc-4eff-a5a8-ee1be3e3f751") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 15:09:04.106162 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:04.106128 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfv7\" (UniqueName: \"kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7\") pod \"network-check-target-mpw6h\" (UID: \"ca0287ff-6de3-46ba-b97f-259d5477f07c\") " pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:04.106338 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:04.106320 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 15:09:04.106412 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:04.106348 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 15:09:04.106412 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:04.106361 2572 projected.go:196] Error preparing data for projected volume kube-api-access-hgfv7 for pod openshift-network-diagnostics/network-check-target-mpw6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 15:09:04.106515 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:04.106427 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7 podName:ca0287ff-6de3-46ba-b97f-259d5477f07c nodeName:}" failed. No retries permitted until 2026-09-08 15:09:08.106406879 +0000 UTC m=+10.258585442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hgfv7" (UniqueName: "kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7") pod "network-check-target-mpw6h" (UID: "ca0287ff-6de3-46ba-b97f-259d5477f07c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 15:09:04.373728 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:04.373677 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:04.373728 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:04.373689 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:04.373728 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:04.373690 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:04.374020 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:04.373919 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:09:04.376913 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:04.374585 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b44p" podUID="f755fdd9-db6c-43ec-bd91-5d0d861aae73" Sep 08 15:09:04.376913 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:04.374711 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpw6h" podUID="ca0287ff-6de3-46ba-b97f-259d5477f07c" Sep 08 15:09:06.366044 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:06.365823 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:06.366044 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:06.365848 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:06.366044 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:06.365953 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b44p" podUID="f755fdd9-db6c-43ec-bd91-5d0d861aae73" Sep 08 15:09:06.366606 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:06.366057 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:09:06.366606 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:06.366109 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:06.366606 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:06.366178 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpw6h" podUID="ca0287ff-6de3-46ba-b97f-259d5477f07c" Sep 08 15:09:07.940377 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:07.940294 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret\") pod \"global-pull-secret-syncer-9b44p\" (UID: \"f755fdd9-db6c-43ec-bd91-5d0d861aae73\") " pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:07.940776 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:07.940442 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 15:09:07.940776 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:07.940509 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret podName:f755fdd9-db6c-43ec-bd91-5d0d861aae73 nodeName:}" failed. No retries permitted until 2026-09-08 15:09:15.940478592 +0000 UTC m=+18.092657149 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret") pod "global-pull-secret-syncer-9b44p" (UID: "f755fdd9-db6c-43ec-bd91-5d0d861aae73") : object "kube-system"/"original-pull-secret" not registered Sep 08 15:09:08.040652 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:08.040620 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs\") pod \"network-metrics-daemon-n5x8x\" (UID: \"1dafedb8-50fc-4eff-a5a8-ee1be3e3f751\") " pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:08.040817 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:08.040743 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 15:09:08.040817 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:08.040789 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs podName:1dafedb8-50fc-4eff-a5a8-ee1be3e3f751 nodeName:}" failed. No retries permitted until 2026-09-08 15:09:16.040775929 +0000 UTC m=+18.192954487 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs") pod "network-metrics-daemon-n5x8x" (UID: "1dafedb8-50fc-4eff-a5a8-ee1be3e3f751") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 15:09:08.141087 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:08.141049 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfv7\" (UniqueName: \"kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7\") pod \"network-check-target-mpw6h\" (UID: \"ca0287ff-6de3-46ba-b97f-259d5477f07c\") " pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:08.141641 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:08.141236 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 15:09:08.141641 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:08.141260 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 15:09:08.141641 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:08.141270 2572 projected.go:196] Error preparing data for projected volume kube-api-access-hgfv7 for pod openshift-network-diagnostics/network-check-target-mpw6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 15:09:08.141641 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:08.141314 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7 podName:ca0287ff-6de3-46ba-b97f-259d5477f07c nodeName:}" failed. No retries permitted until 2026-09-08 15:09:16.141301086 +0000 UTC m=+18.293479643 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hgfv7" (UniqueName: "kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7") pod "network-check-target-mpw6h" (UID: "ca0287ff-6de3-46ba-b97f-259d5477f07c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 15:09:08.365748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:08.365721 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:08.365907 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:08.365752 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:08.367198 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:08.366661 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b44p" podUID="f755fdd9-db6c-43ec-bd91-5d0d861aae73" Sep 08 15:09:08.367198 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:08.367056 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:09:08.367198 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:08.367099 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:08.367198 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:08.367167 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpw6h" podUID="ca0287ff-6de3-46ba-b97f-259d5477f07c" Sep 08 15:09:09.427987 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:09.427761 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal" event={"ID":"1e79f6357314a98cc42b005b66beb364","Type":"ContainerStarted","Data":"892d001a5a333537a7ae3f89cdbf6aee86303803b815744b6d71045962f6d799"} Sep 08 15:09:09.430336 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:09.430310 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wjg4s" event={"ID":"90fade35-fbac-48dd-834c-919ef54da378","Type":"ContainerStarted","Data":"6caf3aca781ac6f11f799f51885ab7d61bf4111b6c579e620bc58ffe272c8c4e"} Sep 08 15:09:09.432319 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:09.432297 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zxbnk" event={"ID":"1dfce943-0703-4e7d-b479-a7616aa67993","Type":"ContainerStarted","Data":"43c304eb32ab89f2fa4b704fa76591706d0e25371aae8fc130d5bea0c549f425"} Sep 08 15:09:09.434198 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:09.434173 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j97ml" event={"ID":"c83abab5-7e60-48e3-b075-bbd45a2a68ed","Type":"ContainerStarted","Data":"d63944e185c0641d8106ccf9c1826b5e236bfa81a0ee20a765ae18d51b37658d"} Sep 08 15:09:09.436363 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:09.436334 2572 generic.go:356] "Generic (PLEG): container finished" podID="4d45089b-2c7d-45d7-883b-417a0bcb9c1b" containerID="cb982b5239bdc602021b959abf9fa2326d3b2697ef8965e450eb50df47da7ddf" exitCode=0 Sep 08 15:09:09.436469 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:09.436452 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mhvv" event={"ID":"4d45089b-2c7d-45d7-883b-417a0bcb9c1b","Type":"ContainerDied","Data":"cb982b5239bdc602021b959abf9fa2326d3b2697ef8965e450eb50df47da7ddf"} Sep 08 15:09:09.439638 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:09.439614 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" event={"ID":"ddc06cac-5923-47a1-81c1-eef7c8aefb24","Type":"ContainerStarted","Data":"f59c858c9df6a6dfb7abdfa4f8972934512cb6c9a98c6dc3c17c853550886d9d"} Sep 08 15:09:09.446605 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:09.446560 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-159.ec2.internal" podStartSLOduration=9.446544291 podStartE2EDuration="9.446544291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:09:09.446420851 +0000 UTC m=+11.598599430" watchObservedRunningTime="2026-09-08 15:09:09.446544291 +0000 UTC m=+11.598722872" Sep 08 15:09:09.478916 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:09.478825 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j97ml" podStartSLOduration=3.41345804 podStartE2EDuration="10.478813111s" totalImagesPullingTime="7.065355071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:09:09.461839457 +0000 UTC m=+11.614018040" watchObservedRunningTime="2026-09-08 15:09:09.478813111 +0000 UTC m=+11.630991690" Sep 08 15:09:09.479020 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:09.478995 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zxbnk" podStartSLOduration=3.081638188 podStartE2EDuration="11.478984744s" totalImagesPullingTime="8.397346556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:09:09.47861416 +0000 UTC m=+11.630792738" watchObservedRunningTime="2026-09-08 15:09:09.478984744 +0000 UTC m=+11.631163325" Sep 08 15:09:09.495120 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:09.495063 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wjg4s" podStartSLOduration=3.10248012 podStartE2EDuration="11.495046533s" totalImagesPullingTime="8.392566413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:09:09.494477021 +0000 UTC m=+11.646655602" watchObservedRunningTime="2026-09-08 15:09:09.495046533 +0000 UTC m=+11.647225113" Sep 08 15:09:09.694936 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:09.694903 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j97ml" Sep 08 15:09:09.695640 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:09.695616 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j97ml" Sep 08 15:09:10.366052 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:10.366025 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:10.366197 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:10.366024 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:10.366197 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:10.366147 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b44p" podUID="f755fdd9-db6c-43ec-bd91-5d0d861aae73" Sep 08 15:09:10.366197 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:10.366164 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:10.366343 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:10.366269 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:09:10.366385 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:10.366343 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpw6h" podUID="ca0287ff-6de3-46ba-b97f-259d5477f07c" Sep 08 15:09:11.445722 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:11.445671 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-57pdq" event={"ID":"0bfdcd24-b0c7-444b-af1a-7d9370b714b4","Type":"ContainerStarted","Data":"61b62df0db90a5f5ee7deccba85f89df628b33d86be61b9d645a3a9311f10951"} Sep 08 15:09:11.446110 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:11.445692 2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 08 15:09:11.461740 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:11.461693 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-57pdq" podStartSLOduration=5.436038803 podStartE2EDuration="12.461679507s" totalImagesPullingTime="7.025640704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:09:11.461597456 +0000 UTC m=+13.613776036" watchObservedRunningTime="2026-09-08 15:09:11.461679507 +0000 UTC m=+13.613858087" Sep 08 15:09:12.365749 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:12.365716 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:12.365749 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:12.365739 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:12.366002 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:12.365843 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b44p" podUID="f755fdd9-db6c-43ec-bd91-5d0d861aae73" Sep 08 15:09:12.366002 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:12.365917 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:09:12.366002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:12.365934 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:12.366131 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:12.366019 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpw6h" podUID="ca0287ff-6de3-46ba-b97f-259d5477f07c" Sep 08 15:09:14.365909 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:14.365878 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:14.366334 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:14.366013 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:09:14.366334 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:14.366250 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:14.366334 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:14.366327 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b44p" podUID="f755fdd9-db6c-43ec-bd91-5d0d861aae73" Sep 08 15:09:14.366438 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:14.366359 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:14.366468 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:14.366444 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpw6h" podUID="ca0287ff-6de3-46ba-b97f-259d5477f07c" Sep 08 15:09:16.004412 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:16.004379 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret\") pod \"global-pull-secret-syncer-9b44p\" (UID: \"f755fdd9-db6c-43ec-bd91-5d0d861aae73\") " pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:16.004916 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:16.004573 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 15:09:16.004916 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:16.004652 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret podName:f755fdd9-db6c-43ec-bd91-5d0d861aae73 nodeName:}" failed. No retries permitted until 2026-09-08 15:09:32.00463151 +0000 UTC m=+34.156810082 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret") pod "global-pull-secret-syncer-9b44p" (UID: "f755fdd9-db6c-43ec-bd91-5d0d861aae73") : object "kube-system"/"original-pull-secret" not registered Sep 08 15:09:16.105322 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:16.105281 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs\") pod \"network-metrics-daemon-n5x8x\" (UID: \"1dafedb8-50fc-4eff-a5a8-ee1be3e3f751\") " pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:16.105454 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:16.105398 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 15:09:16.105454 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:16.105446 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs podName:1dafedb8-50fc-4eff-a5a8-ee1be3e3f751 nodeName:}" failed. No retries permitted until 2026-09-08 15:09:32.105432705 +0000 UTC m=+34.257611263 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs") pod "network-metrics-daemon-n5x8x" (UID: "1dafedb8-50fc-4eff-a5a8-ee1be3e3f751") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 15:09:16.205940 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:16.205911 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfv7\" (UniqueName: \"kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7\") pod \"network-check-target-mpw6h\" (UID: \"ca0287ff-6de3-46ba-b97f-259d5477f07c\") " pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:16.206082 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:16.206061 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 15:09:16.206082 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:16.206080 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 15:09:16.206176 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:16.206089 2572 projected.go:196] Error preparing data for projected volume kube-api-access-hgfv7 for pod openshift-network-diagnostics/network-check-target-mpw6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 15:09:16.206176 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:16.206139 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7 podName:ca0287ff-6de3-46ba-b97f-259d5477f07c nodeName:}" failed. No retries permitted until 2026-09-08 15:09:32.2061258 +0000 UTC m=+34.358304362 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hgfv7" (UniqueName: "kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7") pod "network-check-target-mpw6h" (UID: "ca0287ff-6de3-46ba-b97f-259d5477f07c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 15:09:16.365332 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:16.365256 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:16.365461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:16.365256 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:16.365461 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:16.365381 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:09:16.365461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:16.365269 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:16.365594 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:16.365430 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b44p" podUID="f755fdd9-db6c-43ec-bd91-5d0d861aae73" Sep 08 15:09:16.365594 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:16.365552 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpw6h" podUID="ca0287ff-6de3-46ba-b97f-259d5477f07c" Sep 08 15:09:18.366520 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:18.366470 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:18.366944 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:18.366595 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:18.366944 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:18.366627 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:18.366944 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:18.366635 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:09:18.366944 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:18.366703 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b44p" podUID="f755fdd9-db6c-43ec-bd91-5d0d861aae73" Sep 08 15:09:18.366944 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:18.366783 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpw6h" podUID="ca0287ff-6de3-46ba-b97f-259d5477f07c" Sep 08 15:09:20.365389 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:20.365353 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:20.365389 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:20.365376 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:20.365848 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:20.365455 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:20.365848 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:20.365469 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b44p" podUID="f755fdd9-db6c-43ec-bd91-5d0d861aae73" Sep 08 15:09:20.365848 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:20.365553 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpw6h" podUID="ca0287ff-6de3-46ba-b97f-259d5477f07c" Sep 08 15:09:20.365848 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:20.365627 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:09:22.180009 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:22.179883 2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 08 15:09:22.329690 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:22.329462 2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T15:09:22.180004051Z","UUID":"17dc0919-31c0-4bf0-816f-8f64640621d3","Handler":null,"Name":"","Endpoint":""} Sep 08 15:09:22.330947 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:22.330931 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 08 15:09:22.331037 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:22.330960 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 08 15:09:22.365434 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:22.365414 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:22.365545 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:22.365420 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:22.365610 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:22.365538 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b44p" podUID="f755fdd9-db6c-43ec-bd91-5d0d861aae73" Sep 08 15:09:22.365670 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:22.365612 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:09:22.365726 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:22.365675 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:22.365777 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:22.365749 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpw6h" podUID="ca0287ff-6de3-46ba-b97f-259d5477f07c" Sep 08 15:09:22.468355 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:22.468328 2572 generic.go:356] "Generic (PLEG): container finished" podID="4d45089b-2c7d-45d7-883b-417a0bcb9c1b" containerID="700a9d3c0b0b14ed28ff7f6a8671d1163bc1dde93050a5ab3a07075e57f17604" exitCode=0 Sep 08 15:09:22.468432 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:22.468397 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mhvv" event={"ID":"4d45089b-2c7d-45d7-883b-417a0bcb9c1b","Type":"ContainerDied","Data":"700a9d3c0b0b14ed28ff7f6a8671d1163bc1dde93050a5ab3a07075e57f17604"} Sep 08 15:09:22.470407 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:22.470385 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" event={"ID":"ddc06cac-5923-47a1-81c1-eef7c8aefb24","Type":"ContainerStarted","Data":"1e64da6e35ddfa1bb2d206585fcc52f84180173b9507103b9f4e5055d0731927"} Sep 08 15:09:22.472698 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:22.472678 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" event={"ID":"257cd212-bcaa-405f-8682-6c197ca64024","Type":"ContainerStarted","Data":"5b70160fdd7ac3df156d8fb96934642cbe8529975147fd9084838f4780927beb"} Sep 08 15:09:22.474170 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:22.474147 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pxfh4" event={"ID":"7662e1ba-5c64-4348-844c-1f5fd285ce6f","Type":"ContainerStarted","Data":"45302cc75a5e7f7fc3de50aca15d66890ff62a749823cf30f76c9f3ec9235e3a"} Sep 08 15:09:22.476925 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:22.476906 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" event={"ID":"e4293ce5-30ea-4210-9b84-ec6539d992cb","Type":"ContainerStarted","Data":"f7cf34541ad31117700d8d958090ad95e07d280a1f2a0b3f3fd1ceb38f7280f6"} Sep 08 15:09:22.477018 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:22.476958 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" event={"ID":"e4293ce5-30ea-4210-9b84-ec6539d992cb","Type":"ContainerStarted","Data":"3f7ff7e8edf11e3c7794c5d95f2d02b28e20086b4ff827dd8a4ec3263302f00c"} Sep 08 15:09:22.477018 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:22.476986 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" event={"ID":"e4293ce5-30ea-4210-9b84-ec6539d992cb","Type":"ContainerStarted","Data":"449f63efd833130ae43c4d82fb6d463676a78aa456c0e77d609863b80f57cf9e"} Sep 08 15:09:22.477018 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:22.477013 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" event={"ID":"e4293ce5-30ea-4210-9b84-ec6539d992cb","Type":"ContainerStarted","Data":"962fd92ec09a480faee6666a04ae9211727e35f501664dac5b4a6c8df5057bf7"} Sep 08 15:09:22.477171 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:22.477039 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" event={"ID":"e4293ce5-30ea-4210-9b84-ec6539d992cb","Type":"ContainerStarted","Data":"7a034878186570fa1cfe5701c84f91f64659e365570bd42d5faaca2a86357c71"} Sep 08 15:09:22.511656 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:22.511615 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pxfh4" podStartSLOduration=3.464052705 podStartE2EDuration="23.51160147s" totalImagesPullingTime="20.047548765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:09:22.510625224 +0000 UTC m=+24.662803804" watchObservedRunningTime="2026-09-08 15:09:22.51160147 +0000 UTC m=+24.663780050" Sep 08 15:09:22.531455 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:22.531422 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mmfzq" podStartSLOduration=3.501130087 podStartE2EDuration="23.531411006s" totalImagesPullingTime="20.030280919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:09:22.531321584 +0000 UTC m=+24.683500165" watchObservedRunningTime="2026-09-08 15:09:22.531411006 +0000 UTC m=+24.683589586" Sep 08 15:09:23.483776 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:23.483676 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" event={"ID":"e4293ce5-30ea-4210-9b84-ec6539d992cb","Type":"ContainerStarted","Data":"3b52a707dbe36673348a903d2196a1e80160bdeea2c14917b3d024533e61a274"} Sep 08 15:09:24.365845 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:24.365819 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:24.365969 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:24.365819 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:24.365969 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:24.365917 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b44p" podUID="f755fdd9-db6c-43ec-bd91-5d0d861aae73" Sep 08 15:09:24.365969 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:24.365940 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:24.366134 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:24.366018 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpw6h" podUID="ca0287ff-6de3-46ba-b97f-259d5477f07c" Sep 08 15:09:24.366134 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:24.366093 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:09:24.487871 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:24.487849 2572 generic.go:356] "Generic (PLEG): container finished" podID="4d45089b-2c7d-45d7-883b-417a0bcb9c1b" containerID="546cfecd6e417a3f7953846b4aa4760e7c23fc126baa426863d0fa2d462a0978" exitCode=0 Sep 08 15:09:24.488236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:24.487935 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mhvv" event={"ID":"4d45089b-2c7d-45d7-883b-417a0bcb9c1b","Type":"ContainerDied","Data":"546cfecd6e417a3f7953846b4aa4760e7c23fc126baa426863d0fa2d462a0978"} Sep 08 15:09:24.490239 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:24.490215 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" event={"ID":"ddc06cac-5923-47a1-81c1-eef7c8aefb24","Type":"ContainerStarted","Data":"1c9a15e095ce8d6b91df90eaada17a067a0ead220e7a1c42dec60a39180873c8"} Sep 08 15:09:24.540173 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:24.540133 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swqb4" podStartSLOduration=3.9045479849999998 podStartE2EDuration="25.540111631s" totalImagesPullingTime="21.635563646s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:08:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:09:24.539853857 +0000 UTC m=+26.692032437" watchObservedRunningTime="2026-09-08 15:09:24.540111631 +0000 UTC m=+26.692290210" Sep 08 15:09:25.495106 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:25.495072 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" event={"ID":"e4293ce5-30ea-4210-9b84-ec6539d992cb","Type":"ContainerStarted","Data":"344d68bd5894781cfa117fcbdfc5f205383574061f8299d60f39beb769efa07e"} Sep 08 15:09:26.366240 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:26.366207 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:26.366419 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:26.366217 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:26.366419 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:26.366302 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpw6h" podUID="ca0287ff-6de3-46ba-b97f-259d5477f07c" Sep 08 15:09:26.366574 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:26.366419 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b44p" podUID="f755fdd9-db6c-43ec-bd91-5d0d861aae73" Sep 08 15:09:26.366574 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:26.366473 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:26.366574 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:26.366558 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:09:26.499887 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:26.499855 2572 generic.go:356] "Generic (PLEG): container finished" podID="4d45089b-2c7d-45d7-883b-417a0bcb9c1b" containerID="e6916e88b97c55f9dd7a2fb66a8bf3a85476ac7903143c61e75dfd6fea90bd66" exitCode=0 Sep 08 15:09:26.500241 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:26.499897 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mhvv" event={"ID":"4d45089b-2c7d-45d7-883b-417a0bcb9c1b","Type":"ContainerDied","Data":"e6916e88b97c55f9dd7a2fb66a8bf3a85476ac7903143c61e75dfd6fea90bd66"} Sep 08 15:09:27.282938 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:27.282762 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j97ml" Sep 08 15:09:27.283109 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:27.283047 2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 08 15:09:27.283468 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:27.283443 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j97ml" Sep 08 15:09:27.506269 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:27.506235 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" event={"ID":"e4293ce5-30ea-4210-9b84-ec6539d992cb","Type":"ContainerStarted","Data":"c4c4be30301d6b65bd8c607d2e2ccdcfbb8ddefeb7c8917c6aede973c9079a38"} Sep 08 15:09:27.506928 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:27.506534 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:27.506928 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:27.506605 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:27.506928 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:27.506661 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:27.523063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:27.523038 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:27.523294 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:27.523280 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:09:27.533015 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:27.532944 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" podStartSLOduration=8.489527842 podStartE2EDuration="28.532931904s" totalImagesPullingTime="20.043404062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:09:27.532430117 +0000 UTC m=+29.684608696" watchObservedRunningTime="2026-09-08 15:09:27.532931904 +0000 UTC m=+29.685110513" Sep 08 15:09:28.366246 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:28.366215 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:28.366387 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:28.366364 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:28.366434 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:28.366395 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b44p" podUID="f755fdd9-db6c-43ec-bd91-5d0d861aae73" Sep 08 15:09:28.366434 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:28.366424 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:09:28.366547 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:28.366479 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:28.366630 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:28.366599 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpw6h" podUID="ca0287ff-6de3-46ba-b97f-259d5477f07c" Sep 08 15:09:28.660607 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:28.660527 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mpw6h"] Sep 08 15:09:28.661122 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:28.660620 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:28.661122 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:28.660695 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpw6h" podUID="ca0287ff-6de3-46ba-b97f-259d5477f07c" Sep 08 15:09:28.664337 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:28.664310 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9b44p"] Sep 08 15:09:28.664508 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:28.664392 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:28.664572 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:28.664507 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b44p" podUID="f755fdd9-db6c-43ec-bd91-5d0d861aae73" Sep 08 15:09:28.664916 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:28.664900 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n5x8x"] Sep 08 15:09:28.664979 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:28.664969 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:28.665053 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:28.665039 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:09:30.365832 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:30.365660 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:30.366169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:30.365688 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:30.366169 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:30.365904 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpw6h" podUID="ca0287ff-6de3-46ba-b97f-259d5477f07c" Sep 08 15:09:30.366169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:30.365704 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:30.366169 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:30.366011 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b44p" podUID="f755fdd9-db6c-43ec-bd91-5d0d861aae73" Sep 08 15:09:30.366169 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:30.366085 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:09:32.025980 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:32.025937 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret\") pod \"global-pull-secret-syncer-9b44p\" (UID: \"f755fdd9-db6c-43ec-bd91-5d0d861aae73\") " pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:32.026410 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:32.026040 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 15:09:32.026410 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:32.026122 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret podName:f755fdd9-db6c-43ec-bd91-5d0d861aae73 nodeName:}" failed. No retries permitted until 2026-09-08 15:10:04.026099533 +0000 UTC m=+66.178278111 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret") pod "global-pull-secret-syncer-9b44p" (UID: "f755fdd9-db6c-43ec-bd91-5d0d861aae73") : object "kube-system"/"original-pull-secret" not registered Sep 08 15:09:32.126735 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:32.126699 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs\") pod \"network-metrics-daemon-n5x8x\" (UID: \"1dafedb8-50fc-4eff-a5a8-ee1be3e3f751\") " pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:32.126873 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:32.126840 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 15:09:32.126916 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:32.126909 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs podName:1dafedb8-50fc-4eff-a5a8-ee1be3e3f751 nodeName:}" failed. No retries permitted until 2026-09-08 15:10:04.12688914 +0000 UTC m=+66.279067723 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs") pod "network-metrics-daemon-n5x8x" (UID: "1dafedb8-50fc-4eff-a5a8-ee1be3e3f751") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 15:09:32.228028 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:32.227994 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfv7\" (UniqueName: \"kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7\") pod \"network-check-target-mpw6h\" (UID: \"ca0287ff-6de3-46ba-b97f-259d5477f07c\") " pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:32.228190 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:32.228151 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 15:09:32.228190 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:32.228169 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 15:09:32.228190 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:32.228182 2572 projected.go:196] Error preparing data for projected volume kube-api-access-hgfv7 for pod openshift-network-diagnostics/network-check-target-mpw6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 15:09:32.228312 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:32.228241 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7 podName:ca0287ff-6de3-46ba-b97f-259d5477f07c nodeName:}" failed. No retries permitted until 2026-09-08 15:10:04.228223385 +0000 UTC m=+66.380401947 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hgfv7" (UniqueName: "kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7") pod "network-check-target-mpw6h" (UID: "ca0287ff-6de3-46ba-b97f-259d5477f07c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 15:09:32.365949 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:32.365433 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:32.365949 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:32.365436 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:32.365949 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:32.365576 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b44p" podUID="f755fdd9-db6c-43ec-bd91-5d0d861aae73" Sep 08 15:09:32.365949 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:32.365592 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:32.365949 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:32.365688 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:09:32.365949 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:32.365767 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpw6h" podUID="ca0287ff-6de3-46ba-b97f-259d5477f07c" Sep 08 15:09:34.365971 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.365939 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:34.365971 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.365969 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:34.365971 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.365971 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:34.366754 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:34.366084 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpw6h" podUID="ca0287ff-6de3-46ba-b97f-259d5477f07c" Sep 08 15:09:34.366754 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:34.366116 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:09:34.366754 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:34.366193 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9b44p" podUID="f755fdd9-db6c-43ec-bd91-5d0d861aae73" Sep 08 15:09:34.523393 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.523368 2572 generic.go:356] "Generic (PLEG): container finished" podID="4d45089b-2c7d-45d7-883b-417a0bcb9c1b" containerID="64d8a5b86e92376466bb204ae508b28eeffb3d8be761b1707a8de260417163b6" exitCode=0 Sep 08 15:09:34.523518 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.523406 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mhvv" event={"ID":"4d45089b-2c7d-45d7-883b-417a0bcb9c1b","Type":"ContainerDied","Data":"64d8a5b86e92376466bb204ae508b28eeffb3d8be761b1707a8de260417163b6"} Sep 08 15:09:34.617150 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.617103 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-159.ec2.internal" event="NodeReady" Sep 08 15:09:34.617228 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.617212 2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 08 15:09:34.657153 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.657131 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2"] Sep 08 15:09:34.676624 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.676600 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7974bc45c6-wptrz"] Sep 08 15:09:34.676719 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.676668 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2" Sep 08 15:09:34.679150 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.679127 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 08 15:09:34.679150 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.679151 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 08 15:09:34.679311 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.679162 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-hh9rr\"" Sep 08 15:09:34.679311 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.679163 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 08 15:09:34.679311 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.679221 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 08 15:09:34.686744 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.686726 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd"] Sep 08 15:09:34.686869 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.686852 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.691520 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.691484 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 08 15:09:34.691939 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.691916 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 08 15:09:34.691939 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.691937 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 08 15:09:34.692402 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.692276 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2m6vp\"" Sep 08 15:09:34.697703 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.697361 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 08 15:09:34.702857 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.702843 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn"] Sep 08 15:09:34.702980 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.702956 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:09:34.705098 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.705082 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 08 15:09:34.718789 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.718770 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2"] Sep 08 15:09:34.718881 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.718803 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd"] Sep 08 15:09:34.718881 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.718832 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7974bc45c6-wptrz"] Sep 08 15:09:34.718881 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.718856 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2hz45"] Sep 08 15:09:34.719038 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.718885 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:34.722802 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.722783 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 08 15:09:34.723408 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.723343 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 08 15:09:34.723568 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.723550 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 08 15:09:34.724045 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.724028 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 08 15:09:34.729111 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.729092 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pj4qx"] Sep 08 15:09:34.729214 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.729199 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2hz45" Sep 08 15:09:34.733742 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.733723 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 08 15:09:34.733973 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.733956 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 08 15:09:34.734553 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.734539 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cc4mw\"" Sep 08 15:09:34.739229 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.739211 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn"] Sep 08 15:09:34.739314 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.739241 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pj4qx"] Sep 08 15:09:34.739314 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.739262 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2hz45"] Sep 08 15:09:34.739314 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.739303 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pj4qx" Sep 08 15:09:34.742713 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.742639 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 08 15:09:34.743975 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.743958 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9rn4v\"" Sep 08 15:09:34.744035 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.744016 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 08 15:09:34.744035 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.744021 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 08 15:09:34.843644 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.843618 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/266dc55e-c863-43b4-93ee-e4b5fc671954-config-volume\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:09:34.843730 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.843650 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79ffcd2e-02fe-404a-b0a5-e818335504e1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66fd956cd5-vzchn\" (UID: \"79ffcd2e-02fe-404a-b0a5-e818335504e1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:34.843730 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.843666 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/79ffcd2e-02fe-404a-b0a5-e818335504e1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66fd956cd5-vzchn\" (UID: \"79ffcd2e-02fe-404a-b0a5-e818335504e1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:34.843730 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.843682 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/266dc55e-c863-43b4-93ee-e4b5fc671954-tmp-dir\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:09:34.843730 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.843701 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df9jp\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-kube-api-access-df9jp\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.843856 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.843777 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ffcd2e-02fe-404a-b0a5-e818335504e1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66fd956cd5-vzchn\" (UID: \"79ffcd2e-02fe-404a-b0a5-e818335504e1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:34.843856 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.843809 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-ca-trust-extracted\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.843856 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.843830 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-bound-sa-token\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.843856 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.843851 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f7720ad3-ccdd-42ed-87d9-1c3e783c35d8-klusterlet-config\") pod \"klusterlet-addon-workmgr-787d6696f5-kr4rd\" (UID: \"f7720ad3-ccdd-42ed-87d9-1c3e783c35d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:09:34.843972 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.843870 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee735889-613b-4a94-a579-596fabddb360-cert\") pod \"ingress-canary-pj4qx\" (UID: \"ee735889-613b-4a94-a579-596fabddb360\") " pod="openshift-ingress-canary/ingress-canary-pj4qx" Sep 08 15:09:34.843972 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.843886 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd4k9\" (UniqueName: \"kubernetes.io/projected/79ffcd2e-02fe-404a-b0a5-e818335504e1-kube-api-access-zd4k9\") pod \"cluster-proxy-proxy-agent-66fd956cd5-vzchn\" (UID: \"79ffcd2e-02fe-404a-b0a5-e818335504e1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:34.843972 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.843903 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:09:34.843972 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.843921 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/79ffcd2e-02fe-404a-b0a5-e818335504e1-hub\") pod \"cluster-proxy-proxy-agent-66fd956cd5-vzchn\" (UID: \"79ffcd2e-02fe-404a-b0a5-e818335504e1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:34.843972 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.843937 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.843972 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.843950 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-certificates\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.843972 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.843970 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/79ffcd2e-02fe-404a-b0a5-e818335504e1-ca\") pod \"cluster-proxy-proxy-agent-66fd956cd5-vzchn\" (UID: \"79ffcd2e-02fe-404a-b0a5-e818335504e1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:34.844220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.844012 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff45m\" (UniqueName: \"kubernetes.io/projected/ee735889-613b-4a94-a579-596fabddb360-kube-api-access-ff45m\") pod \"ingress-canary-pj4qx\" (UID: \"ee735889-613b-4a94-a579-596fabddb360\") " pod="openshift-ingress-canary/ingress-canary-pj4qx" Sep 08 15:09:34.844220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.844041 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-installation-pull-secrets\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.844220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.844090 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrzws\" (UniqueName: \"kubernetes.io/projected/266dc55e-c863-43b4-93ee-e4b5fc671954-kube-api-access-mrzws\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:09:34.844220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.844125 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx6rt\" (UniqueName: \"kubernetes.io/projected/cabea983-2f0a-4119-a45a-85eccf8fbd14-kube-api-access-fx6rt\") pod \"managed-serviceaccount-addon-agent-79cc588b94-zd8j2\" (UID: \"cabea983-2f0a-4119-a45a-85eccf8fbd14\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2" Sep 08 15:09:34.844220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.844163 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-trusted-ca\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.844220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.844203 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7720ad3-ccdd-42ed-87d9-1c3e783c35d8-tmp\") pod \"klusterlet-addon-workmgr-787d6696f5-kr4rd\" (UID: \"f7720ad3-ccdd-42ed-87d9-1c3e783c35d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:09:34.844220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.844219 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-image-registry-private-configuration\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.844417 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.844234 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gh95\" (UniqueName: \"kubernetes.io/projected/f7720ad3-ccdd-42ed-87d9-1c3e783c35d8-kube-api-access-8gh95\") pod \"klusterlet-addon-workmgr-787d6696f5-kr4rd\" (UID: \"f7720ad3-ccdd-42ed-87d9-1c3e783c35d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:09:34.844417 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.844247 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cabea983-2f0a-4119-a45a-85eccf8fbd14-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79cc588b94-zd8j2\" (UID: \"cabea983-2f0a-4119-a45a-85eccf8fbd14\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2" Sep 08 15:09:34.944925 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.944881 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7720ad3-ccdd-42ed-87d9-1c3e783c35d8-tmp\") pod \"klusterlet-addon-workmgr-787d6696f5-kr4rd\" (UID: \"f7720ad3-ccdd-42ed-87d9-1c3e783c35d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:09:34.944925 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.944908 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-image-registry-private-configuration\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.944925 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.944924 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gh95\" (UniqueName: \"kubernetes.io/projected/f7720ad3-ccdd-42ed-87d9-1c3e783c35d8-kube-api-access-8gh95\") pod \"klusterlet-addon-workmgr-787d6696f5-kr4rd\" (UID: \"f7720ad3-ccdd-42ed-87d9-1c3e783c35d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:09:34.945086 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.944939 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cabea983-2f0a-4119-a45a-85eccf8fbd14-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79cc588b94-zd8j2\" (UID: \"cabea983-2f0a-4119-a45a-85eccf8fbd14\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2" Sep 08 15:09:34.945086 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.944997 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/266dc55e-c863-43b4-93ee-e4b5fc671954-config-volume\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:09:34.945086 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945038 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79ffcd2e-02fe-404a-b0a5-e818335504e1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66fd956cd5-vzchn\" (UID: \"79ffcd2e-02fe-404a-b0a5-e818335504e1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:34.945086 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945063 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/79ffcd2e-02fe-404a-b0a5-e818335504e1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66fd956cd5-vzchn\" (UID: \"79ffcd2e-02fe-404a-b0a5-e818335504e1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:34.945263 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945191 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/266dc55e-c863-43b4-93ee-e4b5fc671954-tmp-dir\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:09:34.945263 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945224 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9jp\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-kube-api-access-df9jp\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.945263 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945258 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ffcd2e-02fe-404a-b0a5-e818335504e1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66fd956cd5-vzchn\" (UID: \"79ffcd2e-02fe-404a-b0a5-e818335504e1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:34.945399 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945278 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-ca-trust-extracted\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.945399 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945299 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-bound-sa-token\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.945399 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945318 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f7720ad3-ccdd-42ed-87d9-1c3e783c35d8-klusterlet-config\") pod \"klusterlet-addon-workmgr-787d6696f5-kr4rd\" (UID: \"f7720ad3-ccdd-42ed-87d9-1c3e783c35d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:09:34.945399 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945344 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee735889-613b-4a94-a579-596fabddb360-cert\") pod \"ingress-canary-pj4qx\" (UID: \"ee735889-613b-4a94-a579-596fabddb360\") " pod="openshift-ingress-canary/ingress-canary-pj4qx" Sep 08 15:09:34.945399 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945371 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd4k9\" (UniqueName: \"kubernetes.io/projected/79ffcd2e-02fe-404a-b0a5-e818335504e1-kube-api-access-zd4k9\") pod \"cluster-proxy-proxy-agent-66fd956cd5-vzchn\" (UID: \"79ffcd2e-02fe-404a-b0a5-e818335504e1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:34.945399 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945392 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:09:34.945699 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945419 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/79ffcd2e-02fe-404a-b0a5-e818335504e1-hub\") pod \"cluster-proxy-proxy-agent-66fd956cd5-vzchn\" (UID: \"79ffcd2e-02fe-404a-b0a5-e818335504e1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:34.945699 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945438 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.945699 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945459 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-certificates\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.945699 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945465 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/266dc55e-c863-43b4-93ee-e4b5fc671954-tmp-dir\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:09:34.945699 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945483 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/79ffcd2e-02fe-404a-b0a5-e818335504e1-ca\") pod \"cluster-proxy-proxy-agent-66fd956cd5-vzchn\" (UID: \"79ffcd2e-02fe-404a-b0a5-e818335504e1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:34.945699 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945537 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff45m\" (UniqueName: \"kubernetes.io/projected/ee735889-613b-4a94-a579-596fabddb360-kube-api-access-ff45m\") pod \"ingress-canary-pj4qx\" (UID: \"ee735889-613b-4a94-a579-596fabddb360\") " pod="openshift-ingress-canary/ingress-canary-pj4qx" Sep 08 15:09:34.945699 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945563 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-installation-pull-secrets\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.945699 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945598 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrzws\" (UniqueName: \"kubernetes.io/projected/266dc55e-c863-43b4-93ee-e4b5fc671954-kube-api-access-mrzws\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:09:34.945699 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945624 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx6rt\" (UniqueName: \"kubernetes.io/projected/cabea983-2f0a-4119-a45a-85eccf8fbd14-kube-api-access-fx6rt\") pod \"managed-serviceaccount-addon-agent-79cc588b94-zd8j2\" (UID: \"cabea983-2f0a-4119-a45a-85eccf8fbd14\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2" Sep 08 15:09:34.945699 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945646 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/266dc55e-c863-43b4-93ee-e4b5fc671954-config-volume\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:09:34.945699 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.945678 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-trusted-ca\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.946122 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.946069 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-ca-trust-extracted\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.946935 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.946904 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ffcd2e-02fe-404a-b0a5-e818335504e1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-66fd956cd5-vzchn\" (UID: \"79ffcd2e-02fe-404a-b0a5-e818335504e1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:34.947022 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.946999 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7720ad3-ccdd-42ed-87d9-1c3e783c35d8-tmp\") pod \"klusterlet-addon-workmgr-787d6696f5-kr4rd\" (UID: \"f7720ad3-ccdd-42ed-87d9-1c3e783c35d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:09:34.947525 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.947477 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-trusted-ca\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.947921 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:34.947681 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 15:09:34.947921 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:34.947735 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls podName:266dc55e-c863-43b4-93ee-e4b5fc671954 nodeName:}" failed. No retries permitted until 2026-09-08 15:09:35.447718568 +0000 UTC m=+37.599897127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls") pod "dns-default-2hz45" (UID: "266dc55e-c863-43b4-93ee-e4b5fc671954") : secret "dns-default-metrics-tls" not found Sep 08 15:09:34.947921 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:34.947767 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 15:09:34.947921 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:34.947779 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7974bc45c6-wptrz: secret "image-registry-tls" not found Sep 08 15:09:34.947921 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:34.947813 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls podName:6cc295f3-9a80-4e2c-82b1-b8d703944f1f nodeName:}" failed. No retries permitted until 2026-09-08 15:09:35.447801756 +0000 UTC m=+37.599980315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls") pod "image-registry-7974bc45c6-wptrz" (UID: "6cc295f3-9a80-4e2c-82b1-b8d703944f1f") : secret "image-registry-tls" not found Sep 08 15:09:34.948405 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.948237 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-certificates\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.949915 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.949875 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/79ffcd2e-02fe-404a-b0a5-e818335504e1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-66fd956cd5-vzchn\" (UID: \"79ffcd2e-02fe-404a-b0a5-e818335504e1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:34.950009 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.949956 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee735889-613b-4a94-a579-596fabddb360-cert\") pod \"ingress-canary-pj4qx\" (UID: \"ee735889-613b-4a94-a579-596fabddb360\") " pod="openshift-ingress-canary/ingress-canary-pj4qx" Sep 08 15:09:34.950009 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.949972 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-installation-pull-secrets\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.950103 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.950023 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79ffcd2e-02fe-404a-b0a5-e818335504e1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-66fd956cd5-vzchn\" (UID: \"79ffcd2e-02fe-404a-b0a5-e818335504e1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:34.950667 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.950641 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/79ffcd2e-02fe-404a-b0a5-e818335504e1-ca\") pod \"cluster-proxy-proxy-agent-66fd956cd5-vzchn\" (UID: \"79ffcd2e-02fe-404a-b0a5-e818335504e1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:34.950782 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.950760 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-image-registry-private-configuration\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.950982 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.950962 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f7720ad3-ccdd-42ed-87d9-1c3e783c35d8-klusterlet-config\") pod \"klusterlet-addon-workmgr-787d6696f5-kr4rd\" (UID: \"f7720ad3-ccdd-42ed-87d9-1c3e783c35d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:09:34.951068 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.951052 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cabea983-2f0a-4119-a45a-85eccf8fbd14-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79cc588b94-zd8j2\" (UID: \"cabea983-2f0a-4119-a45a-85eccf8fbd14\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2" Sep 08 15:09:34.951306 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.951288 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/79ffcd2e-02fe-404a-b0a5-e818335504e1-hub\") pod \"cluster-proxy-proxy-agent-66fd956cd5-vzchn\" (UID: \"79ffcd2e-02fe-404a-b0a5-e818335504e1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:34.954903 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.954878 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff45m\" (UniqueName: \"kubernetes.io/projected/ee735889-613b-4a94-a579-596fabddb360-kube-api-access-ff45m\") pod \"ingress-canary-pj4qx\" (UID: \"ee735889-613b-4a94-a579-596fabddb360\") " pod="openshift-ingress-canary/ingress-canary-pj4qx" Sep 08 15:09:34.955300 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.955278 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gh95\" (UniqueName: \"kubernetes.io/projected/f7720ad3-ccdd-42ed-87d9-1c3e783c35d8-kube-api-access-8gh95\") pod \"klusterlet-addon-workmgr-787d6696f5-kr4rd\" (UID: \"f7720ad3-ccdd-42ed-87d9-1c3e783c35d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:09:34.955396 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.955377 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df9jp\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-kube-api-access-df9jp\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.957091 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.957065 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd4k9\" (UniqueName: \"kubernetes.io/projected/79ffcd2e-02fe-404a-b0a5-e818335504e1-kube-api-access-zd4k9\") pod \"cluster-proxy-proxy-agent-66fd956cd5-vzchn\" (UID: \"79ffcd2e-02fe-404a-b0a5-e818335504e1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:34.957091 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.957087 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrzws\" (UniqueName: \"kubernetes.io/projected/266dc55e-c863-43b4-93ee-e4b5fc671954-kube-api-access-mrzws\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:09:34.957207 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.957096 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-bound-sa-token\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:34.957310 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.957294 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx6rt\" (UniqueName: \"kubernetes.io/projected/cabea983-2f0a-4119-a45a-85eccf8fbd14-kube-api-access-fx6rt\") pod \"managed-serviceaccount-addon-agent-79cc588b94-zd8j2\" (UID: \"cabea983-2f0a-4119-a45a-85eccf8fbd14\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2" Sep 08 15:09:34.997922 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:34.997902 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2" Sep 08 15:09:35.078707 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:35.078677 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:09:35.083846 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:35.083826 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:09:35.102939 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:35.102735 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pj4qx" Sep 08 15:09:35.315839 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:35.315804 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2"] Sep 08 15:09:35.320230 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:09:35.320208 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcabea983_2f0a_4119_a45a_85eccf8fbd14.slice/crio-741be6d386e66b7442088ddcb2a8f216520e7e5bb0462a1c504f77c84db54670 WatchSource:0}: Error finding container 741be6d386e66b7442088ddcb2a8f216520e7e5bb0462a1c504f77c84db54670: Status 404 returned error can't find the container with id 741be6d386e66b7442088ddcb2a8f216520e7e5bb0462a1c504f77c84db54670 Sep 08 15:09:35.370031 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:35.369984 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd"] Sep 08 15:09:35.374321 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:09:35.374297 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7720ad3_ccdd_42ed_87d9_1c3e783c35d8.slice/crio-97ad5444d960221d9c7e7626644589af7f94b68743eec4e075756b3b407d04a3 WatchSource:0}: Error finding container 97ad5444d960221d9c7e7626644589af7f94b68743eec4e075756b3b407d04a3: Status 404 returned error can't find the container with id 97ad5444d960221d9c7e7626644589af7f94b68743eec4e075756b3b407d04a3 Sep 08 15:09:35.391021 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:35.391000 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn"] Sep 08 15:09:35.393334 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:09:35.393312 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79ffcd2e_02fe_404a_b0a5_e818335504e1.slice/crio-e237e0d047c85fd4d1119b5fd104ee366f27eba5ce3004d958971a18fa0d0c86 WatchSource:0}: Error finding container e237e0d047c85fd4d1119b5fd104ee366f27eba5ce3004d958971a18fa0d0c86: Status 404 returned error can't find the container with id e237e0d047c85fd4d1119b5fd104ee366f27eba5ce3004d958971a18fa0d0c86 Sep 08 15:09:35.421445 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:35.421427 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pj4qx"] Sep 08 15:09:35.422879 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:09:35.422860 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee735889_613b_4a94_a579_596fabddb360.slice/crio-7400686d7839ca5f4c930e4f7828f79151f6185d6b854f3f3b5bfb5b195ca268 WatchSource:0}: Error finding container 7400686d7839ca5f4c930e4f7828f79151f6185d6b854f3f3b5bfb5b195ca268: Status 404 returned error can't find the container with id 7400686d7839ca5f4c930e4f7828f79151f6185d6b854f3f3b5bfb5b195ca268 Sep 08 15:09:35.451455 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:35.451397 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:09:35.451455 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:35.451426 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:35.451643 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:35.451535 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 15:09:35.451643 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:35.451545 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7974bc45c6-wptrz: secret "image-registry-tls" not found Sep 08 15:09:35.451643 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:35.451554 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 15:09:35.451643 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:35.451588 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls podName:6cc295f3-9a80-4e2c-82b1-b8d703944f1f nodeName:}" failed. No retries permitted until 2026-09-08 15:09:36.451574753 +0000 UTC m=+38.603753311 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls") pod "image-registry-7974bc45c6-wptrz" (UID: "6cc295f3-9a80-4e2c-82b1-b8d703944f1f") : secret "image-registry-tls" not found Sep 08 15:09:35.451643 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:35.451608 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls podName:266dc55e-c863-43b4-93ee-e4b5fc671954 nodeName:}" failed. No retries permitted until 2026-09-08 15:09:36.451596176 +0000 UTC m=+38.603774734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls") pod "dns-default-2hz45" (UID: "266dc55e-c863-43b4-93ee-e4b5fc671954") : secret "dns-default-metrics-tls" not found Sep 08 15:09:35.526614 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:35.526583 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2" event={"ID":"cabea983-2f0a-4119-a45a-85eccf8fbd14","Type":"ContainerStarted","Data":"741be6d386e66b7442088ddcb2a8f216520e7e5bb0462a1c504f77c84db54670"} Sep 08 15:09:35.527689 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:35.527670 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" event={"ID":"79ffcd2e-02fe-404a-b0a5-e818335504e1","Type":"ContainerStarted","Data":"e237e0d047c85fd4d1119b5fd104ee366f27eba5ce3004d958971a18fa0d0c86"} Sep 08 15:09:35.528853 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:35.528833 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" event={"ID":"f7720ad3-ccdd-42ed-87d9-1c3e783c35d8","Type":"ContainerStarted","Data":"97ad5444d960221d9c7e7626644589af7f94b68743eec4e075756b3b407d04a3"} Sep 08 15:09:35.530020 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:35.529997 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pj4qx" event={"ID":"ee735889-613b-4a94-a579-596fabddb360","Type":"ContainerStarted","Data":"7400686d7839ca5f4c930e4f7828f79151f6185d6b854f3f3b5bfb5b195ca268"} Sep 08 15:09:35.533089 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:35.533069 2572 generic.go:356] "Generic (PLEG): container finished" podID="4d45089b-2c7d-45d7-883b-417a0bcb9c1b" containerID="a0deef3e350d5b6e8f37bde5b5ef7395a116a593002abd14f2e6b8a1c2a6609a" exitCode=0 Sep 08 15:09:35.533160 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:35.533098 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mhvv" event={"ID":"4d45089b-2c7d-45d7-883b-417a0bcb9c1b","Type":"ContainerDied","Data":"a0deef3e350d5b6e8f37bde5b5ef7395a116a593002abd14f2e6b8a1c2a6609a"} Sep 08 15:09:36.367293 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:36.366606 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:09:36.367293 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:36.367237 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:09:36.368297 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:36.367622 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:09:36.370356 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:36.370003 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 15:09:36.370356 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:36.370104 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 15:09:36.370356 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:36.370240 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 15:09:36.371100 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:36.370826 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 15:09:36.371100 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:36.371112 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t2q2k\"" Sep 08 15:09:36.371368 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:36.371325 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lf7xf\"" Sep 08 15:09:36.459880 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:36.458859 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:09:36.459880 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:36.458907 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:36.459880 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:36.459193 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 15:09:36.459880 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:36.459210 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7974bc45c6-wptrz: secret "image-registry-tls" not found Sep 08 15:09:36.459880 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:36.459279 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls podName:6cc295f3-9a80-4e2c-82b1-b8d703944f1f nodeName:}" failed. No retries permitted until 2026-09-08 15:09:38.45926106 +0000 UTC m=+40.611439623 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls") pod "image-registry-7974bc45c6-wptrz" (UID: "6cc295f3-9a80-4e2c-82b1-b8d703944f1f") : secret "image-registry-tls" not found Sep 08 15:09:36.459880 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:36.459794 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 15:09:36.459880 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:36.459846 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls podName:266dc55e-c863-43b4-93ee-e4b5fc671954 nodeName:}" failed. No retries permitted until 2026-09-08 15:09:38.459830267 +0000 UTC m=+40.612008841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls") pod "dns-default-2hz45" (UID: "266dc55e-c863-43b4-93ee-e4b5fc671954") : secret "dns-default-metrics-tls" not found Sep 08 15:09:36.550876 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:36.550839 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mhvv" event={"ID":"4d45089b-2c7d-45d7-883b-417a0bcb9c1b","Type":"ContainerStarted","Data":"91876c49ddfe7554a917a3f1ca8931de7b0ee7ed1046fbbb19cca17f2b488afb"} Sep 08 15:09:36.575878 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:36.574915 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2mhvv" podStartSLOduration=7.097510407 podStartE2EDuration="37.574896001s" totalImagesPullingTime="30.477385594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:08:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:09:36.571642775 +0000 UTC m=+38.723821368" watchObservedRunningTime="2026-09-08 15:09:36.574896001 +0000 UTC m=+38.727074584" Sep 08 15:09:38.480382 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:38.480297 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:09:38.480382 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:38.480363 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:38.480935 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:38.480593 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 15:09:38.480935 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:38.480607 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7974bc45c6-wptrz: secret "image-registry-tls" not found Sep 08 15:09:38.480935 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:38.480653 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls podName:6cc295f3-9a80-4e2c-82b1-b8d703944f1f nodeName:}" failed. No retries permitted until 2026-09-08 15:09:42.480639735 +0000 UTC m=+44.632818293 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls") pod "image-registry-7974bc45c6-wptrz" (UID: "6cc295f3-9a80-4e2c-82b1-b8d703944f1f") : secret "image-registry-tls" not found Sep 08 15:09:38.481277 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:38.481254 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 15:09:38.481396 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:38.481313 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls podName:266dc55e-c863-43b4-93ee-e4b5fc671954 nodeName:}" failed. No retries permitted until 2026-09-08 15:09:42.481297997 +0000 UTC m=+44.633476561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls") pod "dns-default-2hz45" (UID: "266dc55e-c863-43b4-93ee-e4b5fc671954") : secret "dns-default-metrics-tls" not found Sep 08 15:09:42.511423 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:42.511391 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:42.511812 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:42.511533 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:09:42.511812 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:42.511554 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 15:09:42.511812 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:42.511575 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7974bc45c6-wptrz: secret "image-registry-tls" not found Sep 08 15:09:42.511812 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:42.511637 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls podName:6cc295f3-9a80-4e2c-82b1-b8d703944f1f nodeName:}" failed. No retries permitted until 2026-09-08 15:09:50.511620874 +0000 UTC m=+52.663799455 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls") pod "image-registry-7974bc45c6-wptrz" (UID: "6cc295f3-9a80-4e2c-82b1-b8d703944f1f") : secret "image-registry-tls" not found Sep 08 15:09:42.511812 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:42.511652 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 15:09:42.511812 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:42.511726 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls podName:266dc55e-c863-43b4-93ee-e4b5fc671954 nodeName:}" failed. No retries permitted until 2026-09-08 15:09:50.511708337 +0000 UTC m=+52.663886915 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls") pod "dns-default-2hz45" (UID: "266dc55e-c863-43b4-93ee-e4b5fc671954") : secret "dns-default-metrics-tls" not found Sep 08 15:09:44.576772 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:44.576698 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pj4qx" event={"ID":"ee735889-613b-4a94-a579-596fabddb360","Type":"ContainerStarted","Data":"5d6b03496552197d55b9afd6ff92493af6ff5a8156a0836d743aa0629ffb3114"} Sep 08 15:09:44.578254 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:44.578226 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2" event={"ID":"cabea983-2f0a-4119-a45a-85eccf8fbd14","Type":"ContainerStarted","Data":"b12015b0dc166604b22f89b1dc73ff17e2103316f27f8525f9749fa638d6e8a7"} Sep 08 15:09:44.580299 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:44.580278 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" event={"ID":"79ffcd2e-02fe-404a-b0a5-e818335504e1","Type":"ContainerStarted","Data":"c3f71727cab6d4fa7f17ca30b9aa452200d564a4d56d555e20db9e0f6e87a0b3"} Sep 08 15:09:44.580380 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:44.580311 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" event={"ID":"79ffcd2e-02fe-404a-b0a5-e818335504e1","Type":"ContainerStarted","Data":"19a62ecf9196fb64a0fc255ced2f18cc90d1acdb2f039ca3bf172166792a4749"} Sep 08 15:09:44.580380 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:44.580328 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" event={"ID":"79ffcd2e-02fe-404a-b0a5-e818335504e1","Type":"ContainerStarted","Data":"f933df4b858dbad344de8c3abfd7758aca11389a87d5205a25624d26fc405313"} Sep 08 15:09:44.581758 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:44.581740 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" event={"ID":"f7720ad3-ccdd-42ed-87d9-1c3e783c35d8","Type":"ContainerStarted","Data":"ef99765a4b425f15078078a777bca3bcd5b9c0efb16d5b2d38c35300310cc417"} Sep 08 15:09:44.581953 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:44.581933 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:09:44.583375 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:44.583357 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:09:44.598992 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:44.598936 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pj4qx" podStartSLOduration=2.164422191 podStartE2EDuration="10.598919604s" totalImagesPullingTime="8.434497413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:09:44.598749159 +0000 UTC m=+46.750927742" watchObservedRunningTime="2026-09-08 15:09:44.598919604 +0000 UTC m=+46.751098182" Sep 08 15:09:44.622276 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:44.622234 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" podStartSLOduration=2.69278377 podStartE2EDuration="11.622221654s" totalImagesPullingTime="8.929437884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:09:44.621580112 +0000 UTC m=+46.773758705" watchObservedRunningTime="2026-09-08 15:09:44.622221654 +0000 UTC m=+46.774400228" Sep 08 15:09:44.644028 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:44.643990 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" podStartSLOduration=3.164806359 podStartE2EDuration="11.643978393s" totalImagesPullingTime="8.479172034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:09:44.642426649 +0000 UTC m=+46.794605229" watchObservedRunningTime="2026-09-08 15:09:44.643978393 +0000 UTC m=+46.796156973" Sep 08 15:09:44.660026 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:44.659980 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2" podStartSLOduration=3.123980489 podStartE2EDuration="11.659967698s" totalImagesPullingTime="8.535987209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:09:44.659505598 +0000 UTC m=+46.811684176" watchObservedRunningTime="2026-09-08 15:09:44.659967698 +0000 UTC m=+46.812146279" Sep 08 15:09:50.571879 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:50.571833 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:09:50.572430 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:50.571885 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:09:50.572430 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:50.571997 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 15:09:50.572430 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:50.572039 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 15:09:50.572430 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:50.572057 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7974bc45c6-wptrz: secret "image-registry-tls" not found Sep 08 15:09:50.572430 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:50.572072 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls podName:266dc55e-c863-43b4-93ee-e4b5fc671954 nodeName:}" failed. No retries permitted until 2026-09-08 15:10:06.572054267 +0000 UTC m=+68.724232826 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls") pod "dns-default-2hz45" (UID: "266dc55e-c863-43b4-93ee-e4b5fc671954") : secret "dns-default-metrics-tls" not found Sep 08 15:09:50.572430 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:09:50.572140 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls podName:6cc295f3-9a80-4e2c-82b1-b8d703944f1f nodeName:}" failed. No retries permitted until 2026-09-08 15:10:06.572120915 +0000 UTC m=+68.724299512 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls") pod "image-registry-7974bc45c6-wptrz" (UID: "6cc295f3-9a80-4e2c-82b1-b8d703944f1f") : secret "image-registry-tls" not found Sep 08 15:09:59.521577 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:09:59.521548 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vkmfv" Sep 08 15:10:04.058095 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:04.058056 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret\") pod \"global-pull-secret-syncer-9b44p\" (UID: \"f755fdd9-db6c-43ec-bd91-5d0d861aae73\") " pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:10:04.060770 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:04.060749 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 15:10:04.072496 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:04.072470 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f755fdd9-db6c-43ec-bd91-5d0d861aae73-original-pull-secret\") pod \"global-pull-secret-syncer-9b44p\" (UID: \"f755fdd9-db6c-43ec-bd91-5d0d861aae73\") " pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:10:04.158848 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:04.158811 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs\") pod \"network-metrics-daemon-n5x8x\" (UID: \"1dafedb8-50fc-4eff-a5a8-ee1be3e3f751\") " pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:10:04.161374 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:04.161357 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 15:10:04.169459 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:10:04.169438 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 08 15:10:04.169554 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:10:04.169542 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs podName:1dafedb8-50fc-4eff-a5a8-ee1be3e3f751 nodeName:}" failed. No retries permitted until 2026-09-08 15:11:08.169521883 +0000 UTC m=+130.321700441 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs") pod "network-metrics-daemon-n5x8x" (UID: "1dafedb8-50fc-4eff-a5a8-ee1be3e3f751") : secret "metrics-daemon-secret" not found Sep 08 15:10:04.259342 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:04.259309 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfv7\" (UniqueName: \"kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7\") pod \"network-check-target-mpw6h\" (UID: \"ca0287ff-6de3-46ba-b97f-259d5477f07c\") " pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:10:04.261826 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:04.261806 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 15:10:04.272040 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:04.272019 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 15:10:04.282584 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:04.282557 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgfv7\" (UniqueName: \"kubernetes.io/projected/ca0287ff-6de3-46ba-b97f-259d5477f07c-kube-api-access-hgfv7\") pod \"network-check-target-mpw6h\" (UID: \"ca0287ff-6de3-46ba-b97f-259d5477f07c\") " pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:10:04.284281 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:04.284261 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9b44p" Sep 08 15:10:04.296383 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:04.296360 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lf7xf\"" Sep 08 15:10:04.304477 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:04.304457 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:10:04.566776 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:04.566697 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9b44p"] Sep 08 15:10:04.570379 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:10:04.570354 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf755fdd9_db6c_43ec_bd91_5d0d861aae73.slice/crio-3bfd35d8149da89fc362560a25a8dbfa10e21efb2e3898dc2bf01550b2f8be17 WatchSource:0}: Error finding container 3bfd35d8149da89fc362560a25a8dbfa10e21efb2e3898dc2bf01550b2f8be17: Status 404 returned error can't find the container with id 3bfd35d8149da89fc362560a25a8dbfa10e21efb2e3898dc2bf01550b2f8be17 Sep 08 15:10:04.585331 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:04.585218 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mpw6h"] Sep 08 15:10:04.587385 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:10:04.587363 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0287ff_6de3_46ba_b97f_259d5477f07c.slice/crio-45c6c8f6d34fdc8a9dfa78ae81d23aaaa5222d4d9ed646f81ab134da8aa1a2a6 WatchSource:0}: Error finding container 45c6c8f6d34fdc8a9dfa78ae81d23aaaa5222d4d9ed646f81ab134da8aa1a2a6: Status 404 returned error can't find the container with id 45c6c8f6d34fdc8a9dfa78ae81d23aaaa5222d4d9ed646f81ab134da8aa1a2a6 Sep 08 15:10:04.643314 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:04.643281 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9b44p" event={"ID":"f755fdd9-db6c-43ec-bd91-5d0d861aae73","Type":"ContainerStarted","Data":"3bfd35d8149da89fc362560a25a8dbfa10e21efb2e3898dc2bf01550b2f8be17"} Sep 08 15:10:04.644352 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:04.644326 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mpw6h" event={"ID":"ca0287ff-6de3-46ba-b97f-259d5477f07c","Type":"ContainerStarted","Data":"45c6c8f6d34fdc8a9dfa78ae81d23aaaa5222d4d9ed646f81ab134da8aa1a2a6"} Sep 08 15:10:06.575728 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:06.575692 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:10:06.576188 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:06.575748 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:10:06.576188 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:10:06.575840 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 15:10:06.576188 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:10:06.575878 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 15:10:06.576188 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:10:06.575892 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7974bc45c6-wptrz: secret "image-registry-tls" not found Sep 08 15:10:06.576188 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:10:06.575912 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls podName:266dc55e-c863-43b4-93ee-e4b5fc671954 nodeName:}" failed. No retries permitted until 2026-09-08 15:10:38.575891586 +0000 UTC m=+100.728070157 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls") pod "dns-default-2hz45" (UID: "266dc55e-c863-43b4-93ee-e4b5fc671954") : secret "dns-default-metrics-tls" not found Sep 08 15:10:06.576188 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:10:06.575939 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls podName:6cc295f3-9a80-4e2c-82b1-b8d703944f1f nodeName:}" failed. No retries permitted until 2026-09-08 15:10:38.575928061 +0000 UTC m=+100.728106629 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls") pod "image-registry-7974bc45c6-wptrz" (UID: "6cc295f3-9a80-4e2c-82b1-b8d703944f1f") : secret "image-registry-tls" not found Sep 08 15:10:09.664774 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:09.664682 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mpw6h" event={"ID":"ca0287ff-6de3-46ba-b97f-259d5477f07c","Type":"ContainerStarted","Data":"f0ef0da56fbcec8a536f66815900c4e6c282dd86ea50eabe951b5878431ac427"} Sep 08 15:10:09.665193 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:09.664789 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:10:09.666184 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:09.666160 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9b44p" event={"ID":"f755fdd9-db6c-43ec-bd91-5d0d861aae73","Type":"ContainerStarted","Data":"08eb4e6286a2392eb70deccfecf0d009a0f9d32f878ff17275225bf5ca2ff983"} Sep 08 15:10:09.680532 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:09.680469 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mpw6h" podStartSLOduration=65.915872709 podStartE2EDuration="1m10.680456748s" totalImagesPullingTime="4.764584039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:10:09.679884015 +0000 UTC m=+71.832062606" watchObservedRunningTime="2026-09-08 15:10:09.680456748 +0000 UTC m=+71.832635328" Sep 08 15:10:09.695019 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:09.694979 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9b44p" podStartSLOduration=65.909440874 podStartE2EDuration="1m10.694968362s" totalImagesPullingTime="4.785527488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:10:09.694645349 +0000 UTC m=+71.846823933" watchObservedRunningTime="2026-09-08 15:10:09.694968362 +0000 UTC m=+71.847146941" Sep 08 15:10:38.589863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:38.589828 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:10:38.589863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:38.589871 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:10:38.590298 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:10:38.589983 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 15:10:38.590298 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:10:38.590046 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls podName:266dc55e-c863-43b4-93ee-e4b5fc671954 nodeName:}" failed. No retries permitted until 2026-09-08 15:11:42.590031202 +0000 UTC m=+164.742209759 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls") pod "dns-default-2hz45" (UID: "266dc55e-c863-43b4-93ee-e4b5fc671954") : secret "dns-default-metrics-tls" not found Sep 08 15:10:38.590298 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:10:38.589986 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 15:10:38.590298 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:10:38.590080 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7974bc45c6-wptrz: secret "image-registry-tls" not found Sep 08 15:10:38.590298 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:10:38.590127 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls podName:6cc295f3-9a80-4e2c-82b1-b8d703944f1f nodeName:}" failed. No retries permitted until 2026-09-08 15:11:42.590115791 +0000 UTC m=+164.742294363 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls") pod "image-registry-7974bc45c6-wptrz" (UID: "6cc295f3-9a80-4e2c-82b1-b8d703944f1f") : secret "image-registry-tls" not found Sep 08 15:10:40.672140 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:10:40.672108 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mpw6h" Sep 08 15:11:08.185765 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:11:08.185712 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs\") pod \"network-metrics-daemon-n5x8x\" (UID: \"1dafedb8-50fc-4eff-a5a8-ee1be3e3f751\") " pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:11:08.186250 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:11:08.185854 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 08 15:11:08.186250 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:11:08.185913 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs podName:1dafedb8-50fc-4eff-a5a8-ee1be3e3f751 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:10.185898518 +0000 UTC m=+252.338077076 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs") pod "network-metrics-daemon-n5x8x" (UID: "1dafedb8-50fc-4eff-a5a8-ee1be3e3f751") : secret "metrics-daemon-secret" not found Sep 08 15:11:37.772390 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:11:37.772334 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" podUID="6cc295f3-9a80-4e2c-82b1-b8d703944f1f" Sep 08 15:11:37.789973 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:11:37.789937 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-2hz45" podUID="266dc55e-c863-43b4-93ee-e4b5fc671954" Sep 08 15:11:37.937231 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:11:37.937203 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2hz45" Sep 08 15:11:37.937373 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:11:37.937203 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:11:39.403251 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:11:39.403204 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-n5x8x" podUID="1dafedb8-50fc-4eff-a5a8-ee1be3e3f751" Sep 08 15:11:42.626117 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:11:42.626071 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:11:42.626117 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:11:42.626122 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:11:42.626582 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:11:42.626225 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 15:11:42.626582 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:11:42.626224 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 15:11:42.626582 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:11:42.626237 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7974bc45c6-wptrz: secret "image-registry-tls" not found Sep 08 15:11:42.626582 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:11:42.626302 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls podName:266dc55e-c863-43b4-93ee-e4b5fc671954 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:44.626284262 +0000 UTC m=+286.778462839 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls") pod "dns-default-2hz45" (UID: "266dc55e-c863-43b4-93ee-e4b5fc671954") : secret "dns-default-metrics-tls" not found Sep 08 15:11:42.626582 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:11:42.626378 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls podName:6cc295f3-9a80-4e2c-82b1-b8d703944f1f nodeName:}" failed. No retries permitted until 2026-09-08 15:13:44.626362808 +0000 UTC m=+286.778541366 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls") pod "image-registry-7974bc45c6-wptrz" (UID: "6cc295f3-9a80-4e2c-82b1-b8d703944f1f") : secret "image-registry-tls" not found Sep 08 15:11:44.582813 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:11:44.582744 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" podUID="f7720ad3-ccdd-42ed-87d9-1c3e783c35d8" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.8:8000/readyz\": dial tcp 10.132.0.8:8000: connect: connection refused" Sep 08 15:11:44.962095 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:11:44.962055 2572 generic.go:356] "Generic (PLEG): container finished" podID="cabea983-2f0a-4119-a45a-85eccf8fbd14" containerID="b12015b0dc166604b22f89b1dc73ff17e2103316f27f8525f9749fa638d6e8a7" exitCode=255 Sep 08 15:11:44.962313 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:11:44.962136 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2" event={"ID":"cabea983-2f0a-4119-a45a-85eccf8fbd14","Type":"ContainerDied","Data":"b12015b0dc166604b22f89b1dc73ff17e2103316f27f8525f9749fa638d6e8a7"} Sep 08 15:11:44.962598 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:11:44.962572 2572 scope.go:117] "RemoveContainer" containerID="b12015b0dc166604b22f89b1dc73ff17e2103316f27f8525f9749fa638d6e8a7" Sep 08 15:11:44.963982 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:11:44.963958 2572 generic.go:356] "Generic (PLEG): container finished" podID="f7720ad3-ccdd-42ed-87d9-1c3e783c35d8" containerID="ef99765a4b425f15078078a777bca3bcd5b9c0efb16d5b2d38c35300310cc417" exitCode=1 Sep 08 15:11:44.964063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:11:44.964022 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" event={"ID":"f7720ad3-ccdd-42ed-87d9-1c3e783c35d8","Type":"ContainerDied","Data":"ef99765a4b425f15078078a777bca3bcd5b9c0efb16d5b2d38c35300310cc417"} Sep 08 15:11:44.964342 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:11:44.964327 2572 scope.go:117] "RemoveContainer" containerID="ef99765a4b425f15078078a777bca3bcd5b9c0efb16d5b2d38c35300310cc417" Sep 08 15:11:44.998853 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:11:44.998815 2572 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2" Sep 08 15:11:45.079463 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:11:45.079426 2572 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:11:45.969145 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:11:45.969101 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2" event={"ID":"cabea983-2f0a-4119-a45a-85eccf8fbd14","Type":"ContainerStarted","Data":"f44782cc5b690892841e15b070734686d009d8c78c915526454238013cffb7d1"} Sep 08 15:11:45.970993 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:11:45.970965 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" event={"ID":"f7720ad3-ccdd-42ed-87d9-1c3e783c35d8","Type":"ContainerStarted","Data":"3f563cafbce703815a2d9f5b73e6fd1fe49ac5c5f44fff156fc11832520842b5"} Sep 08 15:11:45.971118 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:11:45.971044 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:11:45.972133 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:11:45.972110 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:11:50.366259 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:11:50.366175 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:12:56.644941 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.644902 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-65p5d"] Sep 08 15:12:56.647782 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.647767 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:12:56.652542 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.652523 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 08 15:12:56.653069 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.652845 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 08 15:12:56.653069 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.652862 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 08 15:12:56.653069 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.652882 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 08 15:12:56.653069 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.652899 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qrxjt\"" Sep 08 15:12:56.662695 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.662673 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-65p5d"] Sep 08 15:12:56.778975 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.778936 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ccff15-f07b-4c80-8f00-96b2d7cd2888-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:12:56.779149 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.778989 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb28g\" (UniqueName: \"kubernetes.io/projected/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-api-access-vb28g\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:12:56.779149 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.779056 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69ccff15-f07b-4c80-8f00-96b2d7cd2888-data-volume\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:12:56.779149 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.779114 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69ccff15-f07b-4c80-8f00-96b2d7cd2888-crio-socket\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:12:56.779149 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.779133 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:12:56.879964 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.879924 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb28g\" (UniqueName: \"kubernetes.io/projected/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-api-access-vb28g\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:12:56.879964 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.879977 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69ccff15-f07b-4c80-8f00-96b2d7cd2888-data-volume\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:12:56.880183 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.880015 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69ccff15-f07b-4c80-8f00-96b2d7cd2888-crio-socket\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:12:56.880183 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.880034 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:12:56.880183 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.880067 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ccff15-f07b-4c80-8f00-96b2d7cd2888-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:12:56.880183 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.880141 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69ccff15-f07b-4c80-8f00-96b2d7cd2888-crio-socket\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:12:56.880183 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:12:56.880153 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 15:12:56.880183 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:12:56.880178 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 15:12:56.880368 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:12:56.880243 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ccff15-f07b-4c80-8f00-96b2d7cd2888-insights-runtime-extractor-tls podName:69ccff15-f07b-4c80-8f00-96b2d7cd2888 nodeName:}" failed. No retries permitted until 2026-09-08 15:12:57.380225075 +0000 UTC m=+239.532403655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69ccff15-f07b-4c80-8f00-96b2d7cd2888-insights-runtime-extractor-tls") pod "insights-runtime-extractor-65p5d" (UID: "69ccff15-f07b-4c80-8f00-96b2d7cd2888") : secret "insights-runtime-extractor-tls" not found Sep 08 15:12:56.880368 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:12:56.880258 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-rbac-proxy-cm podName:69ccff15-f07b-4c80-8f00-96b2d7cd2888 nodeName:}" failed. No retries permitted until 2026-09-08 15:12:57.38025133 +0000 UTC m=+239.532429888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-rbac-proxy-cm") pod "insights-runtime-extractor-65p5d" (UID: "69ccff15-f07b-4c80-8f00-96b2d7cd2888") : configmap "kube-rbac-proxy" not found Sep 08 15:12:56.880442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.880430 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69ccff15-f07b-4c80-8f00-96b2d7cd2888-data-volume\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:12:56.889055 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:56.889026 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb28g\" (UniqueName: \"kubernetes.io/projected/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-api-access-vb28g\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:12:57.384014 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:57.383943 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ccff15-f07b-4c80-8f00-96b2d7cd2888-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:12:57.384425 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:57.384400 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:12:57.384724 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:12:57.384074 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 15:12:57.384906 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:12:57.384894 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ccff15-f07b-4c80-8f00-96b2d7cd2888-insights-runtime-extractor-tls podName:69ccff15-f07b-4c80-8f00-96b2d7cd2888 nodeName:}" failed. No retries permitted until 2026-09-08 15:12:58.384873463 +0000 UTC m=+240.537052039 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69ccff15-f07b-4c80-8f00-96b2d7cd2888-insights-runtime-extractor-tls") pod "insights-runtime-extractor-65p5d" (UID: "69ccff15-f07b-4c80-8f00-96b2d7cd2888") : secret "insights-runtime-extractor-tls" not found Sep 08 15:12:57.385408 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:12:57.384706 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 15:12:57.385599 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:12:57.385587 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-rbac-proxy-cm podName:69ccff15-f07b-4c80-8f00-96b2d7cd2888 nodeName:}" failed. No retries permitted until 2026-09-08 15:12:58.385568695 +0000 UTC m=+240.537747269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-rbac-proxy-cm") pod "insights-runtime-extractor-65p5d" (UID: "69ccff15-f07b-4c80-8f00-96b2d7cd2888") : configmap "kube-rbac-proxy" not found Sep 08 15:12:58.393206 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:58.393172 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ccff15-f07b-4c80-8f00-96b2d7cd2888-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:12:58.393594 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:58.393249 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:12:58.393594 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:12:58.393314 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 15:12:58.393594 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:12:58.393317 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 15:12:58.393594 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:12:58.393375 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-rbac-proxy-cm podName:69ccff15-f07b-4c80-8f00-96b2d7cd2888 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:00.393360137 +0000 UTC m=+242.545538695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-rbac-proxy-cm") pod "insights-runtime-extractor-65p5d" (UID: "69ccff15-f07b-4c80-8f00-96b2d7cd2888") : configmap "kube-rbac-proxy" not found Sep 08 15:12:58.393594 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:12:58.393407 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ccff15-f07b-4c80-8f00-96b2d7cd2888-insights-runtime-extractor-tls podName:69ccff15-f07b-4c80-8f00-96b2d7cd2888 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:00.393393882 +0000 UTC m=+242.545572440 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69ccff15-f07b-4c80-8f00-96b2d7cd2888-insights-runtime-extractor-tls") pod "insights-runtime-extractor-65p5d" (UID: "69ccff15-f07b-4c80-8f00-96b2d7cd2888") : secret "insights-runtime-extractor-tls" not found Sep 08 15:12:58.813649 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:58.813622 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zxbnk_1dfce943-0703-4e7d-b479-a7616aa67993/dns-node-resolver/0.log" Sep 08 15:12:59.810940 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:12:59.810911 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wjg4s_90fade35-fbac-48dd-834c-919ef54da378/node-ca/0.log" Sep 08 15:13:00.410272 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:00.410228 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:13:00.410516 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:00.410289 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ccff15-f07b-4c80-8f00-96b2d7cd2888-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:13:00.410516 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:13:00.410385 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 15:13:00.410516 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:13:00.410409 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 15:13:00.410516 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:13:00.410468 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-rbac-proxy-cm podName:69ccff15-f07b-4c80-8f00-96b2d7cd2888 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:04.410446976 +0000 UTC m=+246.562625534 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-rbac-proxy-cm") pod "insights-runtime-extractor-65p5d" (UID: "69ccff15-f07b-4c80-8f00-96b2d7cd2888") : configmap "kube-rbac-proxy" not found Sep 08 15:13:00.410516 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:13:00.410487 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ccff15-f07b-4c80-8f00-96b2d7cd2888-insights-runtime-extractor-tls podName:69ccff15-f07b-4c80-8f00-96b2d7cd2888 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:04.410478542 +0000 UTC m=+246.562657100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69ccff15-f07b-4c80-8f00-96b2d7cd2888-insights-runtime-extractor-tls") pod "insights-runtime-extractor-65p5d" (UID: "69ccff15-f07b-4c80-8f00-96b2d7cd2888") : secret "insights-runtime-extractor-tls" not found Sep 08 15:13:00.609253 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:00.609202 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pj4qx_ee735889-613b-4a94-a579-596fabddb360/serve-healthcheck-canary/0.log" Sep 08 15:13:04.440393 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:04.440357 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:13:04.440875 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:04.440428 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ccff15-f07b-4c80-8f00-96b2d7cd2888-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:13:04.440875 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:13:04.440483 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 15:13:04.440875 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:13:04.440577 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-rbac-proxy-cm podName:69ccff15-f07b-4c80-8f00-96b2d7cd2888 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:12.440560283 +0000 UTC m=+254.592738864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-rbac-proxy-cm") pod "insights-runtime-extractor-65p5d" (UID: "69ccff15-f07b-4c80-8f00-96b2d7cd2888") : configmap "kube-rbac-proxy" not found Sep 08 15:13:04.440875 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:13:04.440590 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 15:13:04.440875 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:13:04.440642 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69ccff15-f07b-4c80-8f00-96b2d7cd2888-insights-runtime-extractor-tls podName:69ccff15-f07b-4c80-8f00-96b2d7cd2888 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:12.440626578 +0000 UTC m=+254.592805153 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69ccff15-f07b-4c80-8f00-96b2d7cd2888-insights-runtime-extractor-tls") pod "insights-runtime-extractor-65p5d" (UID: "69ccff15-f07b-4c80-8f00-96b2d7cd2888") : secret "insights-runtime-extractor-tls" not found Sep 08 15:13:10.285147 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:10.285102 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs\") pod \"network-metrics-daemon-n5x8x\" (UID: \"1dafedb8-50fc-4eff-a5a8-ee1be3e3f751\") " pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:13:10.287758 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:10.287732 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dafedb8-50fc-4eff-a5a8-ee1be3e3f751-metrics-certs\") pod \"network-metrics-daemon-n5x8x\" (UID: \"1dafedb8-50fc-4eff-a5a8-ee1be3e3f751\") " pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:13:10.473421 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:10.470239 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t2q2k\"" Sep 08 15:13:10.477829 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:10.477796 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n5x8x" Sep 08 15:13:10.748039 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:10.748008 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n5x8x"] Sep 08 15:13:10.751182 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:13:10.751155 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dafedb8_50fc_4eff_a5a8_ee1be3e3f751.slice/crio-23323994c7f6a26ec17afa82632c1d1b74035c563ff9a08ff7be2fe60ef15eaf WatchSource:0}: Error finding container 23323994c7f6a26ec17afa82632c1d1b74035c563ff9a08ff7be2fe60ef15eaf: Status 404 returned error can't find the container with id 23323994c7f6a26ec17afa82632c1d1b74035c563ff9a08ff7be2fe60ef15eaf Sep 08 15:13:11.228518 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:11.228395 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n5x8x" event={"ID":"1dafedb8-50fc-4eff-a5a8-ee1be3e3f751","Type":"ContainerStarted","Data":"23323994c7f6a26ec17afa82632c1d1b74035c563ff9a08ff7be2fe60ef15eaf"} Sep 08 15:13:12.233924 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:12.233895 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n5x8x" event={"ID":"1dafedb8-50fc-4eff-a5a8-ee1be3e3f751","Type":"ContainerStarted","Data":"5605a7e90544f3cd61655d5e23e6beaa086db8508e2d8d461609b017991328da"} Sep 08 15:13:12.504758 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:12.504721 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:13:12.504937 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:12.504780 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ccff15-f07b-4c80-8f00-96b2d7cd2888-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:13:12.505262 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:12.505238 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69ccff15-f07b-4c80-8f00-96b2d7cd2888-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:13:12.507239 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:12.507218 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69ccff15-f07b-4c80-8f00-96b2d7cd2888-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-65p5d\" (UID: \"69ccff15-f07b-4c80-8f00-96b2d7cd2888\") " pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:13:12.559586 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:12.559549 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qrxjt\"" Sep 08 15:13:12.568325 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:12.568296 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-65p5d" Sep 08 15:13:12.842452 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:12.842373 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-65p5d"] Sep 08 15:13:12.845648 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:13:12.845616 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ccff15_f07b_4c80_8f00_96b2d7cd2888.slice/crio-1391e6752fd1a0092cd8bbf6a64c908c78beaeb0e3621bec188577300c75cf2e WatchSource:0}: Error finding container 1391e6752fd1a0092cd8bbf6a64c908c78beaeb0e3621bec188577300c75cf2e: Status 404 returned error can't find the container with id 1391e6752fd1a0092cd8bbf6a64c908c78beaeb0e3621bec188577300c75cf2e Sep 08 15:13:13.238841 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:13.238741 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n5x8x" event={"ID":"1dafedb8-50fc-4eff-a5a8-ee1be3e3f751","Type":"ContainerStarted","Data":"71e5a6383afabd6f1747033f68a7d07e84c44777725e2e45115da0781891c9fb"} Sep 08 15:13:13.240244 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:13.240202 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-65p5d" event={"ID":"69ccff15-f07b-4c80-8f00-96b2d7cd2888","Type":"ContainerStarted","Data":"036c49d48a89846112e35bfbe62895c4d6e97958122be421a7bdafaf5ff1de2b"} Sep 08 15:13:13.240376 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:13.240266 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-65p5d" event={"ID":"69ccff15-f07b-4c80-8f00-96b2d7cd2888","Type":"ContainerStarted","Data":"1391e6752fd1a0092cd8bbf6a64c908c78beaeb0e3621bec188577300c75cf2e"} Sep 08 15:13:13.255309 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:13.255254 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n5x8x" podStartSLOduration=252.875918914 podStartE2EDuration="4m14.255236583s" totalImagesPullingTime="1.379317669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:13.254216939 +0000 UTC m=+255.406395520" watchObservedRunningTime="2026-09-08 15:13:13.255236583 +0000 UTC m=+255.407415202" Sep 08 15:13:14.245113 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:14.245079 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-65p5d" event={"ID":"69ccff15-f07b-4c80-8f00-96b2d7cd2888","Type":"ContainerStarted","Data":"d6c57c499a6cd91efabdb27c57abbc81a055d4d85413c8dfbf33d88ebcf357f4"} Sep 08 15:13:16.255395 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:16.255365 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-65p5d" event={"ID":"69ccff15-f07b-4c80-8f00-96b2d7cd2888","Type":"ContainerStarted","Data":"8de07b6914b4b7b69ac80c7e21020b4983f20d62d17819da7d12f234cef0f287"} Sep 08 15:13:16.272847 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:16.272797 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-65p5d" podStartSLOduration=17.954848468 podStartE2EDuration="20.272782045s" totalImagesPullingTime="2.317933577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:12:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:16.27246444 +0000 UTC m=+258.424643031" watchObservedRunningTime="2026-09-08 15:13:16.272782045 +0000 UTC m=+258.424960621" Sep 08 15:13:25.085767 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:25.085689 2572 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" podUID="79ffcd2e-02fe-404a-b0a5-e818335504e1" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 08 15:13:27.579152 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.579111 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2rlks"] Sep 08 15:13:27.582294 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.582276 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.584571 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.584551 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 08 15:13:27.584694 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.584612 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 08 15:13:27.584694 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.584660 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 08 15:13:27.584816 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.584756 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5v7mq\"" Sep 08 15:13:27.585317 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.585297 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 08 15:13:27.585415 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.585318 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 08 15:13:27.585415 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.585359 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 08 15:13:27.618414 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.618375 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1c2d30a3-ba64-418d-8d60-947b151bd033-node-exporter-wtmp\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.618600 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.618423 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1c2d30a3-ba64-418d-8d60-947b151bd033-node-exporter-textfile\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.618600 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.618470 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c2d30a3-ba64-418d-8d60-947b151bd033-sys\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.618600 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.618522 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1c2d30a3-ba64-418d-8d60-947b151bd033-node-exporter-tls\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.618699 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.618592 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1c2d30a3-ba64-418d-8d60-947b151bd033-root\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.618699 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.618627 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c2d30a3-ba64-418d-8d60-947b151bd033-metrics-client-ca\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.618699 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.618646 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c2d30a3-ba64-418d-8d60-947b151bd033-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.618788 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.618693 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztflf\" (UniqueName: \"kubernetes.io/projected/1c2d30a3-ba64-418d-8d60-947b151bd033-kube-api-access-ztflf\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.618788 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.618728 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1c2d30a3-ba64-418d-8d60-947b151bd033-node-exporter-accelerators-collector-config\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.719480 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.719442 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztflf\" (UniqueName: \"kubernetes.io/projected/1c2d30a3-ba64-418d-8d60-947b151bd033-kube-api-access-ztflf\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.719666 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.719486 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1c2d30a3-ba64-418d-8d60-947b151bd033-node-exporter-accelerators-collector-config\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.719666 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.719549 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1c2d30a3-ba64-418d-8d60-947b151bd033-node-exporter-wtmp\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.719666 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.719580 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1c2d30a3-ba64-418d-8d60-947b151bd033-node-exporter-textfile\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.719666 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.719605 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c2d30a3-ba64-418d-8d60-947b151bd033-sys\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.719666 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.719621 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1c2d30a3-ba64-418d-8d60-947b151bd033-node-exporter-tls\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.719666 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.719657 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1c2d30a3-ba64-418d-8d60-947b151bd033-root\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.719927 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.719686 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c2d30a3-ba64-418d-8d60-947b151bd033-metrics-client-ca\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.719927 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.719707 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c2d30a3-ba64-418d-8d60-947b151bd033-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.719927 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.719712 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c2d30a3-ba64-418d-8d60-947b151bd033-sys\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.719927 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.719761 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1c2d30a3-ba64-418d-8d60-947b151bd033-node-exporter-wtmp\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.719927 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.719773 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1c2d30a3-ba64-418d-8d60-947b151bd033-root\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.719927 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:13:27.719838 2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 08 15:13:27.719927 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.719891 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1c2d30a3-ba64-418d-8d60-947b151bd033-node-exporter-textfile\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.719927 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:13:27.719914 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c2d30a3-ba64-418d-8d60-947b151bd033-node-exporter-tls podName:1c2d30a3-ba64-418d-8d60-947b151bd033 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:28.219893209 +0000 UTC m=+270.372071786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1c2d30a3-ba64-418d-8d60-947b151bd033-node-exporter-tls") pod "node-exporter-2rlks" (UID: "1c2d30a3-ba64-418d-8d60-947b151bd033") : secret "node-exporter-tls" not found Sep 08 15:13:27.720178 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.720095 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1c2d30a3-ba64-418d-8d60-947b151bd033-node-exporter-accelerators-collector-config\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.720221 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.720205 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c2d30a3-ba64-418d-8d60-947b151bd033-metrics-client-ca\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.722137 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.722114 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c2d30a3-ba64-418d-8d60-947b151bd033-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:27.727889 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:27.727867 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztflf\" (UniqueName: \"kubernetes.io/projected/1c2d30a3-ba64-418d-8d60-947b151bd033-kube-api-access-ztflf\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:28.224552 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:28.224516 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1c2d30a3-ba64-418d-8d60-947b151bd033-node-exporter-tls\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:28.226945 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:28.226910 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1c2d30a3-ba64-418d-8d60-947b151bd033-node-exporter-tls\") pod \"node-exporter-2rlks\" (UID: \"1c2d30a3-ba64-418d-8d60-947b151bd033\") " pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:28.491588 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:28.491503 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2rlks" Sep 08 15:13:28.502180 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:13:28.502145 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2d30a3_ba64_418d_8d60_947b151bd033.slice/crio-e4077354421564f402cc7bbc168263468bf086127203c0cb893f486f93e54f6b WatchSource:0}: Error finding container e4077354421564f402cc7bbc168263468bf086127203c0cb893f486f93e54f6b: Status 404 returned error can't find the container with id e4077354421564f402cc7bbc168263468bf086127203c0cb893f486f93e54f6b Sep 08 15:13:29.295825 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:29.295789 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2rlks" event={"ID":"1c2d30a3-ba64-418d-8d60-947b151bd033","Type":"ContainerStarted","Data":"e4077354421564f402cc7bbc168263468bf086127203c0cb893f486f93e54f6b"} Sep 08 15:13:30.300834 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:30.300800 2572 generic.go:356] "Generic (PLEG): container finished" podID="1c2d30a3-ba64-418d-8d60-947b151bd033" containerID="06278ca23eaa38eb25449358c81b92bb77a6e4d06ad0bd3a425bc1b6cbbf3abe" exitCode=0 Sep 08 15:13:30.301228 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:30.300842 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2rlks" event={"ID":"1c2d30a3-ba64-418d-8d60-947b151bd033","Type":"ContainerDied","Data":"06278ca23eaa38eb25449358c81b92bb77a6e4d06ad0bd3a425bc1b6cbbf3abe"} Sep 08 15:13:31.305965 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:31.305934 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2rlks" event={"ID":"1c2d30a3-ba64-418d-8d60-947b151bd033","Type":"ContainerStarted","Data":"5b13e58f91a544b4cddea88605ac4239fd54a16d1aec8909df5ff1e815004d28"} Sep 08 15:13:31.306340 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:31.305986 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2rlks" event={"ID":"1c2d30a3-ba64-418d-8d60-947b151bd033","Type":"ContainerStarted","Data":"a6e97a6cefe652e7583a30407a0b5b3a99ee8e69a95659f40c7fdabb7539eecd"} Sep 08 15:13:31.325860 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:31.325807 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2rlks" podStartSLOduration=3.084033217 podStartE2EDuration="4.325790428s" totalImagesPullingTime="1.173691865s" totalInitContainerRuntime="68.065346ms" isStatefulPod=true podCreationTimestamp="2026-09-08 15:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:31.324402363 +0000 UTC m=+273.476580944" watchObservedRunningTime="2026-09-08 15:13:31.325790428 +0000 UTC m=+273.477969007" Sep 08 15:13:35.084956 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:35.084907 2572 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" podUID="79ffcd2e-02fe-404a-b0a5-e818335504e1" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 08 15:13:40.937894 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:13:40.937856 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" podUID="6cc295f3-9a80-4e2c-82b1-b8d703944f1f" Sep 08 15:13:40.937894 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:13:40.937852 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-2hz45" podUID="266dc55e-c863-43b4-93ee-e4b5fc671954" Sep 08 15:13:41.338025 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:41.337998 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:13:41.338257 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:41.338032 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2hz45" Sep 08 15:13:44.658628 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:44.658586 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:13:44.659004 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:44.658648 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:13:44.661221 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:44.661196 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/266dc55e-c863-43b4-93ee-e4b5fc671954-metrics-tls\") pod \"dns-default-2hz45\" (UID: \"266dc55e-c863-43b4-93ee-e4b5fc671954\") " pod="openshift-dns/dns-default-2hz45" Sep 08 15:13:44.661294 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:44.661255 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls\") pod \"image-registry-7974bc45c6-wptrz\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:13:44.941648 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:44.941562 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cc4mw\"" Sep 08 15:13:44.941648 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:44.941571 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2m6vp\"" Sep 08 15:13:44.949419 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:44.949393 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2hz45" Sep 08 15:13:44.949419 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:44.949420 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:13:45.079249 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.079213 2572 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" podUID="f7720ad3-ccdd-42ed-87d9-1c3e783c35d8" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.8:8000/healthz\": dial tcp 10.132.0.8:8000: connect: connection refused" Sep 08 15:13:45.085247 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.085219 2572 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" podUID="79ffcd2e-02fe-404a-b0a5-e818335504e1" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 08 15:13:45.085339 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.085288 2572 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" Sep 08 15:13:45.085815 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.085785 2572 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"c3f71727cab6d4fa7f17ca30b9aa452200d564a4d56d555e20db9e0f6e87a0b3"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" containerMessage="Container service-proxy failed liveness probe, will be restarted" Sep 08 15:13:45.085855 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.085837 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" podUID="79ffcd2e-02fe-404a-b0a5-e818335504e1" containerName="service-proxy" containerID="cri-o://c3f71727cab6d4fa7f17ca30b9aa452200d564a4d56d555e20db9e0f6e87a0b3" gracePeriod=30 Sep 08 15:13:45.228120 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.228094 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2hz45"] Sep 08 15:13:45.231442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.231204 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7974bc45c6-wptrz"] Sep 08 15:13:45.234116 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:13:45.234057 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266dc55e_c863_43b4_93ee_e4b5fc671954.slice/crio-57f6330da3ed04fd3ea0ddbe59772d5c2853a8ccb743285890c4cc3663e97ec3 WatchSource:0}: Error finding container 57f6330da3ed04fd3ea0ddbe59772d5c2853a8ccb743285890c4cc3663e97ec3: Status 404 returned error can't find the container with id 57f6330da3ed04fd3ea0ddbe59772d5c2853a8ccb743285890c4cc3663e97ec3 Sep 08 15:13:45.236692 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:13:45.236671 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc295f3_9a80_4e2c_82b1_b8d703944f1f.slice/crio-3e5f0fd08b5e3db49e37ff72c92d01402fbeb63b72cd821902e92b460f387028 WatchSource:0}: Error finding container 3e5f0fd08b5e3db49e37ff72c92d01402fbeb63b72cd821902e92b460f387028: Status 404 returned error can't find the container with id 3e5f0fd08b5e3db49e37ff72c92d01402fbeb63b72cd821902e92b460f387028 Sep 08 15:13:45.354641 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.354607 2572 generic.go:356] "Generic (PLEG): container finished" podID="cabea983-2f0a-4119-a45a-85eccf8fbd14" containerID="f44782cc5b690892841e15b070734686d009d8c78c915526454238013cffb7d1" exitCode=255 Sep 08 15:13:45.354818 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.354684 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2" event={"ID":"cabea983-2f0a-4119-a45a-85eccf8fbd14","Type":"ContainerDied","Data":"f44782cc5b690892841e15b070734686d009d8c78c915526454238013cffb7d1"} Sep 08 15:13:45.354818 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.354751 2572 scope.go:117] "RemoveContainer" containerID="b12015b0dc166604b22f89b1dc73ff17e2103316f27f8525f9749fa638d6e8a7" Sep 08 15:13:45.355121 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.355101 2572 scope.go:117] "RemoveContainer" containerID="f44782cc5b690892841e15b070734686d009d8c78c915526454238013cffb7d1" Sep 08 15:13:45.355327 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:13:45.355308 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-79cc588b94-zd8j2_open-cluster-management-agent-addon(cabea983-2f0a-4119-a45a-85eccf8fbd14)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2" podUID="cabea983-2f0a-4119-a45a-85eccf8fbd14" Sep 08 15:13:45.357371 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.357348 2572 generic.go:356] "Generic (PLEG): container finished" podID="79ffcd2e-02fe-404a-b0a5-e818335504e1" containerID="c3f71727cab6d4fa7f17ca30b9aa452200d564a4d56d555e20db9e0f6e87a0b3" exitCode=2 Sep 08 15:13:45.357476 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.357411 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" event={"ID":"79ffcd2e-02fe-404a-b0a5-e818335504e1","Type":"ContainerDied","Data":"c3f71727cab6d4fa7f17ca30b9aa452200d564a4d56d555e20db9e0f6e87a0b3"} Sep 08 15:13:45.357476 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.357458 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-66fd956cd5-vzchn" event={"ID":"79ffcd2e-02fe-404a-b0a5-e818335504e1","Type":"ContainerStarted","Data":"e38f1efe5eab9c0c2047a9d3dd4242dc25504942d181e8384c3e121dd82e3930"} Sep 08 15:13:45.359584 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.359562 2572 generic.go:356] "Generic (PLEG): container finished" podID="f7720ad3-ccdd-42ed-87d9-1c3e783c35d8" containerID="3f563cafbce703815a2d9f5b73e6fd1fe49ac5c5f44fff156fc11832520842b5" exitCode=1 Sep 08 15:13:45.359686 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.359626 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" event={"ID":"f7720ad3-ccdd-42ed-87d9-1c3e783c35d8","Type":"ContainerDied","Data":"3f563cafbce703815a2d9f5b73e6fd1fe49ac5c5f44fff156fc11832520842b5"} Sep 08 15:13:45.360001 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.359982 2572 scope.go:117] "RemoveContainer" containerID="3f563cafbce703815a2d9f5b73e6fd1fe49ac5c5f44fff156fc11832520842b5" Sep 08 15:13:45.360198 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:13:45.360176 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-787d6696f5-kr4rd_open-cluster-management-agent-addon(f7720ad3-ccdd-42ed-87d9-1c3e783c35d8)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" podUID="f7720ad3-ccdd-42ed-87d9-1c3e783c35d8" Sep 08 15:13:45.361316 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.361273 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" event={"ID":"6cc295f3-9a80-4e2c-82b1-b8d703944f1f","Type":"ContainerStarted","Data":"422643089e4048e219e8740a1abcb698a7d9fb24c53e76f0ad81a99c2d0ee5f1"} Sep 08 15:13:45.361316 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.361312 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" event={"ID":"6cc295f3-9a80-4e2c-82b1-b8d703944f1f","Type":"ContainerStarted","Data":"3e5f0fd08b5e3db49e37ff72c92d01402fbeb63b72cd821902e92b460f387028"} Sep 08 15:13:45.361473 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.361372 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:13:45.362555 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.362531 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2hz45" event={"ID":"266dc55e-c863-43b4-93ee-e4b5fc671954","Type":"ContainerStarted","Data":"57f6330da3ed04fd3ea0ddbe59772d5c2853a8ccb743285890c4cc3663e97ec3"} Sep 08 15:13:45.366340 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.366322 2572 scope.go:117] "RemoveContainer" containerID="ef99765a4b425f15078078a777bca3bcd5b9c0efb16d5b2d38c35300310cc417" Sep 08 15:13:45.390608 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.390553 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" podStartSLOduration=286.390534387 podStartE2EDuration="4m46.390534387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:45.389842968 +0000 UTC m=+287.542021548" watchObservedRunningTime="2026-09-08 15:13:45.390534387 +0000 UTC m=+287.542712966" Sep 08 15:13:45.971516 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:45.971403 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:13:46.369620 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:46.369589 2572 scope.go:117] "RemoveContainer" containerID="3f563cafbce703815a2d9f5b73e6fd1fe49ac5c5f44fff156fc11832520842b5" Sep 08 15:13:46.369777 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:13:46.369731 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-787d6696f5-kr4rd_open-cluster-management-agent-addon(f7720ad3-ccdd-42ed-87d9-1c3e783c35d8)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" podUID="f7720ad3-ccdd-42ed-87d9-1c3e783c35d8" Sep 08 15:13:47.374608 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:47.374573 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2hz45" event={"ID":"266dc55e-c863-43b4-93ee-e4b5fc671954","Type":"ContainerStarted","Data":"0e3caf283386d34df800199a0b31275e8384b8ca249ad132a31b49774de6db10"} Sep 08 15:13:47.374968 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:47.374663 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2hz45" event={"ID":"266dc55e-c863-43b4-93ee-e4b5fc671954","Type":"ContainerStarted","Data":"df953ff0d84f9b6255c285a99b72a7e29d8a04ae24459113a793bb908cc1ba30"} Sep 08 15:13:47.374968 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:47.374726 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2hz45" Sep 08 15:13:54.998744 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:54.998702 2572 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2" Sep 08 15:13:54.999132 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:54.999094 2572 scope.go:117] "RemoveContainer" containerID="f44782cc5b690892841e15b070734686d009d8c78c915526454238013cffb7d1" Sep 08 15:13:54.999273 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:13:54.999253 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-79cc588b94-zd8j2_open-cluster-management-agent-addon(cabea983-2f0a-4119-a45a-85eccf8fbd14)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2" podUID="cabea983-2f0a-4119-a45a-85eccf8fbd14" Sep 08 15:13:55.015598 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:55.015534 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2hz45" podStartSLOduration=259.288161644 podStartE2EDuration="4m21.01551555s" totalImagesPullingTime="1.727353906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:47.405745838 +0000 UTC m=+289.557924419" watchObservedRunningTime="2026-09-08 15:13:55.01551555 +0000 UTC m=+297.167694121" Sep 08 15:13:55.079475 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:55.079433 2572 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:13:55.079901 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:55.079878 2572 scope.go:117] "RemoveContainer" containerID="3f563cafbce703815a2d9f5b73e6fd1fe49ac5c5f44fff156fc11832520842b5" Sep 08 15:13:55.402628 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:55.402542 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" event={"ID":"f7720ad3-ccdd-42ed-87d9-1c3e783c35d8","Type":"ContainerStarted","Data":"ed1d982f1d3eeab7a2cca351997c19b19cca2152ab95f58b0f4a5cbf35bef42e"} Sep 08 15:13:55.402850 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:55.402830 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:13:55.403850 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:55.403824 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-787d6696f5-kr4rd" Sep 08 15:13:57.380673 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:57.380643 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2hz45" Sep 08 15:13:58.276769 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:13:58.276740 2572 kubelet.go:1648] "Image garbage collection succeeded" Sep 08 15:14:02.833388 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:02.833358 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2hz45_266dc55e-c863-43b4-93ee-e4b5fc671954/dns/0.log" Sep 08 15:14:03.033351 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:03.033307 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2hz45_266dc55e-c863-43b4-93ee-e4b5fc671954/kube-rbac-proxy/0.log" Sep 08 15:14:04.432222 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:04.432194 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zxbnk_1dfce943-0703-4e7d-b479-a7616aa67993/dns-node-resolver/0.log" Sep 08 15:14:04.832531 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:04.832480 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7974bc45c6-wptrz_6cc295f3-9a80-4e2c-82b1-b8d703944f1f/registry/0.log" Sep 08 15:14:04.954216 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:04.954180 2572 patch_prober.go:28] interesting pod/image-registry-7974bc45c6-wptrz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 08 15:14:04.954380 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:04.954270 2572 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" podUID="6cc295f3-9a80-4e2c-82b1-b8d703944f1f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 15:14:05.432333 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:05.432307 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wjg4s_90fade35-fbac-48dd-834c-919ef54da378/node-ca/0.log" Sep 08 15:14:06.231456 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:06.231427 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pj4qx_ee735889-613b-4a94-a579-596fabddb360/serve-healthcheck-canary/0.log" Sep 08 15:14:06.373538 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:06.373512 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:14:07.366485 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:07.366452 2572 scope.go:117] "RemoveContainer" containerID="f44782cc5b690892841e15b070734686d009d8c78c915526454238013cffb7d1" Sep 08 15:14:07.444508 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:07.444460 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cc588b94-zd8j2" event={"ID":"cabea983-2f0a-4119-a45a-85eccf8fbd14","Type":"ContainerStarted","Data":"906bd44fe4862701d91992216a86fdd7d614c42bd1cb51a377417b4194bcc83b"} Sep 08 15:14:10.940961 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:10.940918 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7974bc45c6-wptrz"] Sep 08 15:14:35.960825 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:35.960757 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" podUID="6cc295f3-9a80-4e2c-82b1-b8d703944f1f" containerName="registry" containerID="cri-o://422643089e4048e219e8740a1abcb698a7d9fb24c53e76f0ad81a99c2d0ee5f1" gracePeriod=30 Sep 08 15:14:36.369853 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:36.369819 2572 patch_prober.go:28] interesting pod/image-registry-7974bc45c6-wptrz container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.132.0.7:5000/healthz\": dial tcp 10.132.0.7:5000: connect: connection refused" start-of-body= Sep 08 15:14:36.370033 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:36.369866 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" podUID="6cc295f3-9a80-4e2c-82b1-b8d703944f1f" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.7:5000/healthz\": dial tcp 10.132.0.7:5000: connect: connection refused" Sep 08 15:14:37.207892 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.207863 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:14:37.238171 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.238136 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls\") pod \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " Sep 08 15:14:37.238171 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.238175 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-certificates\") pod \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " Sep 08 15:14:37.238395 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.238198 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-trusted-ca\") pod \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " Sep 08 15:14:37.238395 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.238257 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-df9jp\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-kube-api-access-df9jp\") pod \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " Sep 08 15:14:37.238395 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.238300 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-installation-pull-secrets\") pod \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " Sep 08 15:14:37.238395 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.238331 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-bound-sa-token\") pod \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " Sep 08 15:14:37.238395 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.238364 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-ca-trust-extracted\") pod \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " Sep 08 15:14:37.238631 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.238403 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-image-registry-private-configuration\") pod \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\" (UID: \"6cc295f3-9a80-4e2c-82b1-b8d703944f1f\") " Sep 08 15:14:37.238677 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.238644 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6cc295f3-9a80-4e2c-82b1-b8d703944f1f" (UID: "6cc295f3-9a80-4e2c-82b1-b8d703944f1f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:14:37.241685 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.239428 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6cc295f3-9a80-4e2c-82b1-b8d703944f1f" (UID: "6cc295f3-9a80-4e2c-82b1-b8d703944f1f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:14:37.242059 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.241977 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6cc295f3-9a80-4e2c-82b1-b8d703944f1f" (UID: "6cc295f3-9a80-4e2c-82b1-b8d703944f1f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 15:14:37.242329 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.242279 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6cc295f3-9a80-4e2c-82b1-b8d703944f1f" (UID: "6cc295f3-9a80-4e2c-82b1-b8d703944f1f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:14:37.242759 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.242720 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6cc295f3-9a80-4e2c-82b1-b8d703944f1f" (UID: "6cc295f3-9a80-4e2c-82b1-b8d703944f1f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:14:37.243149 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.243111 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-kube-api-access-df9jp" (OuterVolumeSpecName: "kube-api-access-df9jp") pod "6cc295f3-9a80-4e2c-82b1-b8d703944f1f" (UID: "6cc295f3-9a80-4e2c-82b1-b8d703944f1f"). InnerVolumeSpecName "kube-api-access-df9jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:14:37.244184 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.244158 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6cc295f3-9a80-4e2c-82b1-b8d703944f1f" (UID: "6cc295f3-9a80-4e2c-82b1-b8d703944f1f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 15:14:37.249363 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.249329 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6cc295f3-9a80-4e2c-82b1-b8d703944f1f" (UID: "6cc295f3-9a80-4e2c-82b1-b8d703944f1f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:14:37.339865 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.339766 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df9jp\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-kube-api-access-df9jp\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:14:37.339865 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.339804 2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-installation-pull-secrets\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:14:37.339865 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.339815 2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-bound-sa-token\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:14:37.339865 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.339824 2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-ca-trust-extracted\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:14:37.339865 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.339834 2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-image-registry-private-configuration\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:14:37.339865 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.339843 2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-tls\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:14:37.339865 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.339851 2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-registry-certificates\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:14:37.339865 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.339859 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6cc295f3-9a80-4e2c-82b1-b8d703944f1f-trusted-ca\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:14:37.552166 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.552132 2572 generic.go:356] "Generic (PLEG): container finished" podID="6cc295f3-9a80-4e2c-82b1-b8d703944f1f" containerID="422643089e4048e219e8740a1abcb698a7d9fb24c53e76f0ad81a99c2d0ee5f1" exitCode=0 Sep 08 15:14:37.552166 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.552178 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" Sep 08 15:14:37.552395 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.552215 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" event={"ID":"6cc295f3-9a80-4e2c-82b1-b8d703944f1f","Type":"ContainerDied","Data":"422643089e4048e219e8740a1abcb698a7d9fb24c53e76f0ad81a99c2d0ee5f1"} Sep 08 15:14:37.552395 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.552270 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7974bc45c6-wptrz" event={"ID":"6cc295f3-9a80-4e2c-82b1-b8d703944f1f","Type":"ContainerDied","Data":"3e5f0fd08b5e3db49e37ff72c92d01402fbeb63b72cd821902e92b460f387028"} Sep 08 15:14:37.552395 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.552303 2572 scope.go:117] "RemoveContainer" containerID="422643089e4048e219e8740a1abcb698a7d9fb24c53e76f0ad81a99c2d0ee5f1" Sep 08 15:14:37.561263 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.561244 2572 scope.go:117] "RemoveContainer" containerID="422643089e4048e219e8740a1abcb698a7d9fb24c53e76f0ad81a99c2d0ee5f1" Sep 08 15:14:37.561675 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:14:37.561651 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"422643089e4048e219e8740a1abcb698a7d9fb24c53e76f0ad81a99c2d0ee5f1\": container with ID starting with 422643089e4048e219e8740a1abcb698a7d9fb24c53e76f0ad81a99c2d0ee5f1 not found: ID does not exist" containerID="422643089e4048e219e8740a1abcb698a7d9fb24c53e76f0ad81a99c2d0ee5f1" Sep 08 15:14:37.561739 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.561685 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"422643089e4048e219e8740a1abcb698a7d9fb24c53e76f0ad81a99c2d0ee5f1"} err="failed to get container status \"422643089e4048e219e8740a1abcb698a7d9fb24c53e76f0ad81a99c2d0ee5f1\": rpc error: code = NotFound desc = could not find container \"422643089e4048e219e8740a1abcb698a7d9fb24c53e76f0ad81a99c2d0ee5f1\": container with ID starting with 422643089e4048e219e8740a1abcb698a7d9fb24c53e76f0ad81a99c2d0ee5f1 not found: ID does not exist" Sep 08 15:14:37.575114 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.575079 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7974bc45c6-wptrz"] Sep 08 15:14:37.579983 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:37.579946 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7974bc45c6-wptrz"] Sep 08 15:14:38.370315 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:14:38.370272 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cc295f3-9a80-4e2c-82b1-b8d703944f1f" path="/var/lib/kubelet/pods/6cc295f3-9a80-4e2c-82b1-b8d703944f1f/volumes" Sep 08 15:18:04.443563 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:04.443529 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8fllv"] Sep 08 15:18:04.444047 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:04.443754 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc295f3-9a80-4e2c-82b1-b8d703944f1f" containerName="registry" Sep 08 15:18:04.444047 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:04.443764 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc295f3-9a80-4e2c-82b1-b8d703944f1f" containerName="registry" Sep 08 15:18:04.444047 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:04.443805 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cc295f3-9a80-4e2c-82b1-b8d703944f1f" containerName="registry" Sep 08 15:18:04.446440 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:04.446425 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-8fllv" Sep 08 15:18:04.448857 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:04.448834 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-nldfr\"" Sep 08 15:18:04.448984 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:04.448834 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 08 15:18:04.449374 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:04.449361 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 08 15:18:04.456238 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:04.456217 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8fllv"] Sep 08 15:18:04.524346 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:04.524322 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fktk\" (UniqueName: \"kubernetes.io/projected/bb333e91-36cb-459f-bc8e-42e80eeedf9b-kube-api-access-6fktk\") pod \"cert-manager-55b4d954d5-8fllv\" (UID: \"bb333e91-36cb-459f-bc8e-42e80eeedf9b\") " pod="cert-manager/cert-manager-55b4d954d5-8fllv" Sep 08 15:18:04.524505 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:04.524382 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb333e91-36cb-459f-bc8e-42e80eeedf9b-bound-sa-token\") pod \"cert-manager-55b4d954d5-8fllv\" (UID: \"bb333e91-36cb-459f-bc8e-42e80eeedf9b\") " pod="cert-manager/cert-manager-55b4d954d5-8fllv" Sep 08 15:18:04.625268 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:04.625232 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb333e91-36cb-459f-bc8e-42e80eeedf9b-bound-sa-token\") pod \"cert-manager-55b4d954d5-8fllv\" (UID: \"bb333e91-36cb-459f-bc8e-42e80eeedf9b\") " pod="cert-manager/cert-manager-55b4d954d5-8fllv" Sep 08 15:18:04.625268 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:04.625266 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fktk\" (UniqueName: \"kubernetes.io/projected/bb333e91-36cb-459f-bc8e-42e80eeedf9b-kube-api-access-6fktk\") pod \"cert-manager-55b4d954d5-8fllv\" (UID: \"bb333e91-36cb-459f-bc8e-42e80eeedf9b\") " pod="cert-manager/cert-manager-55b4d954d5-8fllv" Sep 08 15:18:04.633295 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:04.633269 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb333e91-36cb-459f-bc8e-42e80eeedf9b-bound-sa-token\") pod \"cert-manager-55b4d954d5-8fllv\" (UID: \"bb333e91-36cb-459f-bc8e-42e80eeedf9b\") " pod="cert-manager/cert-manager-55b4d954d5-8fllv" Sep 08 15:18:04.633518 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:04.633476 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fktk\" (UniqueName: \"kubernetes.io/projected/bb333e91-36cb-459f-bc8e-42e80eeedf9b-kube-api-access-6fktk\") pod \"cert-manager-55b4d954d5-8fllv\" (UID: \"bb333e91-36cb-459f-bc8e-42e80eeedf9b\") " pod="cert-manager/cert-manager-55b4d954d5-8fllv" Sep 08 15:18:04.754904 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:04.754878 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-8fllv" Sep 08 15:18:05.023117 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:05.023049 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8fllv"] Sep 08 15:18:05.025790 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:18:05.025763 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb333e91_36cb_459f_bc8e_42e80eeedf9b.slice/crio-e7d7181e98b7b11925876c05659fe815d7a2288c4f5952313bfdae9ee7f3d4c4 WatchSource:0}: Error finding container e7d7181e98b7b11925876c05659fe815d7a2288c4f5952313bfdae9ee7f3d4c4: Status 404 returned error can't find the container with id e7d7181e98b7b11925876c05659fe815d7a2288c4f5952313bfdae9ee7f3d4c4 Sep 08 15:18:05.027669 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:05.027655 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 15:18:05.236085 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:05.236053 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-8fllv" event={"ID":"bb333e91-36cb-459f-bc8e-42e80eeedf9b","Type":"ContainerStarted","Data":"e7d7181e98b7b11925876c05659fe815d7a2288c4f5952313bfdae9ee7f3d4c4"} Sep 08 15:18:08.249370 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:08.249334 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-8fllv" event={"ID":"bb333e91-36cb-459f-bc8e-42e80eeedf9b","Type":"ContainerStarted","Data":"ff94130dc36dd8e42fa217ad8db70eabfb03bcef44a57b2527fe55f8e5289255"} Sep 08 15:18:08.264545 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:08.264475 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-8fllv" podStartSLOduration=1.409772548 podStartE2EDuration="4.26445988s" totalImagesPullingTime="2.854687332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:08.263575361 +0000 UTC m=+550.415753955" watchObservedRunningTime="2026-09-08 15:18:08.26445988 +0000 UTC m=+550.416638462" Sep 08 15:18:28.947375 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:28.947339 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9"] Sep 08 15:18:28.950295 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:28.950278 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" Sep 08 15:18:28.953942 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:28.953913 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 08 15:18:28.954078 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:28.953942 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 08 15:18:28.960586 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:28.960569 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 08 15:18:28.961324 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:28.961308 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 08 15:18:28.961596 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:28.961581 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jgmwt\"" Sep 08 15:18:28.995140 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:28.995103 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9"] Sep 08 15:18:29.089867 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:29.089827 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3eaa0c78-d939-42c9-b831-0f4784f26ba9-webhook-cert\") pod \"opendatahub-operator-controller-manager-66db8bdb84-4lrw9\" (UID: \"3eaa0c78-d939-42c9-b831-0f4784f26ba9\") " pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" Sep 08 15:18:29.089867 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:29.089868 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3eaa0c78-d939-42c9-b831-0f4784f26ba9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66db8bdb84-4lrw9\" (UID: \"3eaa0c78-d939-42c9-b831-0f4784f26ba9\") " pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" Sep 08 15:18:29.090075 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:29.089912 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3eaa0c78-d939-42c9-b831-0f4784f26ba9-manifests\") pod \"opendatahub-operator-controller-manager-66db8bdb84-4lrw9\" (UID: \"3eaa0c78-d939-42c9-b831-0f4784f26ba9\") " pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" Sep 08 15:18:29.090075 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:29.089949 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c5kq\" (UniqueName: \"kubernetes.io/projected/3eaa0c78-d939-42c9-b831-0f4784f26ba9-kube-api-access-2c5kq\") pod \"opendatahub-operator-controller-manager-66db8bdb84-4lrw9\" (UID: \"3eaa0c78-d939-42c9-b831-0f4784f26ba9\") " pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" Sep 08 15:18:29.090075 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:29.090019 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3eaa0c78-d939-42c9-b831-0f4784f26ba9-tmp\") pod \"opendatahub-operator-controller-manager-66db8bdb84-4lrw9\" (UID: \"3eaa0c78-d939-42c9-b831-0f4784f26ba9\") " pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" Sep 08 15:18:29.191231 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:29.191190 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c5kq\" (UniqueName: \"kubernetes.io/projected/3eaa0c78-d939-42c9-b831-0f4784f26ba9-kube-api-access-2c5kq\") pod \"opendatahub-operator-controller-manager-66db8bdb84-4lrw9\" (UID: \"3eaa0c78-d939-42c9-b831-0f4784f26ba9\") " pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" Sep 08 15:18:29.191388 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:29.191241 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3eaa0c78-d939-42c9-b831-0f4784f26ba9-tmp\") pod \"opendatahub-operator-controller-manager-66db8bdb84-4lrw9\" (UID: \"3eaa0c78-d939-42c9-b831-0f4784f26ba9\") " pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" Sep 08 15:18:29.191388 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:29.191279 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3eaa0c78-d939-42c9-b831-0f4784f26ba9-webhook-cert\") pod \"opendatahub-operator-controller-manager-66db8bdb84-4lrw9\" (UID: \"3eaa0c78-d939-42c9-b831-0f4784f26ba9\") " pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" Sep 08 15:18:29.191388 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:29.191303 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3eaa0c78-d939-42c9-b831-0f4784f26ba9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66db8bdb84-4lrw9\" (UID: \"3eaa0c78-d939-42c9-b831-0f4784f26ba9\") " pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" Sep 08 15:18:29.191535 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:29.191476 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3eaa0c78-d939-42c9-b831-0f4784f26ba9-manifests\") pod \"opendatahub-operator-controller-manager-66db8bdb84-4lrw9\" (UID: \"3eaa0c78-d939-42c9-b831-0f4784f26ba9\") " pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" Sep 08 15:18:29.191814 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:29.191791 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3eaa0c78-d939-42c9-b831-0f4784f26ba9-manifests\") pod \"opendatahub-operator-controller-manager-66db8bdb84-4lrw9\" (UID: \"3eaa0c78-d939-42c9-b831-0f4784f26ba9\") " pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" Sep 08 15:18:29.193920 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:29.193895 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3eaa0c78-d939-42c9-b831-0f4784f26ba9-tmp\") pod \"opendatahub-operator-controller-manager-66db8bdb84-4lrw9\" (UID: \"3eaa0c78-d939-42c9-b831-0f4784f26ba9\") " pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" Sep 08 15:18:29.193989 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:29.193927 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3eaa0c78-d939-42c9-b831-0f4784f26ba9-webhook-cert\") pod \"opendatahub-operator-controller-manager-66db8bdb84-4lrw9\" (UID: \"3eaa0c78-d939-42c9-b831-0f4784f26ba9\") " pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" Sep 08 15:18:29.194033 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:29.194024 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3eaa0c78-d939-42c9-b831-0f4784f26ba9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66db8bdb84-4lrw9\" (UID: \"3eaa0c78-d939-42c9-b831-0f4784f26ba9\") " pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" Sep 08 15:18:29.207655 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:29.207592 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c5kq\" (UniqueName: \"kubernetes.io/projected/3eaa0c78-d939-42c9-b831-0f4784f26ba9-kube-api-access-2c5kq\") pod \"opendatahub-operator-controller-manager-66db8bdb84-4lrw9\" (UID: \"3eaa0c78-d939-42c9-b831-0f4784f26ba9\") " pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" Sep 08 15:18:29.260011 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:29.259978 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" Sep 08 15:18:29.540674 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:29.540643 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9"] Sep 08 15:18:29.543706 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:18:29.543677 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eaa0c78_d939_42c9_b831_0f4784f26ba9.slice/crio-e51d59640be79d7fac25686343d8c797fe435c2a033f01c4ef37a23260ff6674 WatchSource:0}: Error finding container e51d59640be79d7fac25686343d8c797fe435c2a033f01c4ef37a23260ff6674: Status 404 returned error can't find the container with id e51d59640be79d7fac25686343d8c797fe435c2a033f01c4ef37a23260ff6674 Sep 08 15:18:30.323982 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:30.323929 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" event={"ID":"3eaa0c78-d939-42c9-b831-0f4784f26ba9","Type":"ContainerStarted","Data":"e51d59640be79d7fac25686343d8c797fe435c2a033f01c4ef37a23260ff6674"} Sep 08 15:18:33.338238 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:33.338201 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" event={"ID":"3eaa0c78-d939-42c9-b831-0f4784f26ba9","Type":"ContainerStarted","Data":"d042a79db781dbfc928f443b1f97a9160f5d0ef3d40ecba7a2e0788b5d3a2eca"} Sep 08 15:18:34.343713 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:34.343683 2572 generic.go:356] "Generic (PLEG): container finished" podID="3eaa0c78-d939-42c9-b831-0f4784f26ba9" containerID="d042a79db781dbfc928f443b1f97a9160f5d0ef3d40ecba7a2e0788b5d3a2eca" exitCode=0 Sep 08 15:18:34.344114 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:34.343769 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" event={"ID":"3eaa0c78-d939-42c9-b831-0f4784f26ba9","Type":"ContainerDied","Data":"d042a79db781dbfc928f443b1f97a9160f5d0ef3d40ecba7a2e0788b5d3a2eca"} Sep 08 15:18:35.348859 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:35.348818 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" event={"ID":"3eaa0c78-d939-42c9-b831-0f4784f26ba9","Type":"ContainerStarted","Data":"021fe693603fde9dbf6b18adfad5b76aad75aaf4c42d1aec561fe9535bd5c622"} Sep 08 15:18:35.349233 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:35.348929 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" Sep 08 15:18:35.391118 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:35.391069 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" podStartSLOduration=3.012928312 podStartE2EDuration="7.391051955s" totalImagesPullingTime="3.331217839s" totalInitContainerRuntime="1.046905804s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:18:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:35.389239204 +0000 UTC m=+577.541417785" watchObservedRunningTime="2026-09-08 15:18:35.391051955 +0000 UTC m=+577.543230591" Sep 08 15:18:46.354627 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:46.354598 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-66db8bdb84-4lrw9" Sep 08 15:18:46.792201 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:46.792173 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v"] Sep 08 15:18:46.800788 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:46.800771 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v" Sep 08 15:18:46.803623 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:46.803597 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 08 15:18:46.803825 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:46.803629 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 08 15:18:46.804100 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:46.803636 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v"] Sep 08 15:18:46.804100 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:46.803669 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 08 15:18:46.804100 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:46.803695 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 08 15:18:46.804100 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:46.803737 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 08 15:18:46.804100 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:46.803751 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mkrrt\"" Sep 08 15:18:46.926788 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:46.926752 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5mrg\" (UniqueName: \"kubernetes.io/projected/09f2130e-c158-4405-a1dc-55dd88b8502c-kube-api-access-q5mrg\") pod \"lws-controller-manager-7f7779588f-v9r6v\" (UID: \"09f2130e-c158-4405-a1dc-55dd88b8502c\") " pod="openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v" Sep 08 15:18:46.926958 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:46.926802 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/09f2130e-c158-4405-a1dc-55dd88b8502c-metrics-cert\") pod \"lws-controller-manager-7f7779588f-v9r6v\" (UID: \"09f2130e-c158-4405-a1dc-55dd88b8502c\") " pod="openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v" Sep 08 15:18:46.926958 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:46.926834 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/09f2130e-c158-4405-a1dc-55dd88b8502c-manager-config\") pod \"lws-controller-manager-7f7779588f-v9r6v\" (UID: \"09f2130e-c158-4405-a1dc-55dd88b8502c\") " pod="openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v" Sep 08 15:18:46.926958 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:46.926850 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09f2130e-c158-4405-a1dc-55dd88b8502c-cert\") pod \"lws-controller-manager-7f7779588f-v9r6v\" (UID: \"09f2130e-c158-4405-a1dc-55dd88b8502c\") " pod="openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v" Sep 08 15:18:47.027456 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:47.027410 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/09f2130e-c158-4405-a1dc-55dd88b8502c-manager-config\") pod \"lws-controller-manager-7f7779588f-v9r6v\" (UID: \"09f2130e-c158-4405-a1dc-55dd88b8502c\") " pod="openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v" Sep 08 15:18:47.027456 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:47.027459 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09f2130e-c158-4405-a1dc-55dd88b8502c-cert\") pod \"lws-controller-manager-7f7779588f-v9r6v\" (UID: \"09f2130e-c158-4405-a1dc-55dd88b8502c\") " pod="openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v" Sep 08 15:18:47.027715 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:47.027510 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5mrg\" (UniqueName: \"kubernetes.io/projected/09f2130e-c158-4405-a1dc-55dd88b8502c-kube-api-access-q5mrg\") pod \"lws-controller-manager-7f7779588f-v9r6v\" (UID: \"09f2130e-c158-4405-a1dc-55dd88b8502c\") " pod="openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v" Sep 08 15:18:47.027715 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:47.027544 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/09f2130e-c158-4405-a1dc-55dd88b8502c-metrics-cert\") pod \"lws-controller-manager-7f7779588f-v9r6v\" (UID: \"09f2130e-c158-4405-a1dc-55dd88b8502c\") " pod="openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v" Sep 08 15:18:47.028095 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:47.028074 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/09f2130e-c158-4405-a1dc-55dd88b8502c-manager-config\") pod \"lws-controller-manager-7f7779588f-v9r6v\" (UID: \"09f2130e-c158-4405-a1dc-55dd88b8502c\") " pod="openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v" Sep 08 15:18:47.030083 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:47.030065 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/09f2130e-c158-4405-a1dc-55dd88b8502c-metrics-cert\") pod \"lws-controller-manager-7f7779588f-v9r6v\" (UID: \"09f2130e-c158-4405-a1dc-55dd88b8502c\") " pod="openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v" Sep 08 15:18:47.030157 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:47.030109 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09f2130e-c158-4405-a1dc-55dd88b8502c-cert\") pod \"lws-controller-manager-7f7779588f-v9r6v\" (UID: \"09f2130e-c158-4405-a1dc-55dd88b8502c\") " pod="openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v" Sep 08 15:18:47.039354 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:47.039327 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5mrg\" (UniqueName: \"kubernetes.io/projected/09f2130e-c158-4405-a1dc-55dd88b8502c-kube-api-access-q5mrg\") pod \"lws-controller-manager-7f7779588f-v9r6v\" (UID: \"09f2130e-c158-4405-a1dc-55dd88b8502c\") " pod="openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v" Sep 08 15:18:47.110428 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:47.110348 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v" Sep 08 15:18:47.380836 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:47.380732 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v"] Sep 08 15:18:47.384988 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:18:47.384964 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f2130e_c158_4405_a1dc_55dd88b8502c.slice/crio-5de5c5f84bd6299d717f5a7bba31e45439f387e788f75284cf63f78d0cfa04d1 WatchSource:0}: Error finding container 5de5c5f84bd6299d717f5a7bba31e45439f387e788f75284cf63f78d0cfa04d1: Status 404 returned error can't find the container with id 5de5c5f84bd6299d717f5a7bba31e45439f387e788f75284cf63f78d0cfa04d1 Sep 08 15:18:47.391009 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:47.390979 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v" event={"ID":"09f2130e-c158-4405-a1dc-55dd88b8502c","Type":"ContainerStarted","Data":"5de5c5f84bd6299d717f5a7bba31e45439f387e788f75284cf63f78d0cfa04d1"} Sep 08 15:18:50.404335 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:50.404302 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v" event={"ID":"09f2130e-c158-4405-a1dc-55dd88b8502c","Type":"ContainerStarted","Data":"dc9b463ecc01b33e0b4e88987bdb67b73e85dd51fb97ed8c56aeff2ecbca9dd7"} Sep 08 15:18:50.404737 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:50.404432 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v" Sep 08 15:18:50.424145 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:50.424096 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v" podStartSLOduration=1.957477226 podStartE2EDuration="4.424082285s" totalImagesPullingTime="2.466605059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:50.42345518 +0000 UTC m=+592.575633760" watchObservedRunningTime="2026-09-08 15:18:50.424082285 +0000 UTC m=+592.576260865" Sep 08 15:18:58.158416 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.158384 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn"] Sep 08 15:18:58.161550 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.161533 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.164480 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.164457 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 08 15:18:58.164480 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.164476 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-sd4n7\"" Sep 08 15:18:58.164668 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.164513 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 08 15:18:58.164668 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.164518 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 15:18:58.185798 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.185770 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn"] Sep 08 15:18:58.207278 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.207257 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqq27\" (UniqueName: \"kubernetes.io/projected/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-kube-api-access-vqq27\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.207385 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.207292 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.207385 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.207325 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.207385 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.207359 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.207385 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.207382 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.207575 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.207430 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.207575 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.207477 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.207575 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.207510 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.207575 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.207530 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.308094 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.308065 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqq27\" (UniqueName: \"kubernetes.io/projected/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-kube-api-access-vqq27\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.308094 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.308098 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.308338 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.308119 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.308338 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.308136 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.308338 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.308160 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.308338 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.308185 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.308338 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.308229 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.308338 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.308257 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.308338 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.308283 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.308708 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.308483 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.308708 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.308568 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.308708 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.308685 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.308812 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.308712 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.310519 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.310481 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.310818 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.310801 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.315041 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.315027 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 15:18:58.319629 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.319611 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.339340 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.339322 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.340689 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.340672 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 08 15:18:58.359345 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.359325 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 08 15:18:58.370597 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.370572 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqq27\" (UniqueName: \"kubernetes.io/projected/e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a-kube-api-access-vqq27\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn\" (UID: \"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.484170 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.484094 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-sd4n7\"" Sep 08 15:18:58.491011 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.490990 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:18:58.779362 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:58.779332 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn"] Sep 08 15:18:58.782367 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:18:58.782324 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63bb5ac_88a9_4477_8e9e_9f0abd3eaf7a.slice/crio-12cac3c879ae2b6dd650f4d7c75b12f7b2151a7999282fcce672b21ebca7f280 WatchSource:0}: Error finding container 12cac3c879ae2b6dd650f4d7c75b12f7b2151a7999282fcce672b21ebca7f280: Status 404 returned error can't find the container with id 12cac3c879ae2b6dd650f4d7c75b12f7b2151a7999282fcce672b21ebca7f280 Sep 08 15:18:59.439416 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:59.439369 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" event={"ID":"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a","Type":"ContainerStarted","Data":"12cac3c879ae2b6dd650f4d7c75b12f7b2151a7999282fcce672b21ebca7f280"} Sep 08 15:18:59.543645 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:59.543612 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-p8s2s"] Sep 08 15:18:59.549136 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:59.549114 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-p8s2s" Sep 08 15:18:59.551685 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:59.551644 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 08 15:18:59.552761 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:59.552576 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-dvdqh\"" Sep 08 15:18:59.558690 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:59.558403 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-p8s2s"] Sep 08 15:18:59.615605 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:59.615552 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfljx\" (UniqueName: \"kubernetes.io/projected/b9b71d4e-2efc-4e58-8863-5cb9c53a071e-kube-api-access-zfljx\") pod \"odh-model-controller-7998c67dc4-p8s2s\" (UID: \"b9b71d4e-2efc-4e58-8863-5cb9c53a071e\") " pod="opendatahub/odh-model-controller-7998c67dc4-p8s2s" Sep 08 15:18:59.615768 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:59.615634 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9b71d4e-2efc-4e58-8863-5cb9c53a071e-cert\") pod \"odh-model-controller-7998c67dc4-p8s2s\" (UID: \"b9b71d4e-2efc-4e58-8863-5cb9c53a071e\") " pod="opendatahub/odh-model-controller-7998c67dc4-p8s2s" Sep 08 15:18:59.716408 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:59.716326 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfljx\" (UniqueName: \"kubernetes.io/projected/b9b71d4e-2efc-4e58-8863-5cb9c53a071e-kube-api-access-zfljx\") pod \"odh-model-controller-7998c67dc4-p8s2s\" (UID: \"b9b71d4e-2efc-4e58-8863-5cb9c53a071e\") " pod="opendatahub/odh-model-controller-7998c67dc4-p8s2s" Sep 08 15:18:59.716408 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:59.716403 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9b71d4e-2efc-4e58-8863-5cb9c53a071e-cert\") pod \"odh-model-controller-7998c67dc4-p8s2s\" (UID: \"b9b71d4e-2efc-4e58-8863-5cb9c53a071e\") " pod="opendatahub/odh-model-controller-7998c67dc4-p8s2s" Sep 08 15:18:59.719335 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:59.719310 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9b71d4e-2efc-4e58-8863-5cb9c53a071e-cert\") pod \"odh-model-controller-7998c67dc4-p8s2s\" (UID: \"b9b71d4e-2efc-4e58-8863-5cb9c53a071e\") " pod="opendatahub/odh-model-controller-7998c67dc4-p8s2s" Sep 08 15:18:59.727486 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:59.727464 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfljx\" (UniqueName: \"kubernetes.io/projected/b9b71d4e-2efc-4e58-8863-5cb9c53a071e-kube-api-access-zfljx\") pod \"odh-model-controller-7998c67dc4-p8s2s\" (UID: \"b9b71d4e-2efc-4e58-8863-5cb9c53a071e\") " pod="opendatahub/odh-model-controller-7998c67dc4-p8s2s" Sep 08 15:18:59.866776 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:18:59.866741 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-p8s2s" Sep 08 15:19:00.160097 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:00.160065 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-thdk7"] Sep 08 15:19:00.163535 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:00.163513 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-thdk7" Sep 08 15:19:00.172572 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:00.172550 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 08 15:19:00.174787 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:00.174663 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-p8s2s"] Sep 08 15:19:00.182580 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:00.182557 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-thdk7"] Sep 08 15:19:00.221778 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:00.221747 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76339afb-f89c-41bd-83b2-5eebc48cf3bb-tls-certs\") pod \"model-serving-api-54d89bd5fc-thdk7\" (UID: \"76339afb-f89c-41bd-83b2-5eebc48cf3bb\") " pod="opendatahub/model-serving-api-54d89bd5fc-thdk7" Sep 08 15:19:00.221924 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:00.221811 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4ns4\" (UniqueName: \"kubernetes.io/projected/76339afb-f89c-41bd-83b2-5eebc48cf3bb-kube-api-access-p4ns4\") pod \"model-serving-api-54d89bd5fc-thdk7\" (UID: \"76339afb-f89c-41bd-83b2-5eebc48cf3bb\") " pod="opendatahub/model-serving-api-54d89bd5fc-thdk7" Sep 08 15:19:00.323051 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:00.323015 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76339afb-f89c-41bd-83b2-5eebc48cf3bb-tls-certs\") pod \"model-serving-api-54d89bd5fc-thdk7\" (UID: \"76339afb-f89c-41bd-83b2-5eebc48cf3bb\") " pod="opendatahub/model-serving-api-54d89bd5fc-thdk7" Sep 08 15:19:00.323231 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:00.323077 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4ns4\" (UniqueName: \"kubernetes.io/projected/76339afb-f89c-41bd-83b2-5eebc48cf3bb-kube-api-access-p4ns4\") pod \"model-serving-api-54d89bd5fc-thdk7\" (UID: \"76339afb-f89c-41bd-83b2-5eebc48cf3bb\") " pod="opendatahub/model-serving-api-54d89bd5fc-thdk7" Sep 08 15:19:00.325943 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:00.325916 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76339afb-f89c-41bd-83b2-5eebc48cf3bb-tls-certs\") pod \"model-serving-api-54d89bd5fc-thdk7\" (UID: \"76339afb-f89c-41bd-83b2-5eebc48cf3bb\") " pod="opendatahub/model-serving-api-54d89bd5fc-thdk7" Sep 08 15:19:00.335390 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:00.335367 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4ns4\" (UniqueName: \"kubernetes.io/projected/76339afb-f89c-41bd-83b2-5eebc48cf3bb-kube-api-access-p4ns4\") pod \"model-serving-api-54d89bd5fc-thdk7\" (UID: \"76339afb-f89c-41bd-83b2-5eebc48cf3bb\") " pod="opendatahub/model-serving-api-54d89bd5fc-thdk7" Sep 08 15:19:00.475218 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:00.475143 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-thdk7" Sep 08 15:19:01.135763 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:19:01.135722 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b71d4e_2efc_4e58_8863_5cb9c53a071e.slice/crio-fa599398f4ee4903ae421f5377d632faa6adaf3d06402247fea361346578c2f8 WatchSource:0}: Error finding container fa599398f4ee4903ae421f5377d632faa6adaf3d06402247fea361346578c2f8: Status 404 returned error can't find the container with id fa599398f4ee4903ae421f5377d632faa6adaf3d06402247fea361346578c2f8 Sep 08 15:19:01.208088 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:01.208052 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:19:01.208178 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:01.208131 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:19:01.208178 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:01.208161 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:19:01.404408 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:01.404337 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-thdk7"] Sep 08 15:19:01.407600 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:19:01.407565 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76339afb_f89c_41bd_83b2_5eebc48cf3bb.slice/crio-21fcfdcebd53d2d252b83f23485a59ce8ad2323760da7f0af1e1b110d0b86214 WatchSource:0}: Error finding container 21fcfdcebd53d2d252b83f23485a59ce8ad2323760da7f0af1e1b110d0b86214: Status 404 returned error can't find the container with id 21fcfdcebd53d2d252b83f23485a59ce8ad2323760da7f0af1e1b110d0b86214 Sep 08 15:19:01.409843 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:01.409822 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7f7779588f-v9r6v" Sep 08 15:19:01.448653 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:01.448617 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-thdk7" event={"ID":"76339afb-f89c-41bd-83b2-5eebc48cf3bb","Type":"ContainerStarted","Data":"21fcfdcebd53d2d252b83f23485a59ce8ad2323760da7f0af1e1b110d0b86214"} Sep 08 15:19:01.450606 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:01.450581 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" event={"ID":"e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a","Type":"ContainerStarted","Data":"27917ea81dbe10aa63a5a18258c9fa65cdc292fae98bb088c02829b8a2131355"} Sep 08 15:19:01.451818 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:01.451795 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-p8s2s" event={"ID":"b9b71d4e-2efc-4e58-8863-5cb9c53a071e","Type":"ContainerStarted","Data":"fa599398f4ee4903ae421f5377d632faa6adaf3d06402247fea361346578c2f8"} Sep 08 15:19:01.491171 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:01.491149 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:19:01.493857 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:01.493817 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" podStartSLOduration=1.070460333 podStartE2EDuration="3.493803154s" totalImagesPullingTime="2.423342821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:19:01.49143135 +0000 UTC m=+603.643609929" watchObservedRunningTime="2026-09-08 15:19:01.493803154 +0000 UTC m=+603.645981734" Sep 08 15:19:02.496810 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:02.496782 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:19:03.461113 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:03.461075 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:19:03.474603 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:03.462198 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn" Sep 08 15:19:04.467112 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:04.467076 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-p8s2s" event={"ID":"b9b71d4e-2efc-4e58-8863-5cb9c53a071e","Type":"ContainerStarted","Data":"be150ab294e9d50fb78a702545d4064beb93d4cf0fbdaa3640917f22f4dfeb2f"} Sep 08 15:19:04.485155 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:04.485091 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-p8s2s" podStartSLOduration=2.708794917 podStartE2EDuration="5.485071116s" totalImagesPullingTime="2.776276199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:18:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:19:04.484630437 +0000 UTC m=+606.636809047" watchObservedRunningTime="2026-09-08 15:19:04.485071116 +0000 UTC m=+606.637249695" Sep 08 15:19:05.471164 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:05.471126 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-p8s2s" Sep 08 15:19:06.475849 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:06.475812 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-thdk7" event={"ID":"76339afb-f89c-41bd-83b2-5eebc48cf3bb","Type":"ContainerStarted","Data":"aae5be739b8153355fb4fd943b4185b2f7f8fdaa3025cd049d6f3ec0e6eb2813"} Sep 08 15:19:06.476220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:06.475893 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-thdk7" Sep 08 15:19:06.537617 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:06.537563 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-thdk7" podStartSLOduration=2.268686883 podStartE2EDuration="6.537543414s" totalImagesPullingTime="4.268856531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:19:06.537169357 +0000 UTC m=+608.689347939" watchObservedRunningTime="2026-09-08 15:19:06.537543414 +0000 UTC m=+608.689721994" Sep 08 15:19:07.364175 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.364141 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t"] Sep 08 15:19:07.367460 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.367441 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.370213 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.370187 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 08 15:19:07.370347 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.370239 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 08 15:19:07.370717 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.370701 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 08 15:19:07.370717 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.370714 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lkgfh\"" Sep 08 15:19:07.410228 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.410197 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t"] Sep 08 15:19:07.484222 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.484181 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/581e98bb-746d-47a3-ab34-19e4d7755fe5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.484717 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.484239 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/581e98bb-746d-47a3-ab34-19e4d7755fe5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.484717 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.484299 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqw79\" (UniqueName: \"kubernetes.io/projected/581e98bb-746d-47a3-ab34-19e4d7755fe5-kube-api-access-zqw79\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.484717 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.484362 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/581e98bb-746d-47a3-ab34-19e4d7755fe5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.484717 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.484435 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/581e98bb-746d-47a3-ab34-19e4d7755fe5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.484717 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.484557 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/581e98bb-746d-47a3-ab34-19e4d7755fe5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.484717 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.484621 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/581e98bb-746d-47a3-ab34-19e4d7755fe5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.585483 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.585445 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/581e98bb-746d-47a3-ab34-19e4d7755fe5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.585696 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.585529 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/581e98bb-746d-47a3-ab34-19e4d7755fe5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.585696 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.585555 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/581e98bb-746d-47a3-ab34-19e4d7755fe5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.585696 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.585601 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/581e98bb-746d-47a3-ab34-19e4d7755fe5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.585995 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.585958 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/581e98bb-746d-47a3-ab34-19e4d7755fe5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.586121 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.586021 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/581e98bb-746d-47a3-ab34-19e4d7755fe5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.586121 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.586057 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqw79\" (UniqueName: \"kubernetes.io/projected/581e98bb-746d-47a3-ab34-19e4d7755fe5-kube-api-access-zqw79\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.587206 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.587154 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/581e98bb-746d-47a3-ab34-19e4d7755fe5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.588279 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.588257 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/581e98bb-746d-47a3-ab34-19e4d7755fe5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.588659 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.588639 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/581e98bb-746d-47a3-ab34-19e4d7755fe5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.588750 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.588729 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/581e98bb-746d-47a3-ab34-19e4d7755fe5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.588967 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.588950 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/581e98bb-746d-47a3-ab34-19e4d7755fe5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.594530 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.594477 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/581e98bb-746d-47a3-ab34-19e4d7755fe5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.595200 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.595176 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqw79\" (UniqueName: \"kubernetes.io/projected/581e98bb-746d-47a3-ab34-19e4d7755fe5-kube-api-access-zqw79\") pod \"istiod-openshift-gateway-5677ccb49b-9mh8t\" (UID: \"581e98bb-746d-47a3-ab34-19e4d7755fe5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:07.676895 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:07.676792 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:08.016760 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:08.016733 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t"] Sep 08 15:19:08.017603 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:19:08.017580 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581e98bb_746d_47a3_ab34_19e4d7755fe5.slice/crio-e5e433af0fbf9715c569730a12e137bdfa55aa223cad6e717c5da0910ba9c064 WatchSource:0}: Error finding container e5e433af0fbf9715c569730a12e137bdfa55aa223cad6e717c5da0910ba9c064: Status 404 returned error can't find the container with id e5e433af0fbf9715c569730a12e137bdfa55aa223cad6e717c5da0910ba9c064 Sep 08 15:19:08.490108 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:08.490013 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" event={"ID":"581e98bb-746d-47a3-ab34-19e4d7755fe5","Type":"ContainerStarted","Data":"e5e433af0fbf9715c569730a12e137bdfa55aa223cad6e717c5da0910ba9c064"} Sep 08 15:19:11.756545 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:11.756454 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:19:11.756805 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:11.756556 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:19:12.508982 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:12.508936 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" event={"ID":"581e98bb-746d-47a3-ab34-19e4d7755fe5","Type":"ContainerStarted","Data":"0e209303d56375652cd0586609c4aa1b7842f402897923b50c7baa81e4bcf3ea"} Sep 08 15:19:12.509161 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:12.509098 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:12.510952 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:12.510921 2572 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-9mh8t container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 15:19:12.511079 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:12.510974 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" podUID="581e98bb-746d-47a3-ab34-19e4d7755fe5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 15:19:12.581956 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:12.581893 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" podStartSLOduration=1.8455019240000001 podStartE2EDuration="5.581874667s" totalImagesPullingTime="3.736372743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:19:12.580138519 +0000 UTC m=+614.732317098" watchObservedRunningTime="2026-09-08 15:19:12.581874667 +0000 UTC m=+614.734053246" Sep 08 15:19:13.513632 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:13.513597 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9mh8t" Sep 08 15:19:16.478229 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:16.478196 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-p8s2s" Sep 08 15:19:17.489603 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:19:17.489567 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-thdk7" Sep 08 15:23:14.975108 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:14.975068 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg"] Sep 08 15:23:14.978368 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:14.978348 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:14.986610 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:14.986589 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-p82cs\"" Sep 08 15:23:14.986745 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:14.986608 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 08 15:23:14.986745 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:14.986609 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 08 15:23:14.986745 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:14.986609 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 08 15:23:14.998022 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:14.997998 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg"] Sep 08 15:23:15.041453 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.041423 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f32b63c-aeb3-4ec5-9801-c95895130a34-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg\" (UID: \"3f32b63c-aeb3-4ec5-9801-c95895130a34\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:15.041610 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.041479 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f32b63c-aeb3-4ec5-9801-c95895130a34-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg\" (UID: \"3f32b63c-aeb3-4ec5-9801-c95895130a34\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:15.041610 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.041571 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f32b63c-aeb3-4ec5-9801-c95895130a34-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg\" (UID: \"3f32b63c-aeb3-4ec5-9801-c95895130a34\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:15.041701 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.041618 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f32b63c-aeb3-4ec5-9801-c95895130a34-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg\" (UID: \"3f32b63c-aeb3-4ec5-9801-c95895130a34\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:15.041701 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.041636 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gtg2\" (UniqueName: \"kubernetes.io/projected/3f32b63c-aeb3-4ec5-9801-c95895130a34-kube-api-access-7gtg2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg\" (UID: \"3f32b63c-aeb3-4ec5-9801-c95895130a34\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:15.041701 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.041655 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f32b63c-aeb3-4ec5-9801-c95895130a34-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg\" (UID: \"3f32b63c-aeb3-4ec5-9801-c95895130a34\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:15.142451 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.142421 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f32b63c-aeb3-4ec5-9801-c95895130a34-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg\" (UID: \"3f32b63c-aeb3-4ec5-9801-c95895130a34\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:15.142601 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.142479 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f32b63c-aeb3-4ec5-9801-c95895130a34-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg\" (UID: \"3f32b63c-aeb3-4ec5-9801-c95895130a34\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:15.142601 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.142562 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f32b63c-aeb3-4ec5-9801-c95895130a34-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg\" (UID: \"3f32b63c-aeb3-4ec5-9801-c95895130a34\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:15.142601 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.142597 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f32b63c-aeb3-4ec5-9801-c95895130a34-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg\" (UID: \"3f32b63c-aeb3-4ec5-9801-c95895130a34\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:15.142765 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.142622 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gtg2\" (UniqueName: \"kubernetes.io/projected/3f32b63c-aeb3-4ec5-9801-c95895130a34-kube-api-access-7gtg2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg\" (UID: \"3f32b63c-aeb3-4ec5-9801-c95895130a34\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:15.142765 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.142654 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f32b63c-aeb3-4ec5-9801-c95895130a34-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg\" (UID: \"3f32b63c-aeb3-4ec5-9801-c95895130a34\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:15.142860 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.142834 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f32b63c-aeb3-4ec5-9801-c95895130a34-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg\" (UID: \"3f32b63c-aeb3-4ec5-9801-c95895130a34\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:15.142979 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.142960 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f32b63c-aeb3-4ec5-9801-c95895130a34-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg\" (UID: \"3f32b63c-aeb3-4ec5-9801-c95895130a34\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:15.143067 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.143045 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f32b63c-aeb3-4ec5-9801-c95895130a34-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg\" (UID: \"3f32b63c-aeb3-4ec5-9801-c95895130a34\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:15.144944 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.144921 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f32b63c-aeb3-4ec5-9801-c95895130a34-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg\" (UID: \"3f32b63c-aeb3-4ec5-9801-c95895130a34\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:15.145128 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.145112 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f32b63c-aeb3-4ec5-9801-c95895130a34-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg\" (UID: \"3f32b63c-aeb3-4ec5-9801-c95895130a34\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:15.156695 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.154094 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gtg2\" (UniqueName: \"kubernetes.io/projected/3f32b63c-aeb3-4ec5-9801-c95895130a34-kube-api-access-7gtg2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg\" (UID: \"3f32b63c-aeb3-4ec5-9801-c95895130a34\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:15.289307 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.289278 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:15.566474 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.566453 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg"] Sep 08 15:23:15.568581 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:23:15.568550 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f32b63c_aeb3_4ec5_9801_c95895130a34.slice/crio-8b59e3376523d0880c5f624ab9299c659a1c540dd07faf571181038c3db77b88 WatchSource:0}: Error finding container 8b59e3376523d0880c5f624ab9299c659a1c540dd07faf571181038c3db77b88: Status 404 returned error can't find the container with id 8b59e3376523d0880c5f624ab9299c659a1c540dd07faf571181038c3db77b88 Sep 08 15:23:15.569918 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:15.569898 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 15:23:16.435103 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:16.435063 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" event={"ID":"3f32b63c-aeb3-4ec5-9801-c95895130a34","Type":"ContainerStarted","Data":"8b59e3376523d0880c5f624ab9299c659a1c540dd07faf571181038c3db77b88"} Sep 08 15:23:17.441774 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:17.441725 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" event={"ID":"3f32b63c-aeb3-4ec5-9801-c95895130a34","Type":"ContainerStarted","Data":"3110156793c0f2c358e0c085ff6d216b8e4ced4813f62cafc017af7e2376dd3d"} Sep 08 15:23:17.470398 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:17.470351 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" podStartSLOduration=2.456393615 podStartE2EDuration="3.470336185s" totalImagesPullingTime="1.01394257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:23:17.468818001 +0000 UTC m=+859.620996580" watchObservedRunningTime="2026-09-08 15:23:17.470336185 +0000 UTC m=+859.622514764" Sep 08 15:23:25.290002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:25.289963 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:25.290479 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:25.290023 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:25.302727 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:25.302700 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:25.486236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:25.486213 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg" Sep 08 15:23:45.947299 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:45.947258 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz"] Sep 08 15:23:45.950692 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:45.950671 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:45.952972 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:45.952937 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 08 15:23:45.961842 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:45.961804 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz"] Sep 08 15:23:46.038201 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.038172 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx495\" (UniqueName: \"kubernetes.io/projected/52110219-9ed0-45a1-ab44-521f68d30b3f-kube-api-access-dx495\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xrnlz\" (UID: \"52110219-9ed0-45a1-ab44-521f68d30b3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:46.038301 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.038208 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52110219-9ed0-45a1-ab44-521f68d30b3f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xrnlz\" (UID: \"52110219-9ed0-45a1-ab44-521f68d30b3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:46.038301 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.038227 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52110219-9ed0-45a1-ab44-521f68d30b3f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xrnlz\" (UID: \"52110219-9ed0-45a1-ab44-521f68d30b3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:46.038301 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.038243 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52110219-9ed0-45a1-ab44-521f68d30b3f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xrnlz\" (UID: \"52110219-9ed0-45a1-ab44-521f68d30b3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:46.038425 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.038312 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52110219-9ed0-45a1-ab44-521f68d30b3f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xrnlz\" (UID: \"52110219-9ed0-45a1-ab44-521f68d30b3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:46.038425 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.038348 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52110219-9ed0-45a1-ab44-521f68d30b3f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xrnlz\" (UID: \"52110219-9ed0-45a1-ab44-521f68d30b3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:46.138785 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.138755 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52110219-9ed0-45a1-ab44-521f68d30b3f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xrnlz\" (UID: \"52110219-9ed0-45a1-ab44-521f68d30b3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:46.138905 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.138807 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx495\" (UniqueName: \"kubernetes.io/projected/52110219-9ed0-45a1-ab44-521f68d30b3f-kube-api-access-dx495\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xrnlz\" (UID: \"52110219-9ed0-45a1-ab44-521f68d30b3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:46.138905 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.138828 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52110219-9ed0-45a1-ab44-521f68d30b3f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xrnlz\" (UID: \"52110219-9ed0-45a1-ab44-521f68d30b3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:46.139016 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.138961 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52110219-9ed0-45a1-ab44-521f68d30b3f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xrnlz\" (UID: \"52110219-9ed0-45a1-ab44-521f68d30b3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:46.139016 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.139007 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52110219-9ed0-45a1-ab44-521f68d30b3f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xrnlz\" (UID: \"52110219-9ed0-45a1-ab44-521f68d30b3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:46.139114 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.139037 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52110219-9ed0-45a1-ab44-521f68d30b3f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xrnlz\" (UID: \"52110219-9ed0-45a1-ab44-521f68d30b3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:46.139176 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.139157 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52110219-9ed0-45a1-ab44-521f68d30b3f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xrnlz\" (UID: \"52110219-9ed0-45a1-ab44-521f68d30b3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:46.139287 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.139270 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52110219-9ed0-45a1-ab44-521f68d30b3f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xrnlz\" (UID: \"52110219-9ed0-45a1-ab44-521f68d30b3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:46.139433 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.139412 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52110219-9ed0-45a1-ab44-521f68d30b3f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xrnlz\" (UID: \"52110219-9ed0-45a1-ab44-521f68d30b3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:46.141246 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.141216 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52110219-9ed0-45a1-ab44-521f68d30b3f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xrnlz\" (UID: \"52110219-9ed0-45a1-ab44-521f68d30b3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:46.141343 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.141330 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52110219-9ed0-45a1-ab44-521f68d30b3f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xrnlz\" (UID: \"52110219-9ed0-45a1-ab44-521f68d30b3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:46.146687 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.146668 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx495\" (UniqueName: \"kubernetes.io/projected/52110219-9ed0-45a1-ab44-521f68d30b3f-kube-api-access-dx495\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xrnlz\" (UID: \"52110219-9ed0-45a1-ab44-521f68d30b3f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:46.277860 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.277837 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:46.550670 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.550602 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz"] Sep 08 15:23:46.553713 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:23:46.553689 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52110219_9ed0_45a1_ab44_521f68d30b3f.slice/crio-ce983aee501d9b1a960590c6522b62ad644e70dc823f4554daa0807c781f83f9 WatchSource:0}: Error finding container ce983aee501d9b1a960590c6522b62ad644e70dc823f4554daa0807c781f83f9: Status 404 returned error can't find the container with id ce983aee501d9b1a960590c6522b62ad644e70dc823f4554daa0807c781f83f9 Sep 08 15:23:46.564127 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:46.564101 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" event={"ID":"52110219-9ed0-45a1-ab44-521f68d30b3f","Type":"ContainerStarted","Data":"ce983aee501d9b1a960590c6522b62ad644e70dc823f4554daa0807c781f83f9"} Sep 08 15:23:47.570112 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:47.570076 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" event={"ID":"52110219-9ed0-45a1-ab44-521f68d30b3f","Type":"ContainerStarted","Data":"1fcf444671cf7053da69ba95a1fa6a9bb6b8710b5fb51c4c684842a277be32e9"} Sep 08 15:23:47.589931 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:47.589732 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" podStartSLOduration=2.388995586 podStartE2EDuration="2.589714079s" totalImagesPullingTime="200.718493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:23:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:23:47.589201896 +0000 UTC m=+889.741380475" watchObservedRunningTime="2026-09-08 15:23:47.589714079 +0000 UTC m=+889.741892660" Sep 08 15:23:56.278584 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:56.278472 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:56.279021 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:56.278596 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:56.291755 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:56.291726 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:23:56.620690 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:23:56.620613 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xrnlz" Sep 08 15:24:02.149319 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.149278 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz"] Sep 08 15:24:02.152716 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.152691 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.155226 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.155198 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 08 15:24:02.167641 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.167612 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz"] Sep 08 15:24:02.247313 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.247284 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e02c78ba-cc89-455a-a661-997905e79abe-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz\" (UID: \"e02c78ba-cc89-455a-a661-997905e79abe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.247411 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.247322 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e02c78ba-cc89-455a-a661-997905e79abe-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz\" (UID: \"e02c78ba-cc89-455a-a661-997905e79abe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.247411 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.247350 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6849\" (UniqueName: \"kubernetes.io/projected/e02c78ba-cc89-455a-a661-997905e79abe-kube-api-access-r6849\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz\" (UID: \"e02c78ba-cc89-455a-a661-997905e79abe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.247527 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.247412 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e02c78ba-cc89-455a-a661-997905e79abe-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz\" (UID: \"e02c78ba-cc89-455a-a661-997905e79abe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.247527 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.247458 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e02c78ba-cc89-455a-a661-997905e79abe-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz\" (UID: \"e02c78ba-cc89-455a-a661-997905e79abe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.247631 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.247544 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e02c78ba-cc89-455a-a661-997905e79abe-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz\" (UID: \"e02c78ba-cc89-455a-a661-997905e79abe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.348646 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.348575 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e02c78ba-cc89-455a-a661-997905e79abe-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz\" (UID: \"e02c78ba-cc89-455a-a661-997905e79abe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.348759 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.348670 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e02c78ba-cc89-455a-a661-997905e79abe-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz\" (UID: \"e02c78ba-cc89-455a-a661-997905e79abe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.348759 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.348700 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6849\" (UniqueName: \"kubernetes.io/projected/e02c78ba-cc89-455a-a661-997905e79abe-kube-api-access-r6849\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz\" (UID: \"e02c78ba-cc89-455a-a661-997905e79abe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.348759 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.348740 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e02c78ba-cc89-455a-a661-997905e79abe-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz\" (UID: \"e02c78ba-cc89-455a-a661-997905e79abe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.348874 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.348780 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e02c78ba-cc89-455a-a661-997905e79abe-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz\" (UID: \"e02c78ba-cc89-455a-a661-997905e79abe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.348874 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.348814 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e02c78ba-cc89-455a-a661-997905e79abe-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz\" (UID: \"e02c78ba-cc89-455a-a661-997905e79abe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.349082 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.349056 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e02c78ba-cc89-455a-a661-997905e79abe-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz\" (UID: \"e02c78ba-cc89-455a-a661-997905e79abe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.349147 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.349080 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e02c78ba-cc89-455a-a661-997905e79abe-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz\" (UID: \"e02c78ba-cc89-455a-a661-997905e79abe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.349258 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.349235 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e02c78ba-cc89-455a-a661-997905e79abe-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz\" (UID: \"e02c78ba-cc89-455a-a661-997905e79abe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.351308 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.351280 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e02c78ba-cc89-455a-a661-997905e79abe-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz\" (UID: \"e02c78ba-cc89-455a-a661-997905e79abe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.351455 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.351434 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e02c78ba-cc89-455a-a661-997905e79abe-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz\" (UID: \"e02c78ba-cc89-455a-a661-997905e79abe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.359864 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.359840 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6849\" (UniqueName: \"kubernetes.io/projected/e02c78ba-cc89-455a-a661-997905e79abe-kube-api-access-r6849\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz\" (UID: \"e02c78ba-cc89-455a-a661-997905e79abe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.465641 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.465585 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:02.740249 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:02.740218 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz"] Sep 08 15:24:02.741906 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:24:02.741883 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02c78ba_cc89_455a_a661_997905e79abe.slice/crio-603320f45948fa11dc1a573ed5c785971ebfe00fc57fae39ff3ee2da644bfc27 WatchSource:0}: Error finding container 603320f45948fa11dc1a573ed5c785971ebfe00fc57fae39ff3ee2da644bfc27: Status 404 returned error can't find the container with id 603320f45948fa11dc1a573ed5c785971ebfe00fc57fae39ff3ee2da644bfc27 Sep 08 15:24:03.644582 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:03.644539 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" event={"ID":"e02c78ba-cc89-455a-a661-997905e79abe","Type":"ContainerStarted","Data":"9fe5028e17fb0ce18786e020948cf2c48c41408ad2b8a1905e1f8402b16871cb"} Sep 08 15:24:03.646823 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:03.644620 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" event={"ID":"e02c78ba-cc89-455a-a661-997905e79abe","Type":"ContainerStarted","Data":"603320f45948fa11dc1a573ed5c785971ebfe00fc57fae39ff3ee2da644bfc27"} Sep 08 15:24:03.666144 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:03.666095 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" podStartSLOduration=1.4415440560000001 podStartE2EDuration="1.666077812s" totalImagesPullingTime="224.533756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:24:03.665649432 +0000 UTC m=+905.817828006" watchObservedRunningTime="2026-09-08 15:24:03.666077812 +0000 UTC m=+905.818256451" Sep 08 15:24:12.466586 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:12.466544 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:12.467110 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:12.466623 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:12.479335 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:12.479307 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:24:12.693581 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:24:12.693546 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz" Sep 08 15:25:55.798335 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.798259 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j"] Sep 08 15:25:55.801258 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.801243 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.803778 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.803759 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d8e721-openshift-default-dockercfg-5hqsq\"" Sep 08 15:25:55.819904 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.819883 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j"] Sep 08 15:25:55.877301 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.877266 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-workload-socket\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.877470 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.877321 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-envoy\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.877470 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.877366 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpppw\" (UniqueName: \"kubernetes.io/projected/6da6535b-af05-4dbb-87f5-06eb503b80f5-kube-api-access-gpppw\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.877470 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.877405 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-podinfo\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.877736 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.877565 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-credential-socket\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.877736 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.877621 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-data\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.877736 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.877666 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-token\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.877736 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.877692 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da6535b-af05-4dbb-87f5-06eb503b80f5-istiod-ca-cert\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.877736 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.877716 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-workload-certs\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.979183 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.979147 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-workload-socket\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.979183 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.979186 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-envoy\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.979371 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.979216 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpppw\" (UniqueName: \"kubernetes.io/projected/6da6535b-af05-4dbb-87f5-06eb503b80f5-kube-api-access-gpppw\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.979371 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.979241 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-podinfo\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.979371 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.979279 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-credential-socket\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.979371 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.979296 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-data\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.979371 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.979317 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-token\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.979371 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.979341 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da6535b-af05-4dbb-87f5-06eb503b80f5-istiod-ca-cert\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.979371 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.979363 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-workload-certs\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.979797 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.979764 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-workload-certs\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.979917 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.979807 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-data\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.979917 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.979854 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-credential-socket\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.980023 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.980002 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-workload-socket\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.980184 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.980167 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da6535b-af05-4dbb-87f5-06eb503b80f5-istiod-ca-cert\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.981743 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.981716 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-envoy\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.981894 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.981878 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-podinfo\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.986821 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.986800 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-token\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:55.987034 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:55.987013 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpppw\" (UniqueName: \"kubernetes.io/projected/6da6535b-af05-4dbb-87f5-06eb503b80f5-kube-api-access-gpppw\") pod \"e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:56.110685 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:56.110615 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:56.391742 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:56.391657 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j"] Sep 08 15:25:56.394946 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:25:56.394919 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da6535b_af05_4dbb_87f5_06eb503b80f5.slice/crio-975dd99b268125a72e28c13cf5b07acb022fbd9702ee05358901dbfd0c218741 WatchSource:0}: Error finding container 975dd99b268125a72e28c13cf5b07acb022fbd9702ee05358901dbfd0c218741: Status 404 returned error can't find the container with id 975dd99b268125a72e28c13cf5b07acb022fbd9702ee05358901dbfd0c218741 Sep 08 15:25:56.397043 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:56.397011 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:25:56.397125 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:56.397077 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:25:56.397125 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:56.397109 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:25:57.138008 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:57.137978 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" event={"ID":"6da6535b-af05-4dbb-87f5-06eb503b80f5","Type":"ContainerStarted","Data":"f2f5101343c046318b17838001e5ca052e061b3943a01787bb136cd2c1012976"} Sep 08 15:25:57.138008 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:57.138023 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" event={"ID":"6da6535b-af05-4dbb-87f5-06eb503b80f5","Type":"ContainerStarted","Data":"975dd99b268125a72e28c13cf5b07acb022fbd9702ee05358901dbfd0c218741"} Sep 08 15:25:57.173217 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:57.173153 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" podStartSLOduration=2.173137755 podStartE2EDuration="2.173137755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:25:57.17082675 +0000 UTC m=+1019.323005329" watchObservedRunningTime="2026-09-08 15:25:57.173137755 +0000 UTC m=+1019.325316335" Sep 08 15:25:58.111418 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:58.111389 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:58.116120 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:58.116097 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:58.142165 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:58.142138 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:25:58.143032 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:25:58.143010 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:26:01.061623 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:01.061593 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj"] Sep 08 15:26:01.064675 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:01.064658 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj" Sep 08 15:26:01.066832 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:01.066803 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d8e721-dockercfg-kxrzd\"" Sep 08 15:26:01.066963 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:01.066835 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d8e721\"" Sep 08 15:26:01.073136 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:01.073113 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj"] Sep 08 15:26:01.125025 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:01.124996 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v9kj\" (UniqueName: \"kubernetes.io/projected/54c5753c-7a27-44ab-8e01-41273618e0f1-kube-api-access-9v9kj\") pod \"payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj\" (UID: \"54c5753c-7a27-44ab-8e01-41273618e0f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj" Sep 08 15:26:01.125147 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:01.125056 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c5753c-7a27-44ab-8e01-41273618e0f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj\" (UID: \"54c5753c-7a27-44ab-8e01-41273618e0f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj" Sep 08 15:26:01.226085 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:01.226057 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v9kj\" (UniqueName: \"kubernetes.io/projected/54c5753c-7a27-44ab-8e01-41273618e0f1-kube-api-access-9v9kj\") pod \"payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj\" (UID: \"54c5753c-7a27-44ab-8e01-41273618e0f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj" Sep 08 15:26:01.226261 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:01.226113 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c5753c-7a27-44ab-8e01-41273618e0f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj\" (UID: \"54c5753c-7a27-44ab-8e01-41273618e0f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj" Sep 08 15:26:01.226796 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:01.226775 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c5753c-7a27-44ab-8e01-41273618e0f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj\" (UID: \"54c5753c-7a27-44ab-8e01-41273618e0f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj" Sep 08 15:26:01.233885 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:01.233853 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v9kj\" (UniqueName: \"kubernetes.io/projected/54c5753c-7a27-44ab-8e01-41273618e0f1-kube-api-access-9v9kj\") pod \"payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj\" (UID: \"54c5753c-7a27-44ab-8e01-41273618e0f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj" Sep 08 15:26:01.374381 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:01.374297 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj" Sep 08 15:26:01.650359 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:01.650292 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj"] Sep 08 15:26:01.654134 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:26:01.654093 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c5753c_7a27_44ab_8e01_41273618e0f1.slice/crio-bbe4668fdc6bad38ef60b73aadff4ca834514a87bec65b1c247b177a0f65292b WatchSource:0}: Error finding container bbe4668fdc6bad38ef60b73aadff4ca834514a87bec65b1c247b177a0f65292b: Status 404 returned error can't find the container with id bbe4668fdc6bad38ef60b73aadff4ca834514a87bec65b1c247b177a0f65292b Sep 08 15:26:02.163669 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:02.163634 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj" event={"ID":"54c5753c-7a27-44ab-8e01-41273618e0f1","Type":"ContainerStarted","Data":"bbe4668fdc6bad38ef60b73aadff4ca834514a87bec65b1c247b177a0f65292b"} Sep 08 15:26:04.175652 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:04.175610 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj" event={"ID":"54c5753c-7a27-44ab-8e01-41273618e0f1","Type":"ContainerStarted","Data":"da11e9ad1ed2901438897aba923a585d1dcdc6041f92f363b3a5489b74c73759"} Sep 08 15:26:04.176040 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:04.175744 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj" Sep 08 15:26:04.201084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:04.201043 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj" podStartSLOduration=1.14051445 podStartE2EDuration="3.201028929s" totalImagesPullingTime="2.060514479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:26:04.199159487 +0000 UTC m=+1026.351338079" watchObservedRunningTime="2026-09-08 15:26:04.201028929 +0000 UTC m=+1026.353207510" Sep 08 15:26:08.156307 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.156271 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l"] Sep 08 15:26:08.163831 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.163080 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" Sep 08 15:26:08.165648 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.165628 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-062f32f0-serving-cert\"" Sep 08 15:26:08.165836 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.165815 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 08 15:26:08.166552 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.166364 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 08 15:26:08.166552 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.166375 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hrbm8\"" Sep 08 15:26:08.167807 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.167664 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 15:26:08.168640 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.168608 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l"] Sep 08 15:26:08.294025 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.293990 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-maas-api-tls\") pod \"maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l\" (UID: \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" Sep 08 15:26:08.294025 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.294025 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzzck\" (UniqueName: \"kubernetes.io/projected/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-kube-api-access-gzzck\") pod \"maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l\" (UID: \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" Sep 08 15:26:08.294241 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.294049 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-metrics-certs\") pod \"maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l\" (UID: \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" Sep 08 15:26:08.294241 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.294126 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-openshift-service-ca\") pod \"maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l\" (UID: \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" Sep 08 15:26:08.394804 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.394772 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-maas-api-tls\") pod \"maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l\" (UID: \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" Sep 08 15:26:08.394804 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.394805 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzzck\" (UniqueName: \"kubernetes.io/projected/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-kube-api-access-gzzck\") pod \"maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l\" (UID: \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" Sep 08 15:26:08.394990 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.394828 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-metrics-certs\") pod \"maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l\" (UID: \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" Sep 08 15:26:08.394990 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.394853 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-openshift-service-ca\") pod \"maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l\" (UID: \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" Sep 08 15:26:08.395552 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.395423 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-openshift-service-ca\") pod \"maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l\" (UID: \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" Sep 08 15:26:08.397361 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.397337 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-metrics-certs\") pod \"maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l\" (UID: \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" Sep 08 15:26:08.397466 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.397382 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-maas-api-tls\") pod \"maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l\" (UID: \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" Sep 08 15:26:08.402628 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.402605 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzzck\" (UniqueName: \"kubernetes.io/projected/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-kube-api-access-gzzck\") pod \"maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l\" (UID: \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" Sep 08 15:26:08.475371 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.475283 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" Sep 08 15:26:08.786378 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:08.786349 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l"] Sep 08 15:26:08.788820 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:26:08.788784 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c15a87c_aa8f_4d3f_bf71_bac2de5c3471.slice/crio-abd2117da3d0ae45e0563eef44594b12e973bd03da2ae7b3be8806ed97365d10 WatchSource:0}: Error finding container abd2117da3d0ae45e0563eef44594b12e973bd03da2ae7b3be8806ed97365d10: Status 404 returned error can't find the container with id abd2117da3d0ae45e0563eef44594b12e973bd03da2ae7b3be8806ed97365d10 Sep 08 15:26:09.203647 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:09.203568 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" event={"ID":"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471","Type":"ContainerStarted","Data":"abd2117da3d0ae45e0563eef44594b12e973bd03da2ae7b3be8806ed97365d10"} Sep 08 15:26:10.575605 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:10.575567 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6"] Sep 08 15:26:10.579220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:10.579196 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6" Sep 08 15:26:10.581264 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:10.581240 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-3bc9dc\"" Sep 08 15:26:10.581672 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:10.581654 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-3bc9dc-dockercfg-hnbnz\"" Sep 08 15:26:10.591332 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:10.591310 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6"] Sep 08 15:26:10.715787 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:10.715681 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmckh\" (UniqueName: \"kubernetes.io/projected/6e7e3a85-e48c-438d-9f30-59412276e100-kube-api-access-kmckh\") pod \"payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6\" (UID: \"6e7e3a85-e48c-438d-9f30-59412276e100\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6" Sep 08 15:26:10.715787 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:10.715755 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7e3a85-e48c-438d-9f30-59412276e100-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6\" (UID: \"6e7e3a85-e48c-438d-9f30-59412276e100\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6" Sep 08 15:26:10.816856 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:10.816817 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmckh\" (UniqueName: \"kubernetes.io/projected/6e7e3a85-e48c-438d-9f30-59412276e100-kube-api-access-kmckh\") pod \"payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6\" (UID: \"6e7e3a85-e48c-438d-9f30-59412276e100\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6" Sep 08 15:26:10.817030 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:10.816887 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7e3a85-e48c-438d-9f30-59412276e100-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6\" (UID: \"6e7e3a85-e48c-438d-9f30-59412276e100\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6" Sep 08 15:26:10.817792 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:10.817767 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7e3a85-e48c-438d-9f30-59412276e100-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6\" (UID: \"6e7e3a85-e48c-438d-9f30-59412276e100\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6" Sep 08 15:26:10.825454 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:10.825267 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmckh\" (UniqueName: \"kubernetes.io/projected/6e7e3a85-e48c-438d-9f30-59412276e100-kube-api-access-kmckh\") pod \"payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6\" (UID: \"6e7e3a85-e48c-438d-9f30-59412276e100\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6" Sep 08 15:26:10.892250 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:10.892155 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6" Sep 08 15:26:11.186672 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:11.186585 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6"] Sep 08 15:26:11.190280 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:26:11.190248 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7e3a85_e48c_438d_9f30_59412276e100.slice/crio-2535791ce9fe0f249eede9a0c58b7549eb3314c16eba32ea2945748b6d31866c WatchSource:0}: Error finding container 2535791ce9fe0f249eede9a0c58b7549eb3314c16eba32ea2945748b6d31866c: Status 404 returned error can't find the container with id 2535791ce9fe0f249eede9a0c58b7549eb3314c16eba32ea2945748b6d31866c Sep 08 15:26:11.215234 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:11.215182 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6" event={"ID":"6e7e3a85-e48c-438d-9f30-59412276e100","Type":"ContainerStarted","Data":"2535791ce9fe0f249eede9a0c58b7549eb3314c16eba32ea2945748b6d31866c"} Sep 08 15:26:12.224436 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:12.224392 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6" event={"ID":"6e7e3a85-e48c-438d-9f30-59412276e100","Type":"ContainerStarted","Data":"b2d744ec534f8c1acbe7610a2ab1394f371eb3a3c6d6fbfa4a2c5ee0768bfa0e"} Sep 08 15:26:12.224914 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:12.224544 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6" Sep 08 15:26:12.244940 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:12.244870 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6" podStartSLOduration=2.244849308 podStartE2EDuration="2.244849308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:26:12.243627994 +0000 UTC m=+1034.395806578" watchObservedRunningTime="2026-09-08 15:26:12.244849308 +0000 UTC m=+1034.397027893" Sep 08 15:26:13.230162 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:13.230122 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" event={"ID":"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471","Type":"ContainerStarted","Data":"eb6b3107eedc102e896d294d6e9702d9a12aee807db5113013b38f7a273ef1fe"} Sep 08 15:26:13.247880 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:13.247838 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" podStartSLOduration=1.461406229 podStartE2EDuration="5.247821994s" totalImagesPullingTime="3.786415765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:26:13.246688555 +0000 UTC m=+1035.398867139" watchObservedRunningTime="2026-09-08 15:26:13.247821994 +0000 UTC m=+1035.400000615" Sep 08 15:26:13.981962 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:13.981158 2572 patch_prober.go:28] interesting pod/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": read tcp 10.132.0.2:33760->10.132.0.23:15021: read: connection reset by peer" start-of-body= Sep 08 15:26:13.981962 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:13.981214 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" podUID="6da6535b-af05-4dbb-87f5-06eb503b80f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": read tcp 10.132.0.2:33760->10.132.0.23:15021: read: connection reset by peer" Sep 08 15:26:14.236839 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:14.236743 2572 generic.go:356] "Generic (PLEG): container finished" podID="6da6535b-af05-4dbb-87f5-06eb503b80f5" containerID="f2f5101343c046318b17838001e5ca052e061b3943a01787bb136cd2c1012976" exitCode=0 Sep 08 15:26:14.236839 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:14.236816 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" event={"ID":"6da6535b-af05-4dbb-87f5-06eb503b80f5","Type":"ContainerDied","Data":"f2f5101343c046318b17838001e5ca052e061b3943a01787bb136cd2c1012976"} Sep 08 15:26:14.237288 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:14.237063 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" Sep 08 15:26:14.237406 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:14.237390 2572 scope.go:117] "RemoveContainer" containerID="f2f5101343c046318b17838001e5ca052e061b3943a01787bb136cd2c1012976" Sep 08 15:26:14.238410 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:14.238377 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:26:14.238559 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:14.238459 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:26:14.238559 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:14.238539 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:26:15.183165 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:15.183136 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj" Sep 08 15:26:15.251516 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:15.249949 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" event={"ID":"6da6535b-af05-4dbb-87f5-06eb503b80f5","Type":"ContainerStarted","Data":"7c3529981c155af9cab9642478ba4b01b5017d15288b92c561f237b416915846"} Sep 08 15:26:16.111178 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:16.111140 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:26:16.111398 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:16.111351 2572 patch_prober.go:28] interesting pod/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" start-of-body= Sep 08 15:26:16.111566 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:16.111411 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" podUID="6da6535b-af05-4dbb-87f5-06eb503b80f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" Sep 08 15:26:17.111545 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:17.111485 2572 patch_prober.go:28] interesting pod/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" start-of-body= Sep 08 15:26:17.111941 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:17.111564 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" podUID="6da6535b-af05-4dbb-87f5-06eb503b80f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" Sep 08 15:26:18.111250 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:18.111214 2572 patch_prober.go:28] interesting pod/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" start-of-body= Sep 08 15:26:18.111415 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:18.111273 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" podUID="6da6535b-af05-4dbb-87f5-06eb503b80f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" Sep 08 15:26:19.111973 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:19.111937 2572 patch_prober.go:28] interesting pod/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" start-of-body= Sep 08 15:26:19.112368 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:19.111994 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" podUID="6da6535b-af05-4dbb-87f5-06eb503b80f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" Sep 08 15:26:20.111433 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:20.111389 2572 patch_prober.go:28] interesting pod/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" start-of-body= Sep 08 15:26:20.111615 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:20.111455 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" podUID="6da6535b-af05-4dbb-87f5-06eb503b80f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.23:15021/healthz/ready\": dial tcp 10.132.0.23:15021: connect: connection refused" Sep 08 15:26:20.260821 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:20.260785 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" Sep 08 15:26:21.114533 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:21.114507 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:26:21.114821 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:21.114800 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:26:21.115459 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:21.115438 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:26:23.232528 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:23.232475 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6" Sep 08 15:26:24.729107 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.729078 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6"] Sep 08 15:26:24.732649 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.732627 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.735668 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.735649 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-677a6c-openshift-default-dockercfg-qbv89\"" Sep 08 15:26:24.756238 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.756217 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6"] Sep 08 15:26:24.846785 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.846740 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-credential-socket\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.847029 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.846805 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-workload-socket\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.847029 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.846969 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq5kx\" (UniqueName: \"kubernetes.io/projected/bccbad6b-5994-4a9b-be33-057ac78e5dc5-kube-api-access-gq5kx\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.847151 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.847061 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-podinfo\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.847151 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.847142 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-envoy\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.847250 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.847167 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-token\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.847250 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.847191 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istiod-ca-cert\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.847349 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.847291 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-data\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.847404 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.847360 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-workload-certs\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.948327 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.948292 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-podinfo\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.948547 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.948362 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-envoy\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.948547 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.948387 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-token\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.948547 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.948423 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istiod-ca-cert\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.948547 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.948465 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-data\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.948769 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.948564 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-workload-certs\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.948769 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.948612 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-credential-socket\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.948769 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.948661 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-workload-socket\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.948769 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.948704 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq5kx\" (UniqueName: \"kubernetes.io/projected/bccbad6b-5994-4a9b-be33-057ac78e5dc5-kube-api-access-gq5kx\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.949455 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.949396 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-credential-socket\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.949941 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.949853 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-data\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.949941 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.949886 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-workload-socket\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.950152 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.950098 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istiod-ca-cert\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.950152 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.950112 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-workload-certs\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.951610 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.951586 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-envoy\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.951888 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.951869 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-podinfo\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.957974 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.957943 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-token\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:24.958244 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:24.958219 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq5kx\" (UniqueName: \"kubernetes.io/projected/bccbad6b-5994-4a9b-be33-057ac78e5dc5-kube-api-access-gq5kx\") pod \"e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:25.042608 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:25.042585 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:25.320436 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:25.320410 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6"] Sep 08 15:26:25.322354 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:26:25.322329 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbccbad6b_5994_4a9b_be33_057ac78e5dc5.slice/crio-71c61ffeb36311f1aebdf6b3baf707af4d7101624a266bf17ec0f3eb06b984cf WatchSource:0}: Error finding container 71c61ffeb36311f1aebdf6b3baf707af4d7101624a266bf17ec0f3eb06b984cf: Status 404 returned error can't find the container with id 71c61ffeb36311f1aebdf6b3baf707af4d7101624a266bf17ec0f3eb06b984cf Sep 08 15:26:25.324738 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:25.324697 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:26:25.324806 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:25.324782 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:26:25.324843 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:25.324824 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:26:26.304724 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:26.304691 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" event={"ID":"bccbad6b-5994-4a9b-be33-057ac78e5dc5","Type":"ContainerStarted","Data":"a45d366c91d1cbf581ce84e6a72178c32dce51700d304bbadd0a64dd035df463"} Sep 08 15:26:26.305153 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:26.304740 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" event={"ID":"bccbad6b-5994-4a9b-be33-057ac78e5dc5","Type":"ContainerStarted","Data":"71c61ffeb36311f1aebdf6b3baf707af4d7101624a266bf17ec0f3eb06b984cf"} Sep 08 15:26:26.325722 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:26.325673 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" podStartSLOduration=2.325657149 podStartE2EDuration="2.325657149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:26:26.323370803 +0000 UTC m=+1048.475549382" watchObservedRunningTime="2026-09-08 15:26:26.325657149 +0000 UTC m=+1048.477835728" Sep 08 15:26:27.043331 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:27.043291 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:27.048265 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:27.048244 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:27.309213 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:27.309131 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:27.310074 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:27.310056 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:30.060851 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.060813 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b"] Sep 08 15:26:30.063869 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.063853 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" Sep 08 15:26:30.070991 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.067400 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-677a6c-serving-cert\"" Sep 08 15:26:30.082870 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.082849 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b"] Sep 08 15:26:30.195865 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.195829 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghqpc\" (UniqueName: \"kubernetes.io/projected/29c11455-7c11-472b-ad6b-4d1168a62eb4-kube-api-access-ghqpc\") pod \"maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b\" (UID: \"29c11455-7c11-472b-ad6b-4d1168a62eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" Sep 08 15:26:30.196031 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.195883 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c11455-7c11-472b-ad6b-4d1168a62eb4-maas-api-tls\") pod \"maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b\" (UID: \"29c11455-7c11-472b-ad6b-4d1168a62eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" Sep 08 15:26:30.196031 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.195911 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c11455-7c11-472b-ad6b-4d1168a62eb4-metrics-certs\") pod \"maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b\" (UID: \"29c11455-7c11-472b-ad6b-4d1168a62eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" Sep 08 15:26:30.196031 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.195983 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c11455-7c11-472b-ad6b-4d1168a62eb4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b\" (UID: \"29c11455-7c11-472b-ad6b-4d1168a62eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" Sep 08 15:26:30.296739 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.296706 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c11455-7c11-472b-ad6b-4d1168a62eb4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b\" (UID: \"29c11455-7c11-472b-ad6b-4d1168a62eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" Sep 08 15:26:30.296887 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.296787 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghqpc\" (UniqueName: \"kubernetes.io/projected/29c11455-7c11-472b-ad6b-4d1168a62eb4-kube-api-access-ghqpc\") pod \"maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b\" (UID: \"29c11455-7c11-472b-ad6b-4d1168a62eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" Sep 08 15:26:30.296887 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.296812 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c11455-7c11-472b-ad6b-4d1168a62eb4-maas-api-tls\") pod \"maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b\" (UID: \"29c11455-7c11-472b-ad6b-4d1168a62eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" Sep 08 15:26:30.296887 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.296827 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c11455-7c11-472b-ad6b-4d1168a62eb4-metrics-certs\") pod \"maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b\" (UID: \"29c11455-7c11-472b-ad6b-4d1168a62eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" Sep 08 15:26:30.297056 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:26:30.296942 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-serving-cert: secret "maas-api-e2e-inf-b-677a6c-serving-cert" not found Sep 08 15:26:30.297056 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:26:30.297013 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c11455-7c11-472b-ad6b-4d1168a62eb4-maas-api-tls podName:29c11455-7c11-472b-ad6b-4d1168a62eb4 nodeName:}" failed. No retries permitted until 2026-09-08 15:26:30.796992346 +0000 UTC m=+1052.949170904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29c11455-7c11-472b-ad6b-4d1168a62eb4-maas-api-tls") pod "maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" (UID: "29c11455-7c11-472b-ad6b-4d1168a62eb4") : secret "maas-api-e2e-inf-b-677a6c-serving-cert" not found Sep 08 15:26:30.297427 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.297403 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c11455-7c11-472b-ad6b-4d1168a62eb4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b\" (UID: \"29c11455-7c11-472b-ad6b-4d1168a62eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" Sep 08 15:26:30.299336 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.299317 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c11455-7c11-472b-ad6b-4d1168a62eb4-metrics-certs\") pod \"maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b\" (UID: \"29c11455-7c11-472b-ad6b-4d1168a62eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" Sep 08 15:26:30.306082 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.306061 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghqpc\" (UniqueName: \"kubernetes.io/projected/29c11455-7c11-472b-ad6b-4d1168a62eb4-kube-api-access-ghqpc\") pod \"maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b\" (UID: \"29c11455-7c11-472b-ad6b-4d1168a62eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" Sep 08 15:26:30.766903 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.766870 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk"] Sep 08 15:26:30.770395 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.770376 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk" Sep 08 15:26:30.772783 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.772759 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-677a6c-dockercfg-7zd5m\"" Sep 08 15:26:30.772874 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.772766 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-677a6c\"" Sep 08 15:26:30.783740 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.783717 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk"] Sep 08 15:26:30.790403 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.790381 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc"] Sep 08 15:26:30.793463 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.793448 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc" Sep 08 15:26:30.801628 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.800980 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c11455-7c11-472b-ad6b-4d1168a62eb4-maas-api-tls\") pod \"maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b\" (UID: \"29c11455-7c11-472b-ad6b-4d1168a62eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" Sep 08 15:26:30.803941 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.803922 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c11455-7c11-472b-ad6b-4d1168a62eb4-maas-api-tls\") pod \"maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b\" (UID: \"29c11455-7c11-472b-ad6b-4d1168a62eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" Sep 08 15:26:30.805370 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.805347 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc"] Sep 08 15:26:30.902459 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.902428 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc\" (UID: \"3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc" Sep 08 15:26:30.902636 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.902464 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dfl9\" (UniqueName: \"kubernetes.io/projected/96c18853-0e4e-4b1e-bf35-51c326a9fe2f-kube-api-access-9dfl9\") pod \"payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk\" (UID: \"96c18853-0e4e-4b1e-bf35-51c326a9fe2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk" Sep 08 15:26:30.902636 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.902510 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96c18853-0e4e-4b1e-bf35-51c326a9fe2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk\" (UID: \"96c18853-0e4e-4b1e-bf35-51c326a9fe2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk" Sep 08 15:26:30.902636 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.902605 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxpx2\" (UniqueName: \"kubernetes.io/projected/3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0-kube-api-access-sxpx2\") pod \"payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc\" (UID: \"3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc" Sep 08 15:26:30.976105 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:30.976076 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" Sep 08 15:26:31.003250 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:31.003222 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxpx2\" (UniqueName: \"kubernetes.io/projected/3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0-kube-api-access-sxpx2\") pod \"payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc\" (UID: \"3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc" Sep 08 15:26:31.003358 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:31.003301 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc\" (UID: \"3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc" Sep 08 15:26:31.003358 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:31.003322 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dfl9\" (UniqueName: \"kubernetes.io/projected/96c18853-0e4e-4b1e-bf35-51c326a9fe2f-kube-api-access-9dfl9\") pod \"payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk\" (UID: \"96c18853-0e4e-4b1e-bf35-51c326a9fe2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk" Sep 08 15:26:31.003358 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:31.003341 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96c18853-0e4e-4b1e-bf35-51c326a9fe2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk\" (UID: \"96c18853-0e4e-4b1e-bf35-51c326a9fe2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk" Sep 08 15:26:31.004009 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:31.003987 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96c18853-0e4e-4b1e-bf35-51c326a9fe2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk\" (UID: \"96c18853-0e4e-4b1e-bf35-51c326a9fe2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk" Sep 08 15:26:31.004085 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:31.004017 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc\" (UID: \"3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc" Sep 08 15:26:31.012686 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:31.012660 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dfl9\" (UniqueName: \"kubernetes.io/projected/96c18853-0e4e-4b1e-bf35-51c326a9fe2f-kube-api-access-9dfl9\") pod \"payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk\" (UID: \"96c18853-0e4e-4b1e-bf35-51c326a9fe2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk" Sep 08 15:26:31.012794 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:31.012667 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxpx2\" (UniqueName: \"kubernetes.io/projected/3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0-kube-api-access-sxpx2\") pod \"payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc\" (UID: \"3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc" Sep 08 15:26:31.081362 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:31.081330 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk" Sep 08 15:26:31.103701 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:31.103647 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc" Sep 08 15:26:31.261126 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:31.261094 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b"] Sep 08 15:26:31.264034 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:26:31.264005 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c11455_7c11_472b_ad6b_4d1168a62eb4.slice/crio-5004dbb0f85da0d54bf0b34ed2b85068508f3c0f6ce5ffb1875177efbad19e09 WatchSource:0}: Error finding container 5004dbb0f85da0d54bf0b34ed2b85068508f3c0f6ce5ffb1875177efbad19e09: Status 404 returned error can't find the container with id 5004dbb0f85da0d54bf0b34ed2b85068508f3c0f6ce5ffb1875177efbad19e09 Sep 08 15:26:31.329434 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:31.329374 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" event={"ID":"29c11455-7c11-472b-ad6b-4d1168a62eb4","Type":"ContainerStarted","Data":"5004dbb0f85da0d54bf0b34ed2b85068508f3c0f6ce5ffb1875177efbad19e09"} Sep 08 15:26:31.357536 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:31.357513 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk"] Sep 08 15:26:31.359329 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:26:31.359297 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c18853_0e4e_4b1e_bf35_51c326a9fe2f.slice/crio-ee5b07660a3f383b8d5d47057d012d19788d230b23ac15dc7eac7fd80a086e1d WatchSource:0}: Error finding container ee5b07660a3f383b8d5d47057d012d19788d230b23ac15dc7eac7fd80a086e1d: Status 404 returned error can't find the container with id ee5b07660a3f383b8d5d47057d012d19788d230b23ac15dc7eac7fd80a086e1d Sep 08 15:26:31.396597 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:31.396569 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc"] Sep 08 15:26:31.396927 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:26:31.396906 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf0bc11_8e1b_4ab9_b67c_9ec7571094d0.slice/crio-f393a15d9dcf0c471964c7ae61f34b4beaa6e4f0a5bb96ab80286401e5358c4e WatchSource:0}: Error finding container f393a15d9dcf0c471964c7ae61f34b4beaa6e4f0a5bb96ab80286401e5358c4e: Status 404 returned error can't find the container with id f393a15d9dcf0c471964c7ae61f34b4beaa6e4f0a5bb96ab80286401e5358c4e Sep 08 15:26:32.335910 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:32.335876 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk" event={"ID":"96c18853-0e4e-4b1e-bf35-51c326a9fe2f","Type":"ContainerStarted","Data":"2bd1d51ed025f8cd318f8f681b336418dd05265e9a4b3e66c10eb0bb0e9356da"} Sep 08 15:26:32.336439 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:32.335938 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk" event={"ID":"96c18853-0e4e-4b1e-bf35-51c326a9fe2f","Type":"ContainerStarted","Data":"ee5b07660a3f383b8d5d47057d012d19788d230b23ac15dc7eac7fd80a086e1d"} Sep 08 15:26:32.336439 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:32.336013 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk" Sep 08 15:26:32.337422 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:32.337401 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" event={"ID":"29c11455-7c11-472b-ad6b-4d1168a62eb4","Type":"ContainerStarted","Data":"195a6e2dc3084f3998a14d6646eda63ba8ec69e62d5a2c3cb63c1136190e761e"} Sep 08 15:26:32.337483 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:32.337448 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" Sep 08 15:26:32.339054 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:32.339034 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc" event={"ID":"3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0","Type":"ContainerStarted","Data":"8088edc1336ac8c63cdce115319ad9d5453abb1e6c6a2c8a6a8c099b4a027260"} Sep 08 15:26:32.339179 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:32.339069 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc" event={"ID":"3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0","Type":"ContainerStarted","Data":"f393a15d9dcf0c471964c7ae61f34b4beaa6e4f0a5bb96ab80286401e5358c4e"} Sep 08 15:26:32.339179 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:32.339098 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc" Sep 08 15:26:32.355595 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:32.355529 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk" podStartSLOduration=2.355477914 podStartE2EDuration="2.355477914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:26:32.355282797 +0000 UTC m=+1054.507461389" watchObservedRunningTime="2026-09-08 15:26:32.355477914 +0000 UTC m=+1054.507656491" Sep 08 15:26:32.375968 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:32.375930 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc" podStartSLOduration=2.37591743 podStartE2EDuration="2.37591743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:26:32.374473178 +0000 UTC m=+1054.526651758" watchObservedRunningTime="2026-09-08 15:26:32.37591743 +0000 UTC m=+1054.528096009" Sep 08 15:26:32.396029 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:32.395993 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" podStartSLOduration=1.825533353 podStartE2EDuration="2.395980633s" totalImagesPullingTime="570.44728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:26:32.394653227 +0000 UTC m=+1054.546831808" watchObservedRunningTime="2026-09-08 15:26:32.395980633 +0000 UTC m=+1054.548159254" Sep 08 15:26:38.349452 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:38.349423 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" Sep 08 15:26:43.348144 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:43.348115 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc" Sep 08 15:26:43.349652 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:43.349627 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk" Sep 08 15:26:48.415236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:48.415204 2572 generic.go:356] "Generic (PLEG): container finished" podID="bccbad6b-5994-4a9b-be33-057ac78e5dc5" containerID="a45d366c91d1cbf581ce84e6a72178c32dce51700d304bbadd0a64dd035df463" exitCode=0 Sep 08 15:26:48.415672 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:48.415266 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" event={"ID":"bccbad6b-5994-4a9b-be33-057ac78e5dc5","Type":"ContainerDied","Data":"a45d366c91d1cbf581ce84e6a72178c32dce51700d304bbadd0a64dd035df463"} Sep 08 15:26:48.415672 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:48.415612 2572 scope.go:117] "RemoveContainer" containerID="a45d366c91d1cbf581ce84e6a72178c32dce51700d304bbadd0a64dd035df463" Sep 08 15:26:48.416697 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:48.416657 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:26:48.416810 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:48.416750 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:26:48.416810 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:48.416804 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:26:48.945017 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:48.943613 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l"] Sep 08 15:26:48.945017 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:48.943860 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" podUID="6c15a87c-aa8f-4d3f-bf71-bac2de5c3471" containerName="maas-api" containerID="cri-o://eb6b3107eedc102e896d294d6e9702d9a12aee807db5113013b38f7a273ef1fe" gracePeriod=30 Sep 08 15:26:49.212161 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.212137 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" Sep 08 15:26:49.256528 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.255825 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-openshift-service-ca\") pod \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\" (UID: \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\") " Sep 08 15:26:49.256528 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.255911 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-maas-api-tls\") pod \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\" (UID: \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\") " Sep 08 15:26:49.256528 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.256021 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-metrics-certs\") pod \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\" (UID: \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\") " Sep 08 15:26:49.256528 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.256056 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzzck\" (UniqueName: \"kubernetes.io/projected/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-kube-api-access-gzzck\") pod \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\" (UID: \"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471\") " Sep 08 15:26:49.257587 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.257532 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c15a87c-aa8f-4d3f-bf71-bac2de5c3471" (UID: "6c15a87c-aa8f-4d3f-bf71-bac2de5c3471"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:26:49.264262 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.264201 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-kube-api-access-gzzck" (OuterVolumeSpecName: "kube-api-access-gzzck") pod "6c15a87c-aa8f-4d3f-bf71-bac2de5c3471" (UID: "6c15a87c-aa8f-4d3f-bf71-bac2de5c3471"). InnerVolumeSpecName "kube-api-access-gzzck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:26:49.264262 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.264217 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c15a87c-aa8f-4d3f-bf71-bac2de5c3471" (UID: "6c15a87c-aa8f-4d3f-bf71-bac2de5c3471"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 15:26:49.264735 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.264705 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6c15a87c-aa8f-4d3f-bf71-bac2de5c3471" (UID: "6c15a87c-aa8f-4d3f-bf71-bac2de5c3471"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 15:26:49.359547 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.358055 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-metrics-certs\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:26:49.359547 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.358095 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzzck\" (UniqueName: \"kubernetes.io/projected/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-kube-api-access-gzzck\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:26:49.359547 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.358115 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-openshift-service-ca\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:26:49.359547 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.358128 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471-maas-api-tls\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:26:49.421688 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.421653 2572 generic.go:356] "Generic (PLEG): container finished" podID="6c15a87c-aa8f-4d3f-bf71-bac2de5c3471" containerID="eb6b3107eedc102e896d294d6e9702d9a12aee807db5113013b38f7a273ef1fe" exitCode=0 Sep 08 15:26:49.422125 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.421695 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" Sep 08 15:26:49.422125 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.421750 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" event={"ID":"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471","Type":"ContainerDied","Data":"eb6b3107eedc102e896d294d6e9702d9a12aee807db5113013b38f7a273ef1fe"} Sep 08 15:26:49.422125 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.421809 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l" event={"ID":"6c15a87c-aa8f-4d3f-bf71-bac2de5c3471","Type":"ContainerDied","Data":"abd2117da3d0ae45e0563eef44594b12e973bd03da2ae7b3be8806ed97365d10"} Sep 08 15:26:49.422125 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.421834 2572 scope.go:117] "RemoveContainer" containerID="eb6b3107eedc102e896d294d6e9702d9a12aee807db5113013b38f7a273ef1fe" Sep 08 15:26:49.424057 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.424019 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" event={"ID":"bccbad6b-5994-4a9b-be33-057ac78e5dc5","Type":"ContainerStarted","Data":"6d5c36f8bd8dd3a5c56249c86c1abe1d3a142453fa12edd83594ca2277884577"} Sep 08 15:26:49.431439 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.431420 2572 scope.go:117] "RemoveContainer" containerID="eb6b3107eedc102e896d294d6e9702d9a12aee807db5113013b38f7a273ef1fe" Sep 08 15:26:49.431812 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:26:49.431794 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb6b3107eedc102e896d294d6e9702d9a12aee807db5113013b38f7a273ef1fe\": container with ID starting with eb6b3107eedc102e896d294d6e9702d9a12aee807db5113013b38f7a273ef1fe not found: ID does not exist" containerID="eb6b3107eedc102e896d294d6e9702d9a12aee807db5113013b38f7a273ef1fe" Sep 08 15:26:49.431875 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.431821 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb6b3107eedc102e896d294d6e9702d9a12aee807db5113013b38f7a273ef1fe"} err="failed to get container status \"eb6b3107eedc102e896d294d6e9702d9a12aee807db5113013b38f7a273ef1fe\": rpc error: code = NotFound desc = could not find container \"eb6b3107eedc102e896d294d6e9702d9a12aee807db5113013b38f7a273ef1fe\": container with ID starting with eb6b3107eedc102e896d294d6e9702d9a12aee807db5113013b38f7a273ef1fe not found: ID does not exist" Sep 08 15:26:49.465872 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.465701 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l"] Sep 08 15:26:49.466561 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:49.466534 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-062f32f0-6db94979b6-k4j5l"] Sep 08 15:26:50.043690 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:50.043651 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:50.043897 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:50.043827 2572 patch_prober.go:28] interesting pod/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body= Sep 08 15:26:50.043897 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:50.043876 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" podUID="bccbad6b-5994-4a9b-be33-057ac78e5dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" Sep 08 15:26:50.374767 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:50.374671 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c15a87c-aa8f-4d3f-bf71-bac2de5c3471" path="/var/lib/kubelet/pods/6c15a87c-aa8f-4d3f-bf71-bac2de5c3471/volumes" Sep 08 15:26:51.043037 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:51.042985 2572 patch_prober.go:28] interesting pod/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body= Sep 08 15:26:51.043510 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:51.043066 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" podUID="bccbad6b-5994-4a9b-be33-057ac78e5dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" Sep 08 15:26:52.043163 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:52.043125 2572 patch_prober.go:28] interesting pod/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body= Sep 08 15:26:52.043878 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:52.043843 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" podUID="bccbad6b-5994-4a9b-be33-057ac78e5dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" Sep 08 15:26:53.044248 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:53.043769 2572 patch_prober.go:28] interesting pod/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body= Sep 08 15:26:53.044248 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:53.043838 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" podUID="bccbad6b-5994-4a9b-be33-057ac78e5dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" Sep 08 15:26:54.043177 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:54.043135 2572 patch_prober.go:28] interesting pod/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body= Sep 08 15:26:54.043602 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:54.043573 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" podUID="bccbad6b-5994-4a9b-be33-057ac78e5dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" Sep 08 15:26:54.242111 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:54.242066 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5"] Sep 08 15:26:54.242593 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:54.242402 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c15a87c-aa8f-4d3f-bf71-bac2de5c3471" containerName="maas-api" Sep 08 15:26:54.242593 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:54.242413 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c15a87c-aa8f-4d3f-bf71-bac2de5c3471" containerName="maas-api" Sep 08 15:26:54.242593 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:54.242473 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c15a87c-aa8f-4d3f-bf71-bac2de5c3471" containerName="maas-api" Sep 08 15:26:54.246824 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:54.246794 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5" Sep 08 15:26:54.248952 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:54.248924 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d1aa3ad8\"" Sep 08 15:26:54.249100 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:54.248988 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d1aa3ad8-dockercfg-5xx27\"" Sep 08 15:26:54.254399 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:54.254374 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5"] Sep 08 15:26:54.307543 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:54.307435 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hflzq\" (UniqueName: \"kubernetes.io/projected/132744a4-326a-4c25-b671-58cb3708dfdd-kube-api-access-hflzq\") pod \"payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5\" (UID: \"132744a4-326a-4c25-b671-58cb3708dfdd\") " pod="openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5" Sep 08 15:26:54.307543 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:54.307476 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/132744a4-326a-4c25-b671-58cb3708dfdd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5\" (UID: \"132744a4-326a-4c25-b671-58cb3708dfdd\") " pod="openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5" Sep 08 15:26:54.408453 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:54.408418 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hflzq\" (UniqueName: \"kubernetes.io/projected/132744a4-326a-4c25-b671-58cb3708dfdd-kube-api-access-hflzq\") pod \"payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5\" (UID: \"132744a4-326a-4c25-b671-58cb3708dfdd\") " pod="openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5" Sep 08 15:26:54.408671 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:54.408462 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/132744a4-326a-4c25-b671-58cb3708dfdd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5\" (UID: \"132744a4-326a-4c25-b671-58cb3708dfdd\") " pod="openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5" Sep 08 15:26:54.409121 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:54.409100 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/132744a4-326a-4c25-b671-58cb3708dfdd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5\" (UID: \"132744a4-326a-4c25-b671-58cb3708dfdd\") " pod="openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5" Sep 08 15:26:54.416826 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:54.416802 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hflzq\" (UniqueName: \"kubernetes.io/projected/132744a4-326a-4c25-b671-58cb3708dfdd-kube-api-access-hflzq\") pod \"payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5\" (UID: \"132744a4-326a-4c25-b671-58cb3708dfdd\") " pod="openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5" Sep 08 15:26:54.558198 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:54.558091 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5" Sep 08 15:26:54.840248 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:54.840221 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5"] Sep 08 15:26:54.842317 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:26:54.842285 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132744a4_326a_4c25_b671_58cb3708dfdd.slice/crio-24d65255a05e40a6652d1f66422cddebe17d81f98a7306bf856892b7b2c310ca WatchSource:0}: Error finding container 24d65255a05e40a6652d1f66422cddebe17d81f98a7306bf856892b7b2c310ca: Status 404 returned error can't find the container with id 24d65255a05e40a6652d1f66422cddebe17d81f98a7306bf856892b7b2c310ca Sep 08 15:26:55.047035 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:55.047010 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:55.047406 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:55.047386 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:55.048266 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:55.048246 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:26:55.458458 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:55.458420 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5" event={"ID":"132744a4-326a-4c25-b671-58cb3708dfdd","Type":"ContainerStarted","Data":"85835cc1c4c78062d9740d1efaf87a36aee47a61167d779d5a38b5aa219fc9e5"} Sep 08 15:26:55.458875 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:55.458529 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5" event={"ID":"132744a4-326a-4c25-b671-58cb3708dfdd","Type":"ContainerStarted","Data":"24d65255a05e40a6652d1f66422cddebe17d81f98a7306bf856892b7b2c310ca"} Sep 08 15:26:55.458875 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:55.458573 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5" Sep 08 15:26:55.478614 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:26:55.478559 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5" podStartSLOduration=1.4785401089999999 podStartE2EDuration="1.478540109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:26:55.4772936 +0000 UTC m=+1077.629472180" watchObservedRunningTime="2026-09-08 15:26:55.478540109 +0000 UTC m=+1077.630718693" Sep 08 15:27:06.464563 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:06.464531 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5" Sep 08 15:27:18.349130 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:18.349100 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h"] Sep 08 15:27:18.353687 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:18.353671 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h" Sep 08 15:27:18.356189 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:18.356159 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8qqvg\"" Sep 08 15:27:18.359617 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:18.359591 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h"] Sep 08 15:27:18.510321 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:18.510288 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22e76ac5-cc27-45d8-8297-b37cb13cec29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h\" (UID: \"22e76ac5-cc27-45d8-8297-b37cb13cec29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h" Sep 08 15:27:18.611364 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:18.611303 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22e76ac5-cc27-45d8-8297-b37cb13cec29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h\" (UID: \"22e76ac5-cc27-45d8-8297-b37cb13cec29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h" Sep 08 15:27:18.611861 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:18.611842 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22e76ac5-cc27-45d8-8297-b37cb13cec29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h\" (UID: \"22e76ac5-cc27-45d8-8297-b37cb13cec29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h" Sep 08 15:27:18.665128 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:18.665106 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h" Sep 08 15:27:18.934809 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:18.934788 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h"] Sep 08 15:27:18.937401 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:27:18.937374 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e76ac5_cc27_45d8_8297_b37cb13cec29.slice/crio-ebc9c14e41a9ec480a0ef08be6fc204af06c7673d70304943edd2403f351fdbb WatchSource:0}: Error finding container ebc9c14e41a9ec480a0ef08be6fc204af06c7673d70304943edd2403f351fdbb: Status 404 returned error can't find the container with id ebc9c14e41a9ec480a0ef08be6fc204af06c7673d70304943edd2403f351fdbb Sep 08 15:27:19.561746 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:19.561710 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h" event={"ID":"22e76ac5-cc27-45d8-8297-b37cb13cec29","Type":"ContainerStarted","Data":"ebc9c14e41a9ec480a0ef08be6fc204af06c7673d70304943edd2403f351fdbb"} Sep 08 15:27:21.571232 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:21.571179 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h" event={"ID":"22e76ac5-cc27-45d8-8297-b37cb13cec29","Type":"ContainerStarted","Data":"d9be5bcd40dc6a66f242207da8333b9b49c369ea81f44bbcb7c51346b1169972"} Sep 08 15:27:21.594369 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:21.594328 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h" podStartSLOduration=1.232235696 podStartE2EDuration="3.594314481s" totalImagesPullingTime="2.362078785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:27:21.592823375 +0000 UTC m=+1103.745001953" watchObservedRunningTime="2026-09-08 15:27:21.594314481 +0000 UTC m=+1103.746493059" Sep 08 15:27:22.576365 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:22.576334 2572 generic.go:356] "Generic (PLEG): container finished" podID="22e76ac5-cc27-45d8-8297-b37cb13cec29" containerID="d9be5bcd40dc6a66f242207da8333b9b49c369ea81f44bbcb7c51346b1169972" exitCode=0 Sep 08 15:27:22.576751 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:22.576416 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h" event={"ID":"22e76ac5-cc27-45d8-8297-b37cb13cec29","Type":"ContainerDied","Data":"d9be5bcd40dc6a66f242207da8333b9b49c369ea81f44bbcb7c51346b1169972"} Sep 08 15:27:23.702325 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:23.702302 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h" Sep 08 15:27:23.854114 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:23.854061 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22e76ac5-cc27-45d8-8297-b37cb13cec29-maas-api-service-ca\") pod \"22e76ac5-cc27-45d8-8297-b37cb13cec29\" (UID: \"22e76ac5-cc27-45d8-8297-b37cb13cec29\") " Sep 08 15:27:23.854365 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:23.854344 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e76ac5-cc27-45d8-8297-b37cb13cec29-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22e76ac5-cc27-45d8-8297-b37cb13cec29" (UID: "22e76ac5-cc27-45d8-8297-b37cb13cec29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:27:23.955021 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:23.955000 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22e76ac5-cc27-45d8-8297-b37cb13cec29-maas-api-service-ca\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:27:24.585768 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:24.585739 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h" event={"ID":"22e76ac5-cc27-45d8-8297-b37cb13cec29","Type":"ContainerDied","Data":"ebc9c14e41a9ec480a0ef08be6fc204af06c7673d70304943edd2403f351fdbb"} Sep 08 15:27:24.585895 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:24.585771 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h" Sep 08 15:27:24.585895 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:24.585791 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebc9c14e41a9ec480a0ef08be6fc204af06c7673d70304943edd2403f351fdbb" Sep 08 15:27:31.852847 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:31.852815 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc9dc-w5dkk"] Sep 08 15:27:31.853213 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:31.853160 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22e76ac5-cc27-45d8-8297-b37cb13cec29" containerName="revoke-keys" Sep 08 15:27:31.853213 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:31.853170 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="22e76ac5-cc27-45d8-8297-b37cb13cec29" containerName="revoke-keys" Sep 08 15:27:31.853284 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:31.853223 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="22e76ac5-cc27-45d8-8297-b37cb13cec29" containerName="revoke-keys" Sep 08 15:27:31.856186 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:31.856166 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc9dc-w5dkk" Sep 08 15:27:31.858277 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:31.858256 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8qqvg\"" Sep 08 15:27:31.863850 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:31.863826 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc9dc-w5dkk"] Sep 08 15:27:32.017630 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:32.017598 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-3bc9dc-w5dkk\" (UID: \"5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc9dc-w5dkk" Sep 08 15:27:32.118738 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:32.118661 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-3bc9dc-w5dkk\" (UID: \"5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc9dc-w5dkk" Sep 08 15:27:32.119269 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:32.119246 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-3bc9dc-w5dkk\" (UID: \"5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc9dc-w5dkk" Sep 08 15:27:32.167699 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:32.167672 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc9dc-w5dkk" Sep 08 15:27:32.435667 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:32.435645 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc9dc-w5dkk"] Sep 08 15:27:32.438071 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:27:32.438043 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd544fd_7aa2_44f3_89c3_6b6dbea11e3c.slice/crio-1788dce2294ff2b51a781e155a6d1b8e9aadf7effd8ea06b4525c204a3db336b WatchSource:0}: Error finding container 1788dce2294ff2b51a781e155a6d1b8e9aadf7effd8ea06b4525c204a3db336b: Status 404 returned error can't find the container with id 1788dce2294ff2b51a781e155a6d1b8e9aadf7effd8ea06b4525c204a3db336b Sep 08 15:27:32.619550 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:32.619521 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc9dc-w5dkk" event={"ID":"5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c","Type":"ContainerStarted","Data":"1788dce2294ff2b51a781e155a6d1b8e9aadf7effd8ea06b4525c204a3db336b"} Sep 08 15:27:33.627724 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:33.627681 2572 generic.go:356] "Generic (PLEG): container finished" podID="5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c" containerID="acc541eb906b997abb315ece906b96c387de39760f4a0168acdb30e6cb7b9466" exitCode=0 Sep 08 15:27:33.628172 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:33.627768 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc9dc-w5dkk" event={"ID":"5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c","Type":"ContainerDied","Data":"acc541eb906b997abb315ece906b96c387de39760f4a0168acdb30e6cb7b9466"} Sep 08 15:27:34.764784 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:34.764759 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc9dc-w5dkk" Sep 08 15:27:34.942234 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:34.942155 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c-maas-api-service-ca\") pod \"5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c\" (UID: \"5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c\") " Sep 08 15:27:34.942539 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:34.942482 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c" (UID: "5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:27:35.043230 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:35.043198 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c-maas-api-service-ca\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:27:35.426274 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:35.426242 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5"] Sep 08 15:27:35.426562 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:35.426515 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5" podUID="132744a4-326a-4c25-b671-58cb3708dfdd" containerName="payload-processing" containerID="cri-o://85835cc1c4c78062d9740d1efaf87a36aee47a61167d779d5a38b5aa219fc9e5" gracePeriod=30 Sep 08 15:27:35.641609 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:35.641576 2572 generic.go:356] "Generic (PLEG): container finished" podID="132744a4-326a-4c25-b671-58cb3708dfdd" containerID="85835cc1c4c78062d9740d1efaf87a36aee47a61167d779d5a38b5aa219fc9e5" exitCode=0 Sep 08 15:27:35.641758 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:35.641633 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5" event={"ID":"132744a4-326a-4c25-b671-58cb3708dfdd","Type":"ContainerDied","Data":"85835cc1c4c78062d9740d1efaf87a36aee47a61167d779d5a38b5aa219fc9e5"} Sep 08 15:27:35.643129 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:35.643106 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc9dc-w5dkk" event={"ID":"5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c","Type":"ContainerDied","Data":"1788dce2294ff2b51a781e155a6d1b8e9aadf7effd8ea06b4525c204a3db336b"} Sep 08 15:27:35.643297 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:35.643147 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc9dc-w5dkk" Sep 08 15:27:35.643297 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:35.643148 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1788dce2294ff2b51a781e155a6d1b8e9aadf7effd8ea06b4525c204a3db336b" Sep 08 15:27:35.681376 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:35.681319 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5" Sep 08 15:27:35.683087 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:35.683063 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h"] Sep 08 15:27:35.686624 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:35.686603 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d1aa3ad8-7rb9h"] Sep 08 15:27:35.849748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:35.849717 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hflzq\" (UniqueName: \"kubernetes.io/projected/132744a4-326a-4c25-b671-58cb3708dfdd-kube-api-access-hflzq\") pod \"132744a4-326a-4c25-b671-58cb3708dfdd\" (UID: \"132744a4-326a-4c25-b671-58cb3708dfdd\") " Sep 08 15:27:35.850095 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:35.849763 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/132744a4-326a-4c25-b671-58cb3708dfdd-plugins-config-volume\") pod \"132744a4-326a-4c25-b671-58cb3708dfdd\" (UID: \"132744a4-326a-4c25-b671-58cb3708dfdd\") " Sep 08 15:27:35.850210 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:35.850184 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/132744a4-326a-4c25-b671-58cb3708dfdd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "132744a4-326a-4c25-b671-58cb3708dfdd" (UID: "132744a4-326a-4c25-b671-58cb3708dfdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:27:35.852055 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:35.852036 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132744a4-326a-4c25-b671-58cb3708dfdd-kube-api-access-hflzq" (OuterVolumeSpecName: "kube-api-access-hflzq") pod "132744a4-326a-4c25-b671-58cb3708dfdd" (UID: "132744a4-326a-4c25-b671-58cb3708dfdd"). InnerVolumeSpecName "kube-api-access-hflzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:27:35.950830 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:35.950748 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hflzq\" (UniqueName: \"kubernetes.io/projected/132744a4-326a-4c25-b671-58cb3708dfdd-kube-api-access-hflzq\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:27:35.950830 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:35.950775 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/132744a4-326a-4c25-b671-58cb3708dfdd-plugins-config-volume\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:27:36.370824 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.370795 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22e76ac5-cc27-45d8-8297-b37cb13cec29" path="/var/lib/kubelet/pods/22e76ac5-cc27-45d8-8297-b37cb13cec29/volumes" Sep 08 15:27:36.598519 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.598473 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb"] Sep 08 15:27:36.598831 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.598819 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="132744a4-326a-4c25-b671-58cb3708dfdd" containerName="payload-processing" Sep 08 15:27:36.598878 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.598833 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="132744a4-326a-4c25-b671-58cb3708dfdd" containerName="payload-processing" Sep 08 15:27:36.598878 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.598853 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c" containerName="revoke-keys" Sep 08 15:27:36.598878 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.598858 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c" containerName="revoke-keys" Sep 08 15:27:36.598956 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.598915 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c" containerName="revoke-keys" Sep 08 15:27:36.598956 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.598924 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="132744a4-326a-4c25-b671-58cb3708dfdd" containerName="payload-processing" Sep 08 15:27:36.601968 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.601952 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.605475 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.605459 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9e22317f-openshift-default-dockercfg-4frd7\"" Sep 08 15:27:36.621009 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.620954 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb"] Sep 08 15:27:36.648550 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.648518 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5" event={"ID":"132744a4-326a-4c25-b671-58cb3708dfdd","Type":"ContainerDied","Data":"24d65255a05e40a6652d1f66422cddebe17d81f98a7306bf856892b7b2c310ca"} Sep 08 15:27:36.648675 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.648577 2572 scope.go:117] "RemoveContainer" containerID="85835cc1c4c78062d9740d1efaf87a36aee47a61167d779d5a38b5aa219fc9e5" Sep 08 15:27:36.648675 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.648529 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5" Sep 08 15:27:36.664782 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.664756 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5"] Sep 08 15:27:36.668644 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.666766 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d1aa3ad8-5575964694-8nht5"] Sep 08 15:27:36.758184 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.758141 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-podinfo\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.758374 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.758192 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j4c8\" (UniqueName: \"kubernetes.io/projected/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-kube-api-access-7j4c8\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.758374 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.758248 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-credential-socket\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.758374 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.758299 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-workload-socket\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.758374 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.758320 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-data\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.758557 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.758382 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-workload-certs\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.758557 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.758451 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istiod-ca-cert\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.758557 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.758472 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-token\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.758557 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.758522 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-envoy\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.859180 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.859143 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istiod-ca-cert\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.859180 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.859180 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-token\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.859702 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.859198 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-envoy\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.859702 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.859246 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-podinfo\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.859702 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.859277 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j4c8\" (UniqueName: \"kubernetes.io/projected/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-kube-api-access-7j4c8\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.859702 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.859432 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-credential-socket\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.859702 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.859544 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-workload-socket\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.859702 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.859572 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-data\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.859702 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.859634 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-workload-certs\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.860203 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.859829 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-credential-socket\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.860203 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.859912 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-workload-socket\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.860203 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.859955 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-workload-certs\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.860203 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.859982 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istiod-ca-cert\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.860203 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.860035 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-data\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.861701 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.861680 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-envoy\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.861997 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.861977 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-podinfo\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.874970 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.874915 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-token\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.876121 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.876106 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j4c8\" (UniqueName: \"kubernetes.io/projected/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-kube-api-access-7j4c8\") pod \"e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:36.912413 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:36.912380 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:37.186967 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:37.186886 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb"] Sep 08 15:27:37.189406 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:27:37.189376 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9c4124_e078_45a9_a671_16fb1a1c7ea0.slice/crio-4ad058da7ab002d6a9fb207bf7c378c4e2aad0e00a16abca8209bc50c15c48fb WatchSource:0}: Error finding container 4ad058da7ab002d6a9fb207bf7c378c4e2aad0e00a16abca8209bc50c15c48fb: Status 404 returned error can't find the container with id 4ad058da7ab002d6a9fb207bf7c378c4e2aad0e00a16abca8209bc50c15c48fb Sep 08 15:27:37.191599 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:37.191570 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:27:37.191657 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:37.191633 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:27:37.191691 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:37.191661 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:27:37.659170 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:37.659129 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" event={"ID":"6c9c4124-e078-45a9-a671-16fb1a1c7ea0","Type":"ContainerStarted","Data":"b7be2fa31005692ee82be5d419f7d427ccafdc262ee6fe597b8c6ef8c80d9d8a"} Sep 08 15:27:37.659315 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:37.659197 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" event={"ID":"6c9c4124-e078-45a9-a671-16fb1a1c7ea0","Type":"ContainerStarted","Data":"4ad058da7ab002d6a9fb207bf7c378c4e2aad0e00a16abca8209bc50c15c48fb"} Sep 08 15:27:37.680731 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:37.680676 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" podStartSLOduration=1.6806590350000001 podStartE2EDuration="1.680659035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:27:37.679583237 +0000 UTC m=+1119.831761819" watchObservedRunningTime="2026-09-08 15:27:37.680659035 +0000 UTC m=+1119.832837614" Sep 08 15:27:37.913282 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:37.913188 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:38.372037 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:38.372005 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="132744a4-326a-4c25-b671-58cb3708dfdd" path="/var/lib/kubelet/pods/132744a4-326a-4c25-b671-58cb3708dfdd/volumes" Sep 08 15:27:38.918379 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:38.918342 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:39.667450 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:39.667419 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:39.668581 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:39.668563 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:42.102781 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.102748 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6"] Sep 08 15:27:42.103214 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.103013 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6" podUID="6e7e3a85-e48c-438d-9f30-59412276e100" containerName="payload-pre-processing" containerID="cri-o://b2d744ec534f8c1acbe7610a2ab1394f371eb3a3c6d6fbfa4a2c5ee0768bfa0e" gracePeriod=30 Sep 08 15:27:42.316773 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.316742 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc9dc-w5dkk"] Sep 08 15:27:42.324732 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.324708 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc9dc-w5dkk"] Sep 08 15:27:42.347376 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.347354 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6" Sep 08 15:27:42.371092 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.371037 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c" path="/var/lib/kubelet/pods/5fd544fd-7aa2-44f3-89c3-6b6dbea11e3c/volumes" Sep 08 15:27:42.507538 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.507509 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmckh\" (UniqueName: \"kubernetes.io/projected/6e7e3a85-e48c-438d-9f30-59412276e100-kube-api-access-kmckh\") pod \"6e7e3a85-e48c-438d-9f30-59412276e100\" (UID: \"6e7e3a85-e48c-438d-9f30-59412276e100\") " Sep 08 15:27:42.507672 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.507594 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7e3a85-e48c-438d-9f30-59412276e100-plugins-config-volume\") pod \"6e7e3a85-e48c-438d-9f30-59412276e100\" (UID: \"6e7e3a85-e48c-438d-9f30-59412276e100\") " Sep 08 15:27:42.507946 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.507928 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e7e3a85-e48c-438d-9f30-59412276e100-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e7e3a85-e48c-438d-9f30-59412276e100" (UID: "6e7e3a85-e48c-438d-9f30-59412276e100"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:27:42.509615 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.509591 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7e3a85-e48c-438d-9f30-59412276e100-kube-api-access-kmckh" (OuterVolumeSpecName: "kube-api-access-kmckh") pod "6e7e3a85-e48c-438d-9f30-59412276e100" (UID: "6e7e3a85-e48c-438d-9f30-59412276e100"). InnerVolumeSpecName "kube-api-access-kmckh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:27:42.608525 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.608474 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7e3a85-e48c-438d-9f30-59412276e100-plugins-config-volume\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:27:42.608525 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.608516 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmckh\" (UniqueName: \"kubernetes.io/projected/6e7e3a85-e48c-438d-9f30-59412276e100-kube-api-access-kmckh\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:27:42.682689 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.682602 2572 generic.go:356] "Generic (PLEG): container finished" podID="6e7e3a85-e48c-438d-9f30-59412276e100" containerID="b2d744ec534f8c1acbe7610a2ab1394f371eb3a3c6d6fbfa4a2c5ee0768bfa0e" exitCode=0 Sep 08 15:27:42.682833 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.682690 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6" Sep 08 15:27:42.682833 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.682687 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6" event={"ID":"6e7e3a85-e48c-438d-9f30-59412276e100","Type":"ContainerDied","Data":"b2d744ec534f8c1acbe7610a2ab1394f371eb3a3c6d6fbfa4a2c5ee0768bfa0e"} Sep 08 15:27:42.682833 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.682816 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6" event={"ID":"6e7e3a85-e48c-438d-9f30-59412276e100","Type":"ContainerDied","Data":"2535791ce9fe0f249eede9a0c58b7549eb3314c16eba32ea2945748b6d31866c"} Sep 08 15:27:42.682950 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.682848 2572 scope.go:117] "RemoveContainer" containerID="b2d744ec534f8c1acbe7610a2ab1394f371eb3a3c6d6fbfa4a2c5ee0768bfa0e" Sep 08 15:27:42.692404 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.692384 2572 scope.go:117] "RemoveContainer" containerID="b2d744ec534f8c1acbe7610a2ab1394f371eb3a3c6d6fbfa4a2c5ee0768bfa0e" Sep 08 15:27:42.692719 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:27:42.692694 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d744ec534f8c1acbe7610a2ab1394f371eb3a3c6d6fbfa4a2c5ee0768bfa0e\": container with ID starting with b2d744ec534f8c1acbe7610a2ab1394f371eb3a3c6d6fbfa4a2c5ee0768bfa0e not found: ID does not exist" containerID="b2d744ec534f8c1acbe7610a2ab1394f371eb3a3c6d6fbfa4a2c5ee0768bfa0e" Sep 08 15:27:42.692792 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.692729 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d744ec534f8c1acbe7610a2ab1394f371eb3a3c6d6fbfa4a2c5ee0768bfa0e"} err="failed to get container status \"b2d744ec534f8c1acbe7610a2ab1394f371eb3a3c6d6fbfa4a2c5ee0768bfa0e\": rpc error: code = NotFound desc = could not find container \"b2d744ec534f8c1acbe7610a2ab1394f371eb3a3c6d6fbfa4a2c5ee0768bfa0e\": container with ID starting with b2d744ec534f8c1acbe7610a2ab1394f371eb3a3c6d6fbfa4a2c5ee0768bfa0e not found: ID does not exist" Sep 08 15:27:42.702889 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.702863 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6"] Sep 08 15:27:42.708417 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:42.708396 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3bc9dc-576697d878-lp7p6"] Sep 08 15:27:44.371468 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:44.371434 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e7e3a85-e48c-438d-9f30-59412276e100" path="/var/lib/kubelet/pods/6e7e3a85-e48c-438d-9f30-59412276e100/volumes" Sep 08 15:27:58.766005 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:58.765971 2572 generic.go:356] "Generic (PLEG): container finished" podID="6c9c4124-e078-45a9-a671-16fb1a1c7ea0" containerID="b7be2fa31005692ee82be5d419f7d427ccafdc262ee6fe597b8c6ef8c80d9d8a" exitCode=0 Sep 08 15:27:58.766426 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:58.766044 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" event={"ID":"6c9c4124-e078-45a9-a671-16fb1a1c7ea0","Type":"ContainerDied","Data":"b7be2fa31005692ee82be5d419f7d427ccafdc262ee6fe597b8c6ef8c80d9d8a"} Sep 08 15:27:58.766426 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:58.766405 2572 scope.go:117] "RemoveContainer" containerID="b7be2fa31005692ee82be5d419f7d427ccafdc262ee6fe597b8c6ef8c80d9d8a" Sep 08 15:27:58.767348 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:58.767317 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:27:58.767401 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:58.767388 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:27:58.767436 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:58.767421 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:27:59.775025 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:59.774989 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" event={"ID":"6c9c4124-e078-45a9-a671-16fb1a1c7ea0","Type":"ContainerStarted","Data":"cd4b76cc870915a50eec47260c86b025def69b6a0967fb70041f3990bd08e799"} Sep 08 15:27:59.913106 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:59.913076 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:27:59.913262 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:59.913217 2572 patch_prober.go:28] interesting pod/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Sep 08 15:27:59.913262 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:59.913254 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" podUID="6c9c4124-e078-45a9-a671-16fb1a1c7ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Sep 08 15:27:59.928194 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:59.928167 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp"] Sep 08 15:27:59.928549 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:59.928521 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e7e3a85-e48c-438d-9f30-59412276e100" containerName="payload-pre-processing" Sep 08 15:27:59.928549 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:59.928546 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e7e3a85-e48c-438d-9f30-59412276e100" containerName="payload-pre-processing" Sep 08 15:27:59.928647 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:59.928603 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e7e3a85-e48c-438d-9f30-59412276e100" containerName="payload-pre-processing" Sep 08 15:27:59.931577 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:59.931559 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp" Sep 08 15:27:59.934200 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:59.934175 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-6de9d1\"" Sep 08 15:27:59.934431 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:59.934410 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-6de9d1-dockercfg-nplz5\"" Sep 08 15:27:59.943513 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:59.942699 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp"] Sep 08 15:27:59.943874 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:59.943850 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a973172-a051-4479-8089-8aaf9b5d8247-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp\" (UID: \"4a973172-a051-4479-8089-8aaf9b5d8247\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp" Sep 08 15:27:59.944124 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:27:59.944104 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb975\" (UniqueName: \"kubernetes.io/projected/4a973172-a051-4479-8089-8aaf9b5d8247-kube-api-access-vb975\") pod \"payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp\" (UID: \"4a973172-a051-4479-8089-8aaf9b5d8247\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp" Sep 08 15:28:00.044931 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:00.044837 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb975\" (UniqueName: \"kubernetes.io/projected/4a973172-a051-4479-8089-8aaf9b5d8247-kube-api-access-vb975\") pod \"payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp\" (UID: \"4a973172-a051-4479-8089-8aaf9b5d8247\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp" Sep 08 15:28:00.045645 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:00.045612 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a973172-a051-4479-8089-8aaf9b5d8247-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp\" (UID: \"4a973172-a051-4479-8089-8aaf9b5d8247\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp" Sep 08 15:28:00.046810 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:00.046708 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a973172-a051-4479-8089-8aaf9b5d8247-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp\" (UID: \"4a973172-a051-4479-8089-8aaf9b5d8247\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp" Sep 08 15:28:00.074285 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:00.074255 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb975\" (UniqueName: \"kubernetes.io/projected/4a973172-a051-4479-8089-8aaf9b5d8247-kube-api-access-vb975\") pod \"payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp\" (UID: \"4a973172-a051-4479-8089-8aaf9b5d8247\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp" Sep 08 15:28:00.266713 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:00.266675 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp" Sep 08 15:28:00.537013 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:00.536749 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp"] Sep 08 15:28:00.557649 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:28:00.557620 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a973172_a051_4479_8089_8aaf9b5d8247.slice/crio-300a4557157b2b1742d8fec2bf8013e40337a24bef7213677342ce7f217a35bf WatchSource:0}: Error finding container 300a4557157b2b1742d8fec2bf8013e40337a24bef7213677342ce7f217a35bf: Status 404 returned error can't find the container with id 300a4557157b2b1742d8fec2bf8013e40337a24bef7213677342ce7f217a35bf Sep 08 15:28:00.781658 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:00.781623 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp" event={"ID":"4a973172-a051-4479-8089-8aaf9b5d8247","Type":"ContainerStarted","Data":"5ed1257e2e92c6f9a5f1bd1d7bb74e0edf0082b5cd926cada2ebc8bf9eca86e5"} Sep 08 15:28:00.782086 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:00.781679 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp" event={"ID":"4a973172-a051-4479-8089-8aaf9b5d8247","Type":"ContainerStarted","Data":"300a4557157b2b1742d8fec2bf8013e40337a24bef7213677342ce7f217a35bf"} Sep 08 15:28:00.798828 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:00.798788 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp" podStartSLOduration=1.7987739409999999 podStartE2EDuration="1.798773941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:28:00.797891066 +0000 UTC m=+1142.950069646" watchObservedRunningTime="2026-09-08 15:28:00.798773941 +0000 UTC m=+1142.950952520" Sep 08 15:28:00.913523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:00.913422 2572 patch_prober.go:28] interesting pod/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Sep 08 15:28:00.913767 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:00.913740 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" podUID="6c9c4124-e078-45a9-a671-16fb1a1c7ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Sep 08 15:28:01.786886 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:01.786856 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp" Sep 08 15:28:01.913264 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:01.913235 2572 patch_prober.go:28] interesting pod/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Sep 08 15:28:01.913431 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:01.913282 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" podUID="6c9c4124-e078-45a9-a671-16fb1a1c7ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Sep 08 15:28:02.913339 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:02.913306 2572 patch_prober.go:28] interesting pod/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Sep 08 15:28:02.913732 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:02.913357 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" podUID="6c9c4124-e078-45a9-a671-16fb1a1c7ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Sep 08 15:28:03.913095 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:03.913057 2572 patch_prober.go:28] interesting pod/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Sep 08 15:28:03.913257 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:03.913117 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" podUID="6c9c4124-e078-45a9-a671-16fb1a1c7ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Sep 08 15:28:04.833866 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:04.833824 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e22317f-wt52m"] Sep 08 15:28:04.837238 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:04.837216 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e22317f-wt52m" Sep 08 15:28:04.839647 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:04.839625 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8qqvg\"" Sep 08 15:28:04.844655 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:04.844634 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e22317f-wt52m"] Sep 08 15:28:04.890134 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:04.890090 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/188c57be-d5b0-4d78-a886-975f197d266c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e22317f-wt52m\" (UID: \"188c57be-d5b0-4d78-a886-975f197d266c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e22317f-wt52m" Sep 08 15:28:04.990840 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:04.990807 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/188c57be-d5b0-4d78-a886-975f197d266c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e22317f-wt52m\" (UID: \"188c57be-d5b0-4d78-a886-975f197d266c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e22317f-wt52m" Sep 08 15:28:04.991445 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:04.991421 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/188c57be-d5b0-4d78-a886-975f197d266c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e22317f-wt52m\" (UID: \"188c57be-d5b0-4d78-a886-975f197d266c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e22317f-wt52m" Sep 08 15:28:05.047214 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:05.047178 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:28:05.047475 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:05.047458 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:28:05.048922 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:05.048898 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:28:05.149185 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:05.149119 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e22317f-wt52m" Sep 08 15:28:05.420894 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:05.420834 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e22317f-wt52m"] Sep 08 15:28:05.424314 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:28:05.423759 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188c57be_d5b0_4d78_a886_975f197d266c.slice/crio-b1aba50afcc4a9b8030cd9b74f31a6429a184be36f54ff970094fe9681926d9b WatchSource:0}: Error finding container b1aba50afcc4a9b8030cd9b74f31a6429a184be36f54ff970094fe9681926d9b: Status 404 returned error can't find the container with id b1aba50afcc4a9b8030cd9b74f31a6429a184be36f54ff970094fe9681926d9b Sep 08 15:28:05.808339 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:05.808308 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e22317f-wt52m" event={"ID":"188c57be-d5b0-4d78-a886-975f197d266c","Type":"ContainerStarted","Data":"4a9f79e55b4cdf8a3ed1031bfe848a1767693061173451f72fbe2714a7fe9b82"} Sep 08 15:28:05.808437 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:05.808362 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e22317f-wt52m" event={"ID":"188c57be-d5b0-4d78-a886-975f197d266c","Type":"ContainerStarted","Data":"b1aba50afcc4a9b8030cd9b74f31a6429a184be36f54ff970094fe9681926d9b"} Sep 08 15:28:05.824604 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:05.824562 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e22317f-wt52m" podStartSLOduration=1.824546682 podStartE2EDuration="1.824546682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:28:05.822371714 +0000 UTC m=+1147.974550304" watchObservedRunningTime="2026-09-08 15:28:05.824546682 +0000 UTC m=+1147.976725261" Sep 08 15:28:06.819740 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:06.819703 2572 generic.go:356] "Generic (PLEG): container finished" podID="188c57be-d5b0-4d78-a886-975f197d266c" containerID="4a9f79e55b4cdf8a3ed1031bfe848a1767693061173451f72fbe2714a7fe9b82" exitCode=0 Sep 08 15:28:06.820102 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:06.819794 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e22317f-wt52m" event={"ID":"188c57be-d5b0-4d78-a886-975f197d266c","Type":"ContainerDied","Data":"4a9f79e55b4cdf8a3ed1031bfe848a1767693061173451f72fbe2714a7fe9b82"} Sep 08 15:28:07.955310 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:07.955286 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e22317f-wt52m" Sep 08 15:28:08.016112 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:08.016083 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/188c57be-d5b0-4d78-a886-975f197d266c-maas-api-service-ca\") pod \"188c57be-d5b0-4d78-a886-975f197d266c\" (UID: \"188c57be-d5b0-4d78-a886-975f197d266c\") " Sep 08 15:28:08.016401 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:08.016379 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188c57be-d5b0-4d78-a886-975f197d266c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "188c57be-d5b0-4d78-a886-975f197d266c" (UID: "188c57be-d5b0-4d78-a886-975f197d266c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:28:08.116980 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:08.116921 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/188c57be-d5b0-4d78-a886-975f197d266c-maas-api-service-ca\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:28:08.829359 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:08.829333 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e22317f-wt52m" Sep 08 15:28:08.829521 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:08.829365 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e22317f-wt52m" event={"ID":"188c57be-d5b0-4d78-a886-975f197d266c","Type":"ContainerDied","Data":"b1aba50afcc4a9b8030cd9b74f31a6429a184be36f54ff970094fe9681926d9b"} Sep 08 15:28:08.829521 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:08.829407 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1aba50afcc4a9b8030cd9b74f31a6429a184be36f54ff970094fe9681926d9b" Sep 08 15:28:10.885826 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:10.885793 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e22317f-wt52m"] Sep 08 15:28:10.890564 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:10.890542 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e22317f-wt52m"] Sep 08 15:28:12.081972 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:12.081944 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb"] Sep 08 15:28:12.082417 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:12.082116 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" podUID="6c9c4124-e078-45a9-a671-16fb1a1c7ea0" containerName="istio-proxy" containerID="cri-o://cd4b76cc870915a50eec47260c86b025def69b6a0967fb70041f3990bd08e799" gracePeriod=30 Sep 08 15:28:12.371071 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:12.370995 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="188c57be-d5b0-4d78-a886-975f197d266c" path="/var/lib/kubelet/pods/188c57be-d5b0-4d78-a886-975f197d266c/volumes" Sep 08 15:28:12.796654 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:12.796625 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp" Sep 08 15:28:17.330248 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.330227 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:28:17.400593 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.400517 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-credential-socket\") pod \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " Sep 08 15:28:17.400593 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.400570 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-podinfo\") pod \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " Sep 08 15:28:17.400593 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.400597 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-envoy\") pod \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " Sep 08 15:28:17.400806 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.400616 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-token\") pod \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " Sep 08 15:28:17.400806 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.400668 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istiod-ca-cert\") pod \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " Sep 08 15:28:17.400806 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.400715 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-workload-certs\") pod \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " Sep 08 15:28:17.400806 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.400743 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-data\") pod \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " Sep 08 15:28:17.400806 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.400772 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j4c8\" (UniqueName: \"kubernetes.io/projected/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-kube-api-access-7j4c8\") pod \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " Sep 08 15:28:17.400806 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.400797 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-workload-socket\") pod \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\" (UID: \"6c9c4124-e078-45a9-a671-16fb1a1c7ea0\") " Sep 08 15:28:17.401097 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.400791 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6c9c4124-e078-45a9-a671-16fb1a1c7ea0" (UID: "6c9c4124-e078-45a9-a671-16fb1a1c7ea0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:28:17.401176 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.400971 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-data" (OuterVolumeSpecName: "istio-data") pod "6c9c4124-e078-45a9-a671-16fb1a1c7ea0" (UID: "6c9c4124-e078-45a9-a671-16fb1a1c7ea0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:28:17.401176 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.401161 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6c9c4124-e078-45a9-a671-16fb1a1c7ea0" (UID: "6c9c4124-e078-45a9-a671-16fb1a1c7ea0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:28:17.401176 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.401153 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6c9c4124-e078-45a9-a671-16fb1a1c7ea0" (UID: "6c9c4124-e078-45a9-a671-16fb1a1c7ea0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:28:17.401570 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.401523 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6c9c4124-e078-45a9-a671-16fb1a1c7ea0" (UID: "6c9c4124-e078-45a9-a671-16fb1a1c7ea0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:28:17.403001 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.402968 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6c9c4124-e078-45a9-a671-16fb1a1c7ea0" (UID: "6c9c4124-e078-45a9-a671-16fb1a1c7ea0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 15:28:17.403096 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.403032 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6c9c4124-e078-45a9-a671-16fb1a1c7ea0" (UID: "6c9c4124-e078-45a9-a671-16fb1a1c7ea0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:28:17.403096 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.403043 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-kube-api-access-7j4c8" (OuterVolumeSpecName: "kube-api-access-7j4c8") pod "6c9c4124-e078-45a9-a671-16fb1a1c7ea0" (UID: "6c9c4124-e078-45a9-a671-16fb1a1c7ea0"). InnerVolumeSpecName "kube-api-access-7j4c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:28:17.403096 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.403051 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-token" (OuterVolumeSpecName: "istio-token") pod "6c9c4124-e078-45a9-a671-16fb1a1c7ea0" (UID: "6c9c4124-e078-45a9-a671-16fb1a1c7ea0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:28:17.502187 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.502159 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-workload-socket\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:28:17.502187 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.502186 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-credential-socket\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:28:17.502344 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.502200 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-podinfo\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:28:17.502344 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.502213 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-envoy\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:28:17.502344 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.502225 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-token\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:28:17.502344 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.502236 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istiod-ca-cert\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:28:17.502344 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.502246 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-workload-certs\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:28:17.502344 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.502253 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-istio-data\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:28:17.502344 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.502261 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j4c8\" (UniqueName: \"kubernetes.io/projected/6c9c4124-e078-45a9-a671-16fb1a1c7ea0-kube-api-access-7j4c8\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:28:17.729225 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.729155 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz"] Sep 08 15:28:17.729476 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.729464 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c9c4124-e078-45a9-a671-16fb1a1c7ea0" containerName="istio-proxy" Sep 08 15:28:17.729540 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.729478 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c9c4124-e078-45a9-a671-16fb1a1c7ea0" containerName="istio-proxy" Sep 08 15:28:17.729540 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.729486 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c9c4124-e078-45a9-a671-16fb1a1c7ea0" containerName="istio-proxy" Sep 08 15:28:17.729540 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.729506 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c9c4124-e078-45a9-a671-16fb1a1c7ea0" containerName="istio-proxy" Sep 08 15:28:17.729540 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.729520 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="188c57be-d5b0-4d78-a886-975f197d266c" containerName="revoke-keys" Sep 08 15:28:17.729540 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.729526 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="188c57be-d5b0-4d78-a886-975f197d266c" containerName="revoke-keys" Sep 08 15:28:17.729680 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.729583 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c9c4124-e078-45a9-a671-16fb1a1c7ea0" containerName="istio-proxy" Sep 08 15:28:17.729680 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.729590 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c9c4124-e078-45a9-a671-16fb1a1c7ea0" containerName="istio-proxy" Sep 08 15:28:17.729680 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.729596 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="188c57be-d5b0-4d78-a886-975f197d266c" containerName="revoke-keys" Sep 08 15:28:17.732699 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.732685 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz" Sep 08 15:28:17.734967 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.734950 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-152e35a7-dockercfg-94gz5\"" Sep 08 15:28:17.735508 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.735469 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-152e35a7\"" Sep 08 15:28:17.742202 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.742175 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz"] Sep 08 15:28:17.804958 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.804928 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlwn4\" (UniqueName: \"kubernetes.io/projected/9311eb9a-25b6-447f-bb84-39f6eec12b98-kube-api-access-wlwn4\") pod \"payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz\" (UID: \"9311eb9a-25b6-447f-bb84-39f6eec12b98\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz" Sep 08 15:28:17.805117 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.804999 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9311eb9a-25b6-447f-bb84-39f6eec12b98-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz\" (UID: \"9311eb9a-25b6-447f-bb84-39f6eec12b98\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz" Sep 08 15:28:17.878663 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.878624 2572 generic.go:356] "Generic (PLEG): container finished" podID="6c9c4124-e078-45a9-a671-16fb1a1c7ea0" containerID="cd4b76cc870915a50eec47260c86b025def69b6a0967fb70041f3990bd08e799" exitCode=0 Sep 08 15:28:17.878812 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.878669 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" event={"ID":"6c9c4124-e078-45a9-a671-16fb1a1c7ea0","Type":"ContainerDied","Data":"cd4b76cc870915a50eec47260c86b025def69b6a0967fb70041f3990bd08e799"} Sep 08 15:28:17.878812 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.878690 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" Sep 08 15:28:17.878812 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.878756 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb" event={"ID":"6c9c4124-e078-45a9-a671-16fb1a1c7ea0","Type":"ContainerDied","Data":"4ad058da7ab002d6a9fb207bf7c378c4e2aad0e00a16abca8209bc50c15c48fb"} Sep 08 15:28:17.878812 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.878782 2572 scope.go:117] "RemoveContainer" containerID="cd4b76cc870915a50eec47260c86b025def69b6a0967fb70041f3990bd08e799" Sep 08 15:28:17.888089 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.888071 2572 scope.go:117] "RemoveContainer" containerID="b7be2fa31005692ee82be5d419f7d427ccafdc262ee6fe597b8c6ef8c80d9d8a" Sep 08 15:28:17.895843 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.895829 2572 scope.go:117] "RemoveContainer" containerID="cd4b76cc870915a50eec47260c86b025def69b6a0967fb70041f3990bd08e799" Sep 08 15:28:17.896082 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:28:17.896066 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4b76cc870915a50eec47260c86b025def69b6a0967fb70041f3990bd08e799\": container with ID starting with cd4b76cc870915a50eec47260c86b025def69b6a0967fb70041f3990bd08e799 not found: ID does not exist" containerID="cd4b76cc870915a50eec47260c86b025def69b6a0967fb70041f3990bd08e799" Sep 08 15:28:17.896132 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.896088 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4b76cc870915a50eec47260c86b025def69b6a0967fb70041f3990bd08e799"} err="failed to get container status \"cd4b76cc870915a50eec47260c86b025def69b6a0967fb70041f3990bd08e799\": rpc error: code = NotFound desc = could not find container \"cd4b76cc870915a50eec47260c86b025def69b6a0967fb70041f3990bd08e799\": container with ID starting with cd4b76cc870915a50eec47260c86b025def69b6a0967fb70041f3990bd08e799 not found: ID does not exist" Sep 08 15:28:17.896132 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.896102 2572 scope.go:117] "RemoveContainer" containerID="b7be2fa31005692ee82be5d419f7d427ccafdc262ee6fe597b8c6ef8c80d9d8a" Sep 08 15:28:17.896351 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:28:17.896339 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7be2fa31005692ee82be5d419f7d427ccafdc262ee6fe597b8c6ef8c80d9d8a\": container with ID starting with b7be2fa31005692ee82be5d419f7d427ccafdc262ee6fe597b8c6ef8c80d9d8a not found: ID does not exist" containerID="b7be2fa31005692ee82be5d419f7d427ccafdc262ee6fe597b8c6ef8c80d9d8a" Sep 08 15:28:17.896391 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.896353 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7be2fa31005692ee82be5d419f7d427ccafdc262ee6fe597b8c6ef8c80d9d8a"} err="failed to get container status \"b7be2fa31005692ee82be5d419f7d427ccafdc262ee6fe597b8c6ef8c80d9d8a\": rpc error: code = NotFound desc = could not find container \"b7be2fa31005692ee82be5d419f7d427ccafdc262ee6fe597b8c6ef8c80d9d8a\": container with ID starting with b7be2fa31005692ee82be5d419f7d427ccafdc262ee6fe597b8c6ef8c80d9d8a not found: ID does not exist" Sep 08 15:28:17.903087 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.903063 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb"] Sep 08 15:28:17.905415 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.905394 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlwn4\" (UniqueName: \"kubernetes.io/projected/9311eb9a-25b6-447f-bb84-39f6eec12b98-kube-api-access-wlwn4\") pod \"payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz\" (UID: \"9311eb9a-25b6-447f-bb84-39f6eec12b98\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz" Sep 08 15:28:17.905520 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.905453 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9311eb9a-25b6-447f-bb84-39f6eec12b98-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz\" (UID: \"9311eb9a-25b6-447f-bb84-39f6eec12b98\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz" Sep 08 15:28:17.906085 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.906068 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9311eb9a-25b6-447f-bb84-39f6eec12b98-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz\" (UID: \"9311eb9a-25b6-447f-bb84-39f6eec12b98\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz" Sep 08 15:28:17.911140 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.911118 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9e22317f-openshift-default-785bdf679d-wh7mb"] Sep 08 15:28:17.915355 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:17.915337 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlwn4\" (UniqueName: \"kubernetes.io/projected/9311eb9a-25b6-447f-bb84-39f6eec12b98-kube-api-access-wlwn4\") pod \"payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz\" (UID: \"9311eb9a-25b6-447f-bb84-39f6eec12b98\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz" Sep 08 15:28:18.043053 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:18.043029 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz" Sep 08 15:28:18.314171 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:18.314099 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz"] Sep 08 15:28:18.318130 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:28:18.318105 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9311eb9a_25b6_447f_bb84_39f6eec12b98.slice/crio-e118161ea35a93cf74201bf2f6ff80e805b406663a4657ae220182c93df0e2c8 WatchSource:0}: Error finding container e118161ea35a93cf74201bf2f6ff80e805b406663a4657ae220182c93df0e2c8: Status 404 returned error can't find the container with id e118161ea35a93cf74201bf2f6ff80e805b406663a4657ae220182c93df0e2c8 Sep 08 15:28:18.372578 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:18.372556 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c9c4124-e078-45a9-a671-16fb1a1c7ea0" path="/var/lib/kubelet/pods/6c9c4124-e078-45a9-a671-16fb1a1c7ea0/volumes" Sep 08 15:28:18.885637 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:18.885601 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz" event={"ID":"9311eb9a-25b6-447f-bb84-39f6eec12b98","Type":"ContainerStarted","Data":"27311fcc7ea64899e0ff0aee1370c6e686f1900b3f67748759881b0e8d05573f"} Sep 08 15:28:18.885781 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:18.885658 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz" event={"ID":"9311eb9a-25b6-447f-bb84-39f6eec12b98","Type":"ContainerStarted","Data":"e118161ea35a93cf74201bf2f6ff80e805b406663a4657ae220182c93df0e2c8"} Sep 08 15:28:18.885781 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:18.885719 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz" Sep 08 15:28:18.909654 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:18.909606 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz" podStartSLOduration=1.909595656 podStartE2EDuration="1.909595656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:28:18.909076935 +0000 UTC m=+1161.061255513" watchObservedRunningTime="2026-09-08 15:28:18.909595656 +0000 UTC m=+1161.061774234" Sep 08 15:28:29.892177 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:29.892108 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz" Sep 08 15:28:40.617959 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:40.617931 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-152e35a7-qxqf4"] Sep 08 15:28:40.621319 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:40.621302 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-152e35a7-qxqf4" Sep 08 15:28:40.623292 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:40.623272 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8qqvg\"" Sep 08 15:28:40.628353 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:40.628304 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-152e35a7-qxqf4"] Sep 08 15:28:40.683039 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:40.683011 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63389106-4c3b-40e4-93c5-a7013193f42b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-152e35a7-qxqf4\" (UID: \"63389106-4c3b-40e4-93c5-a7013193f42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-152e35a7-qxqf4" Sep 08 15:28:40.784168 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:40.784146 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63389106-4c3b-40e4-93c5-a7013193f42b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-152e35a7-qxqf4\" (UID: \"63389106-4c3b-40e4-93c5-a7013193f42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-152e35a7-qxqf4" Sep 08 15:28:40.784704 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:40.784687 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63389106-4c3b-40e4-93c5-a7013193f42b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-152e35a7-qxqf4\" (UID: \"63389106-4c3b-40e4-93c5-a7013193f42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-152e35a7-qxqf4" Sep 08 15:28:40.932596 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:40.932540 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-152e35a7-qxqf4" Sep 08 15:28:41.200933 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:41.200818 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-152e35a7-qxqf4"] Sep 08 15:28:41.203126 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:28:41.203094 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63389106_4c3b_40e4_93c5_a7013193f42b.slice/crio-a9ec5d964c82617164264638c0d91bbfc5036106d7068fe5891ffb11e6499bfb WatchSource:0}: Error finding container a9ec5d964c82617164264638c0d91bbfc5036106d7068fe5891ffb11e6499bfb: Status 404 returned error can't find the container with id a9ec5d964c82617164264638c0d91bbfc5036106d7068fe5891ffb11e6499bfb Sep 08 15:28:41.992312 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:41.992287 2572 generic.go:356] "Generic (PLEG): container finished" podID="63389106-4c3b-40e4-93c5-a7013193f42b" containerID="ec5278888f8ae16a482c37fd7f4237a651b9baa2745dda2cd7c9c9d3ab890561" exitCode=0 Sep 08 15:28:41.992622 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:41.992367 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-152e35a7-qxqf4" event={"ID":"63389106-4c3b-40e4-93c5-a7013193f42b","Type":"ContainerDied","Data":"ec5278888f8ae16a482c37fd7f4237a651b9baa2745dda2cd7c9c9d3ab890561"} Sep 08 15:28:41.992622 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:41.992410 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-152e35a7-qxqf4" event={"ID":"63389106-4c3b-40e4-93c5-a7013193f42b","Type":"ContainerStarted","Data":"a9ec5d964c82617164264638c0d91bbfc5036106d7068fe5891ffb11e6499bfb"} Sep 08 15:28:43.125583 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:43.125563 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-152e35a7-qxqf4" Sep 08 15:28:43.202478 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:43.202452 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63389106-4c3b-40e4-93c5-a7013193f42b-maas-api-service-ca\") pod \"63389106-4c3b-40e4-93c5-a7013193f42b\" (UID: \"63389106-4c3b-40e4-93c5-a7013193f42b\") " Sep 08 15:28:43.202770 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:43.202749 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63389106-4c3b-40e4-93c5-a7013193f42b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "63389106-4c3b-40e4-93c5-a7013193f42b" (UID: "63389106-4c3b-40e4-93c5-a7013193f42b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:28:43.303415 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:43.303368 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63389106-4c3b-40e4-93c5-a7013193f42b-maas-api-service-ca\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:28:44.002902 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:44.002875 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-152e35a7-qxqf4" Sep 08 15:28:44.003040 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:44.002882 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-152e35a7-qxqf4" event={"ID":"63389106-4c3b-40e4-93c5-a7013193f42b","Type":"ContainerDied","Data":"a9ec5d964c82617164264638c0d91bbfc5036106d7068fe5891ffb11e6499bfb"} Sep 08 15:28:44.003040 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:44.003027 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9ec5d964c82617164264638c0d91bbfc5036106d7068fe5891ffb11e6499bfb" Sep 08 15:28:47.696482 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:47.696449 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz"] Sep 08 15:28:47.696889 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:47.696735 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz" podUID="9311eb9a-25b6-447f-bb84-39f6eec12b98" containerName="payload-pre-processing" containerID="cri-o://27311fcc7ea64899e0ff0aee1370c6e686f1900b3f67748759881b0e8d05573f" gracePeriod=30 Sep 08 15:28:47.845051 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:47.845013 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-152e35a7-qxqf4"] Sep 08 15:28:47.847187 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:47.847160 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-152e35a7-qxqf4"] Sep 08 15:28:47.937215 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:47.937192 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz" Sep 08 15:28:48.022882 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:48.022848 2572 generic.go:356] "Generic (PLEG): container finished" podID="9311eb9a-25b6-447f-bb84-39f6eec12b98" containerID="27311fcc7ea64899e0ff0aee1370c6e686f1900b3f67748759881b0e8d05573f" exitCode=0 Sep 08 15:28:48.023045 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:48.022897 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz" Sep 08 15:28:48.023045 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:48.022893 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz" event={"ID":"9311eb9a-25b6-447f-bb84-39f6eec12b98","Type":"ContainerDied","Data":"27311fcc7ea64899e0ff0aee1370c6e686f1900b3f67748759881b0e8d05573f"} Sep 08 15:28:48.023045 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:48.022962 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz" event={"ID":"9311eb9a-25b6-447f-bb84-39f6eec12b98","Type":"ContainerDied","Data":"e118161ea35a93cf74201bf2f6ff80e805b406663a4657ae220182c93df0e2c8"} Sep 08 15:28:48.023045 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:48.022996 2572 scope.go:117] "RemoveContainer" containerID="27311fcc7ea64899e0ff0aee1370c6e686f1900b3f67748759881b0e8d05573f" Sep 08 15:28:48.031128 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:48.031109 2572 scope.go:117] "RemoveContainer" containerID="27311fcc7ea64899e0ff0aee1370c6e686f1900b3f67748759881b0e8d05573f" Sep 08 15:28:48.031407 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:28:48.031388 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27311fcc7ea64899e0ff0aee1370c6e686f1900b3f67748759881b0e8d05573f\": container with ID starting with 27311fcc7ea64899e0ff0aee1370c6e686f1900b3f67748759881b0e8d05573f not found: ID does not exist" containerID="27311fcc7ea64899e0ff0aee1370c6e686f1900b3f67748759881b0e8d05573f" Sep 08 15:28:48.031456 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:48.031415 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27311fcc7ea64899e0ff0aee1370c6e686f1900b3f67748759881b0e8d05573f"} err="failed to get container status \"27311fcc7ea64899e0ff0aee1370c6e686f1900b3f67748759881b0e8d05573f\": rpc error: code = NotFound desc = could not find container \"27311fcc7ea64899e0ff0aee1370c6e686f1900b3f67748759881b0e8d05573f\": container with ID starting with 27311fcc7ea64899e0ff0aee1370c6e686f1900b3f67748759881b0e8d05573f not found: ID does not exist" Sep 08 15:28:48.038842 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:48.038826 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9311eb9a-25b6-447f-bb84-39f6eec12b98-plugins-config-volume\") pod \"9311eb9a-25b6-447f-bb84-39f6eec12b98\" (UID: \"9311eb9a-25b6-447f-bb84-39f6eec12b98\") " Sep 08 15:28:48.038944 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:48.038934 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlwn4\" (UniqueName: \"kubernetes.io/projected/9311eb9a-25b6-447f-bb84-39f6eec12b98-kube-api-access-wlwn4\") pod \"9311eb9a-25b6-447f-bb84-39f6eec12b98\" (UID: \"9311eb9a-25b6-447f-bb84-39f6eec12b98\") " Sep 08 15:28:48.039201 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:48.039178 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9311eb9a-25b6-447f-bb84-39f6eec12b98-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9311eb9a-25b6-447f-bb84-39f6eec12b98" (UID: "9311eb9a-25b6-447f-bb84-39f6eec12b98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:28:48.040891 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:48.040875 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9311eb9a-25b6-447f-bb84-39f6eec12b98-kube-api-access-wlwn4" (OuterVolumeSpecName: "kube-api-access-wlwn4") pod "9311eb9a-25b6-447f-bb84-39f6eec12b98" (UID: "9311eb9a-25b6-447f-bb84-39f6eec12b98"). InnerVolumeSpecName "kube-api-access-wlwn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:28:48.140345 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:48.140309 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlwn4\" (UniqueName: \"kubernetes.io/projected/9311eb9a-25b6-447f-bb84-39f6eec12b98-kube-api-access-wlwn4\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:28:48.140345 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:48.140338 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9311eb9a-25b6-447f-bb84-39f6eec12b98-plugins-config-volume\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:28:48.344022 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:48.343998 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz"] Sep 08 15:28:48.348246 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:48.348219 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-152e35a7-79bbdc6d4b-sjgkz"] Sep 08 15:28:48.375829 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:48.375802 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63389106-4c3b-40e4-93c5-a7013193f42b" path="/var/lib/kubelet/pods/63389106-4c3b-40e4-93c5-a7013193f42b/volumes" Sep 08 15:28:48.376125 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:28:48.376112 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9311eb9a-25b6-447f-bb84-39f6eec12b98" path="/var/lib/kubelet/pods/9311eb9a-25b6-447f-bb84-39f6eec12b98/volumes" Sep 08 15:29:20.396616 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:29:20.396586 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp"] Sep 08 15:29:20.397101 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:29:20.396782 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp" podUID="4a973172-a051-4479-8089-8aaf9b5d8247" containerName="payload-pre-processing" containerID="cri-o://5ed1257e2e92c6f9a5f1bd1d7bb74e0edf0082b5cd926cada2ebc8bf9eca86e5" gracePeriod=30 Sep 08 15:29:20.649179 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:29:20.649124 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp" Sep 08 15:29:20.687576 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:29:20.687542 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb975\" (UniqueName: \"kubernetes.io/projected/4a973172-a051-4479-8089-8aaf9b5d8247-kube-api-access-vb975\") pod \"4a973172-a051-4479-8089-8aaf9b5d8247\" (UID: \"4a973172-a051-4479-8089-8aaf9b5d8247\") " Sep 08 15:29:20.687685 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:29:20.687664 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a973172-a051-4479-8089-8aaf9b5d8247-plugins-config-volume\") pod \"4a973172-a051-4479-8089-8aaf9b5d8247\" (UID: \"4a973172-a051-4479-8089-8aaf9b5d8247\") " Sep 08 15:29:20.688034 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:29:20.688005 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a973172-a051-4479-8089-8aaf9b5d8247-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4a973172-a051-4479-8089-8aaf9b5d8247" (UID: "4a973172-a051-4479-8089-8aaf9b5d8247"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:29:20.689464 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:29:20.689436 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a973172-a051-4479-8089-8aaf9b5d8247-kube-api-access-vb975" (OuterVolumeSpecName: "kube-api-access-vb975") pod "4a973172-a051-4479-8089-8aaf9b5d8247" (UID: "4a973172-a051-4479-8089-8aaf9b5d8247"). InnerVolumeSpecName "kube-api-access-vb975". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:29:20.788323 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:29:20.788295 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb975\" (UniqueName: \"kubernetes.io/projected/4a973172-a051-4479-8089-8aaf9b5d8247-kube-api-access-vb975\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:29:20.788323 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:29:20.788320 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a973172-a051-4479-8089-8aaf9b5d8247-plugins-config-volume\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:29:21.179502 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:29:21.179455 2572 generic.go:356] "Generic (PLEG): container finished" podID="4a973172-a051-4479-8089-8aaf9b5d8247" containerID="5ed1257e2e92c6f9a5f1bd1d7bb74e0edf0082b5cd926cada2ebc8bf9eca86e5" exitCode=0 Sep 08 15:29:21.179645 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:29:21.179522 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp" Sep 08 15:29:21.179645 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:29:21.179528 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp" event={"ID":"4a973172-a051-4479-8089-8aaf9b5d8247","Type":"ContainerDied","Data":"5ed1257e2e92c6f9a5f1bd1d7bb74e0edf0082b5cd926cada2ebc8bf9eca86e5"} Sep 08 15:29:21.179645 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:29:21.179585 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp" event={"ID":"4a973172-a051-4479-8089-8aaf9b5d8247","Type":"ContainerDied","Data":"300a4557157b2b1742d8fec2bf8013e40337a24bef7213677342ce7f217a35bf"} Sep 08 15:29:21.179645 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:29:21.179608 2572 scope.go:117] "RemoveContainer" containerID="5ed1257e2e92c6f9a5f1bd1d7bb74e0edf0082b5cd926cada2ebc8bf9eca86e5" Sep 08 15:29:21.187889 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:29:21.187874 2572 scope.go:117] "RemoveContainer" containerID="5ed1257e2e92c6f9a5f1bd1d7bb74e0edf0082b5cd926cada2ebc8bf9eca86e5" Sep 08 15:29:21.188155 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:29:21.188137 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed1257e2e92c6f9a5f1bd1d7bb74e0edf0082b5cd926cada2ebc8bf9eca86e5\": container with ID starting with 5ed1257e2e92c6f9a5f1bd1d7bb74e0edf0082b5cd926cada2ebc8bf9eca86e5 not found: ID does not exist" containerID="5ed1257e2e92c6f9a5f1bd1d7bb74e0edf0082b5cd926cada2ebc8bf9eca86e5" Sep 08 15:29:21.188213 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:29:21.188164 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed1257e2e92c6f9a5f1bd1d7bb74e0edf0082b5cd926cada2ebc8bf9eca86e5"} err="failed to get container status \"5ed1257e2e92c6f9a5f1bd1d7bb74e0edf0082b5cd926cada2ebc8bf9eca86e5\": rpc error: code = NotFound desc = could not find container \"5ed1257e2e92c6f9a5f1bd1d7bb74e0edf0082b5cd926cada2ebc8bf9eca86e5\": container with ID starting with 5ed1257e2e92c6f9a5f1bd1d7bb74e0edf0082b5cd926cada2ebc8bf9eca86e5 not found: ID does not exist" Sep 08 15:29:21.200992 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:29:21.200972 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp"] Sep 08 15:29:21.208547 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:29:21.208529 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6de9d1-85b757894b-cnbgp"] Sep 08 15:29:22.370902 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:29:22.370865 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a973172-a051-4479-8089-8aaf9b5d8247" path="/var/lib/kubelet/pods/4a973172-a051-4479-8089-8aaf9b5d8247/volumes" Sep 08 15:30:00.229059 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:00.229025 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814690-dm49t"] Sep 08 15:30:00.229468 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:00.229334 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a973172-a051-4479-8089-8aaf9b5d8247" containerName="payload-pre-processing" Sep 08 15:30:00.229468 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:00.229345 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a973172-a051-4479-8089-8aaf9b5d8247" containerName="payload-pre-processing" Sep 08 15:30:00.229468 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:00.229357 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9311eb9a-25b6-447f-bb84-39f6eec12b98" containerName="payload-pre-processing" Sep 08 15:30:00.229468 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:00.229363 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9311eb9a-25b6-447f-bb84-39f6eec12b98" containerName="payload-pre-processing" Sep 08 15:30:00.229468 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:00.229372 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63389106-4c3b-40e4-93c5-a7013193f42b" containerName="revoke-keys" Sep 08 15:30:00.229468 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:00.229378 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="63389106-4c3b-40e4-93c5-a7013193f42b" containerName="revoke-keys" Sep 08 15:30:00.229468 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:00.229426 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a973172-a051-4479-8089-8aaf9b5d8247" containerName="payload-pre-processing" Sep 08 15:30:00.229468 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:00.229433 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="63389106-4c3b-40e4-93c5-a7013193f42b" containerName="revoke-keys" Sep 08 15:30:00.229468 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:00.229439 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9311eb9a-25b6-447f-bb84-39f6eec12b98" containerName="payload-pre-processing" Sep 08 15:30:00.233363 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:00.233348 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814690-dm49t" Sep 08 15:30:00.244634 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:00.244612 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8qqvg\"" Sep 08 15:30:00.252832 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:00.252795 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814690-dm49t" Sep 08 15:30:00.254091 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:00.254072 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814690-dm49t"] Sep 08 15:30:00.570529 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:00.570502 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814690-dm49t"] Sep 08 15:30:00.573418 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:30:00.573394 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90d1787_ea44_47bb_aedf_482ea76c029a.slice/crio-5840c5e8df06f08dc5a8837cf9b26b4de2eae93a266619c90645478c56db2c53 WatchSource:0}: Error finding container 5840c5e8df06f08dc5a8837cf9b26b4de2eae93a266619c90645478c56db2c53: Status 404 returned error can't find the container with id 5840c5e8df06f08dc5a8837cf9b26b4de2eae93a266619c90645478c56db2c53 Sep 08 15:30:01.360235 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:01.360210 2572 generic.go:356] "Generic (PLEG): container finished" podID="e90d1787-ea44-47bb-aedf-482ea76c029a" containerID="44274120b8067440dbdc44518114117288290c05d71eaf314fdbcc405cd027f1" exitCode=0 Sep 08 15:30:01.360587 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:01.360300 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814690-dm49t" event={"ID":"e90d1787-ea44-47bb-aedf-482ea76c029a","Type":"ContainerDied","Data":"44274120b8067440dbdc44518114117288290c05d71eaf314fdbcc405cd027f1"} Sep 08 15:30:01.360587 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:01.360351 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814690-dm49t" event={"ID":"e90d1787-ea44-47bb-aedf-482ea76c029a","Type":"ContainerStarted","Data":"5840c5e8df06f08dc5a8837cf9b26b4de2eae93a266619c90645478c56db2c53"} Sep 08 15:30:02.496598 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:02.496576 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814690-dm49t" Sep 08 15:30:03.369900 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:03.369878 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814690-dm49t" Sep 08 15:30:03.370041 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:03.369913 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814690-dm49t" event={"ID":"e90d1787-ea44-47bb-aedf-482ea76c029a","Type":"ContainerDied","Data":"5840c5e8df06f08dc5a8837cf9b26b4de2eae93a266619c90645478c56db2c53"} Sep 08 15:30:03.370041 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:03.369952 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5840c5e8df06f08dc5a8837cf9b26b4de2eae93a266619c90645478c56db2c53" Sep 08 15:30:18.129917 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:18.129884 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj"] Sep 08 15:30:18.130293 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:18.130087 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj" podUID="54c5753c-7a27-44ab-8e01-41273618e0f1" containerName="payload-pre-processing" containerID="cri-o://da11e9ad1ed2901438897aba923a585d1dcdc6041f92f363b3a5489b74c73759" gracePeriod=30 Sep 08 15:30:18.398482 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:18.398415 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj" Sep 08 15:30:18.437682 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:18.437642 2572 generic.go:356] "Generic (PLEG): container finished" podID="54c5753c-7a27-44ab-8e01-41273618e0f1" containerID="da11e9ad1ed2901438897aba923a585d1dcdc6041f92f363b3a5489b74c73759" exitCode=0 Sep 08 15:30:18.437808 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:18.437699 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj" Sep 08 15:30:18.437808 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:18.437726 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj" event={"ID":"54c5753c-7a27-44ab-8e01-41273618e0f1","Type":"ContainerDied","Data":"da11e9ad1ed2901438897aba923a585d1dcdc6041f92f363b3a5489b74c73759"} Sep 08 15:30:18.437808 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:18.437801 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj" event={"ID":"54c5753c-7a27-44ab-8e01-41273618e0f1","Type":"ContainerDied","Data":"bbe4668fdc6bad38ef60b73aadff4ca834514a87bec65b1c247b177a0f65292b"} Sep 08 15:30:18.437930 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:18.437824 2572 scope.go:117] "RemoveContainer" containerID="da11e9ad1ed2901438897aba923a585d1dcdc6041f92f363b3a5489b74c73759" Sep 08 15:30:18.446441 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:18.446421 2572 scope.go:117] "RemoveContainer" containerID="da11e9ad1ed2901438897aba923a585d1dcdc6041f92f363b3a5489b74c73759" Sep 08 15:30:18.446703 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:30:18.446684 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da11e9ad1ed2901438897aba923a585d1dcdc6041f92f363b3a5489b74c73759\": container with ID starting with da11e9ad1ed2901438897aba923a585d1dcdc6041f92f363b3a5489b74c73759 not found: ID does not exist" containerID="da11e9ad1ed2901438897aba923a585d1dcdc6041f92f363b3a5489b74c73759" Sep 08 15:30:18.446758 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:18.446710 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da11e9ad1ed2901438897aba923a585d1dcdc6041f92f363b3a5489b74c73759"} err="failed to get container status \"da11e9ad1ed2901438897aba923a585d1dcdc6041f92f363b3a5489b74c73759\": rpc error: code = NotFound desc = could not find container \"da11e9ad1ed2901438897aba923a585d1dcdc6041f92f363b3a5489b74c73759\": container with ID starting with da11e9ad1ed2901438897aba923a585d1dcdc6041f92f363b3a5489b74c73759 not found: ID does not exist" Sep 08 15:30:18.520426 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:18.520394 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c5753c-7a27-44ab-8e01-41273618e0f1-plugins-config-volume\") pod \"54c5753c-7a27-44ab-8e01-41273618e0f1\" (UID: \"54c5753c-7a27-44ab-8e01-41273618e0f1\") " Sep 08 15:30:18.520599 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:18.520441 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9v9kj\" (UniqueName: \"kubernetes.io/projected/54c5753c-7a27-44ab-8e01-41273618e0f1-kube-api-access-9v9kj\") pod \"54c5753c-7a27-44ab-8e01-41273618e0f1\" (UID: \"54c5753c-7a27-44ab-8e01-41273618e0f1\") " Sep 08 15:30:18.520793 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:18.520756 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c5753c-7a27-44ab-8e01-41273618e0f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "54c5753c-7a27-44ab-8e01-41273618e0f1" (UID: "54c5753c-7a27-44ab-8e01-41273618e0f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:30:18.522588 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:18.522567 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c5753c-7a27-44ab-8e01-41273618e0f1-kube-api-access-9v9kj" (OuterVolumeSpecName: "kube-api-access-9v9kj") pod "54c5753c-7a27-44ab-8e01-41273618e0f1" (UID: "54c5753c-7a27-44ab-8e01-41273618e0f1"). InnerVolumeSpecName "kube-api-access-9v9kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:30:18.621114 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:18.621084 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c5753c-7a27-44ab-8e01-41273618e0f1-plugins-config-volume\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:30:18.621114 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:18.621107 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v9kj\" (UniqueName: \"kubernetes.io/projected/54c5753c-7a27-44ab-8e01-41273618e0f1-kube-api-access-9v9kj\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:30:18.807019 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:18.806991 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj"] Sep 08 15:30:18.826545 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:18.826526 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d8e721-5f544b9d54-bk4kj"] Sep 08 15:30:20.370944 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:20.370910 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54c5753c-7a27-44ab-8e01-41273618e0f1" path="/var/lib/kubelet/pods/54c5753c-7a27-44ab-8e01-41273618e0f1/volumes" Sep 08 15:30:29.250915 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:29.248004 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j"] Sep 08 15:30:29.250915 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:29.248344 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" podUID="6da6535b-af05-4dbb-87f5-06eb503b80f5" containerName="istio-proxy" containerID="cri-o://7c3529981c155af9cab9642478ba4b01b5017d15288b92c561f237b416915846" gracePeriod=30 Sep 08 15:30:34.485023 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.485000 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:30:34.511717 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.511684 2572 generic.go:356] "Generic (PLEG): container finished" podID="6da6535b-af05-4dbb-87f5-06eb503b80f5" containerID="7c3529981c155af9cab9642478ba4b01b5017d15288b92c561f237b416915846" exitCode=0 Sep 08 15:30:34.511840 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.511739 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" Sep 08 15:30:34.511840 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.511763 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" event={"ID":"6da6535b-af05-4dbb-87f5-06eb503b80f5","Type":"ContainerDied","Data":"7c3529981c155af9cab9642478ba4b01b5017d15288b92c561f237b416915846"} Sep 08 15:30:34.511927 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.511842 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j" event={"ID":"6da6535b-af05-4dbb-87f5-06eb503b80f5","Type":"ContainerDied","Data":"975dd99b268125a72e28c13cf5b07acb022fbd9702ee05358901dbfd0c218741"} Sep 08 15:30:34.511927 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.511866 2572 scope.go:117] "RemoveContainer" containerID="7c3529981c155af9cab9642478ba4b01b5017d15288b92c561f237b416915846" Sep 08 15:30:34.519960 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.519940 2572 scope.go:117] "RemoveContainer" containerID="f2f5101343c046318b17838001e5ca052e061b3943a01787bb136cd2c1012976" Sep 08 15:30:34.527184 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.527166 2572 scope.go:117] "RemoveContainer" containerID="7c3529981c155af9cab9642478ba4b01b5017d15288b92c561f237b416915846" Sep 08 15:30:34.527465 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:30:34.527447 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c3529981c155af9cab9642478ba4b01b5017d15288b92c561f237b416915846\": container with ID starting with 7c3529981c155af9cab9642478ba4b01b5017d15288b92c561f237b416915846 not found: ID does not exist" containerID="7c3529981c155af9cab9642478ba4b01b5017d15288b92c561f237b416915846" Sep 08 15:30:34.527523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.527473 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c3529981c155af9cab9642478ba4b01b5017d15288b92c561f237b416915846"} err="failed to get container status \"7c3529981c155af9cab9642478ba4b01b5017d15288b92c561f237b416915846\": rpc error: code = NotFound desc = could not find container \"7c3529981c155af9cab9642478ba4b01b5017d15288b92c561f237b416915846\": container with ID starting with 7c3529981c155af9cab9642478ba4b01b5017d15288b92c561f237b416915846 not found: ID does not exist" Sep 08 15:30:34.527523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.527510 2572 scope.go:117] "RemoveContainer" containerID="f2f5101343c046318b17838001e5ca052e061b3943a01787bb136cd2c1012976" Sep 08 15:30:34.527781 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:30:34.527764 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f5101343c046318b17838001e5ca052e061b3943a01787bb136cd2c1012976\": container with ID starting with f2f5101343c046318b17838001e5ca052e061b3943a01787bb136cd2c1012976 not found: ID does not exist" containerID="f2f5101343c046318b17838001e5ca052e061b3943a01787bb136cd2c1012976" Sep 08 15:30:34.527837 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.527786 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f5101343c046318b17838001e5ca052e061b3943a01787bb136cd2c1012976"} err="failed to get container status \"f2f5101343c046318b17838001e5ca052e061b3943a01787bb136cd2c1012976\": rpc error: code = NotFound desc = could not find container \"f2f5101343c046318b17838001e5ca052e061b3943a01787bb136cd2c1012976\": container with ID starting with f2f5101343c046318b17838001e5ca052e061b3943a01787bb136cd2c1012976 not found: ID does not exist" Sep 08 15:30:34.640940 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.640919 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-podinfo\") pod \"6da6535b-af05-4dbb-87f5-06eb503b80f5\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " Sep 08 15:30:34.641058 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.640951 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-workload-certs\") pod \"6da6535b-af05-4dbb-87f5-06eb503b80f5\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " Sep 08 15:30:34.641058 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.641008 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-envoy\") pod \"6da6535b-af05-4dbb-87f5-06eb503b80f5\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " Sep 08 15:30:34.641058 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.641044 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-credential-socket\") pod \"6da6535b-af05-4dbb-87f5-06eb503b80f5\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " Sep 08 15:30:34.641208 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.641075 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-token\") pod \"6da6535b-af05-4dbb-87f5-06eb503b80f5\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " Sep 08 15:30:34.641208 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.641117 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da6535b-af05-4dbb-87f5-06eb503b80f5-istiod-ca-cert\") pod \"6da6535b-af05-4dbb-87f5-06eb503b80f5\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " Sep 08 15:30:34.641208 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.641144 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpppw\" (UniqueName: \"kubernetes.io/projected/6da6535b-af05-4dbb-87f5-06eb503b80f5-kube-api-access-gpppw\") pod \"6da6535b-af05-4dbb-87f5-06eb503b80f5\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " Sep 08 15:30:34.641355 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.641211 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-workload-socket\") pod \"6da6535b-af05-4dbb-87f5-06eb503b80f5\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " Sep 08 15:30:34.641355 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.641235 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-data\") pod \"6da6535b-af05-4dbb-87f5-06eb503b80f5\" (UID: \"6da6535b-af05-4dbb-87f5-06eb503b80f5\") " Sep 08 15:30:34.641355 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.641238 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6da6535b-af05-4dbb-87f5-06eb503b80f5" (UID: "6da6535b-af05-4dbb-87f5-06eb503b80f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:30:34.641521 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.641387 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6da6535b-af05-4dbb-87f5-06eb503b80f5" (UID: "6da6535b-af05-4dbb-87f5-06eb503b80f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:30:34.641521 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.641448 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-workload-certs\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:30:34.641521 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.641510 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6da6535b-af05-4dbb-87f5-06eb503b80f5" (UID: "6da6535b-af05-4dbb-87f5-06eb503b80f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:30:34.641703 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.641684 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "6da6535b-af05-4dbb-87f5-06eb503b80f5" (UID: "6da6535b-af05-4dbb-87f5-06eb503b80f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:30:34.642153 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.642081 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da6535b-af05-4dbb-87f5-06eb503b80f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6da6535b-af05-4dbb-87f5-06eb503b80f5" (UID: "6da6535b-af05-4dbb-87f5-06eb503b80f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:30:34.643166 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.643146 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6da6535b-af05-4dbb-87f5-06eb503b80f5" (UID: "6da6535b-af05-4dbb-87f5-06eb503b80f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 15:30:34.643276 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.643256 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6da6535b-af05-4dbb-87f5-06eb503b80f5" (UID: "6da6535b-af05-4dbb-87f5-06eb503b80f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:30:34.643333 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.643317 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "6da6535b-af05-4dbb-87f5-06eb503b80f5" (UID: "6da6535b-af05-4dbb-87f5-06eb503b80f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:30:34.643616 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.643588 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da6535b-af05-4dbb-87f5-06eb503b80f5-kube-api-access-gpppw" (OuterVolumeSpecName: "kube-api-access-gpppw") pod "6da6535b-af05-4dbb-87f5-06eb503b80f5" (UID: "6da6535b-af05-4dbb-87f5-06eb503b80f5"). InnerVolumeSpecName "kube-api-access-gpppw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:30:34.741905 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.741880 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-envoy\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:30:34.741905 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.741905 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-credential-socket\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:30:34.742031 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.741918 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-token\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:30:34.742031 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.741930 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da6535b-af05-4dbb-87f5-06eb503b80f5-istiod-ca-cert\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:30:34.742031 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.741942 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpppw\" (UniqueName: \"kubernetes.io/projected/6da6535b-af05-4dbb-87f5-06eb503b80f5-kube-api-access-gpppw\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:30:34.742031 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.741953 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-workload-socket\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:30:34.742031 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.741966 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-data\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:30:34.742031 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.741978 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da6535b-af05-4dbb-87f5-06eb503b80f5-istio-podinfo\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:30:34.844480 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.844458 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j"] Sep 08 15:30:34.862084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:34.862061 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d8e721-openshift-default-6d566876b8-67t2j"] Sep 08 15:30:36.371694 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:36.371669 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6da6535b-af05-4dbb-87f5-06eb503b80f5" path="/var/lib/kubelet/pods/6da6535b-af05-4dbb-87f5-06eb503b80f5/volumes" Sep 08 15:30:40.749931 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:40.749892 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b"] Sep 08 15:30:40.750355 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:40.750152 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" podUID="29c11455-7c11-472b-ad6b-4d1168a62eb4" containerName="maas-api" containerID="cri-o://195a6e2dc3084f3998a14d6646eda63ba8ec69e62d5a2c3cb63c1136190e761e" gracePeriod=30 Sep 08 15:30:40.982291 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:40.982259 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc"] Sep 08 15:30:40.982476 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:40.982448 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc" podUID="3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0" containerName="payload-processing" containerID="cri-o://8088edc1336ac8c63cdce115319ad9d5453abb1e6c6a2c8a6a8c099b4a027260" gracePeriod=30 Sep 08 15:30:40.996139 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:40.996118 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" Sep 08 15:30:41.001724 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.001668 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk"] Sep 08 15:30:41.001902 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.001881 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk" podUID="96c18853-0e4e-4b1e-bf35-51c326a9fe2f" containerName="payload-pre-processing" containerID="cri-o://2bd1d51ed025f8cd318f8f681b336418dd05265e9a4b3e66c10eb0bb0e9356da" gracePeriod=30 Sep 08 15:30:41.086942 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.086917 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghqpc\" (UniqueName: \"kubernetes.io/projected/29c11455-7c11-472b-ad6b-4d1168a62eb4-kube-api-access-ghqpc\") pod \"29c11455-7c11-472b-ad6b-4d1168a62eb4\" (UID: \"29c11455-7c11-472b-ad6b-4d1168a62eb4\") " Sep 08 15:30:41.087033 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.086996 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c11455-7c11-472b-ad6b-4d1168a62eb4-maas-api-tls\") pod \"29c11455-7c11-472b-ad6b-4d1168a62eb4\" (UID: \"29c11455-7c11-472b-ad6b-4d1168a62eb4\") " Sep 08 15:30:41.087033 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.087016 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c11455-7c11-472b-ad6b-4d1168a62eb4-metrics-certs\") pod \"29c11455-7c11-472b-ad6b-4d1168a62eb4\" (UID: \"29c11455-7c11-472b-ad6b-4d1168a62eb4\") " Sep 08 15:30:41.087161 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.087067 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c11455-7c11-472b-ad6b-4d1168a62eb4-openshift-service-ca\") pod \"29c11455-7c11-472b-ad6b-4d1168a62eb4\" (UID: \"29c11455-7c11-472b-ad6b-4d1168a62eb4\") " Sep 08 15:30:41.087705 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.087675 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c11455-7c11-472b-ad6b-4d1168a62eb4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "29c11455-7c11-472b-ad6b-4d1168a62eb4" (UID: "29c11455-7c11-472b-ad6b-4d1168a62eb4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:30:41.089112 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.089088 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c11455-7c11-472b-ad6b-4d1168a62eb4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "29c11455-7c11-472b-ad6b-4d1168a62eb4" (UID: "29c11455-7c11-472b-ad6b-4d1168a62eb4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 15:30:41.089231 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.089121 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c11455-7c11-472b-ad6b-4d1168a62eb4-kube-api-access-ghqpc" (OuterVolumeSpecName: "kube-api-access-ghqpc") pod "29c11455-7c11-472b-ad6b-4d1168a62eb4" (UID: "29c11455-7c11-472b-ad6b-4d1168a62eb4"). InnerVolumeSpecName "kube-api-access-ghqpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:30:41.089292 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.089249 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c11455-7c11-472b-ad6b-4d1168a62eb4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "29c11455-7c11-472b-ad6b-4d1168a62eb4" (UID: "29c11455-7c11-472b-ad6b-4d1168a62eb4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 15:30:41.188440 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.188410 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c11455-7c11-472b-ad6b-4d1168a62eb4-maas-api-tls\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:30:41.188575 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.188443 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c11455-7c11-472b-ad6b-4d1168a62eb4-metrics-certs\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:30:41.188575 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.188458 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c11455-7c11-472b-ad6b-4d1168a62eb4-openshift-service-ca\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:30:41.188575 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.188472 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghqpc\" (UniqueName: \"kubernetes.io/projected/29c11455-7c11-472b-ad6b-4d1168a62eb4-kube-api-access-ghqpc\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:30:41.250784 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.250755 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc" Sep 08 15:30:41.259507 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.259469 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk" Sep 08 15:30:41.389683 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.389653 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxpx2\" (UniqueName: \"kubernetes.io/projected/3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0-kube-api-access-sxpx2\") pod \"3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0\" (UID: \"3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0\") " Sep 08 15:30:41.389858 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.389698 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96c18853-0e4e-4b1e-bf35-51c326a9fe2f-plugins-config-volume\") pod \"96c18853-0e4e-4b1e-bf35-51c326a9fe2f\" (UID: \"96c18853-0e4e-4b1e-bf35-51c326a9fe2f\") " Sep 08 15:30:41.389858 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.389726 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0-plugins-config-volume\") pod \"3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0\" (UID: \"3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0\") " Sep 08 15:30:41.389858 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.389755 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dfl9\" (UniqueName: \"kubernetes.io/projected/96c18853-0e4e-4b1e-bf35-51c326a9fe2f-kube-api-access-9dfl9\") pod \"96c18853-0e4e-4b1e-bf35-51c326a9fe2f\" (UID: \"96c18853-0e4e-4b1e-bf35-51c326a9fe2f\") " Sep 08 15:30:41.390114 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.390091 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0" (UID: "3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:30:41.390184 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.390090 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c18853-0e4e-4b1e-bf35-51c326a9fe2f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96c18853-0e4e-4b1e-bf35-51c326a9fe2f" (UID: "96c18853-0e4e-4b1e-bf35-51c326a9fe2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:30:41.391839 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.391813 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c18853-0e4e-4b1e-bf35-51c326a9fe2f-kube-api-access-9dfl9" (OuterVolumeSpecName: "kube-api-access-9dfl9") pod "96c18853-0e4e-4b1e-bf35-51c326a9fe2f" (UID: "96c18853-0e4e-4b1e-bf35-51c326a9fe2f"). InnerVolumeSpecName "kube-api-access-9dfl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:30:41.391921 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.391898 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0-kube-api-access-sxpx2" (OuterVolumeSpecName: "kube-api-access-sxpx2") pod "3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0" (UID: "3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0"). InnerVolumeSpecName "kube-api-access-sxpx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:30:41.490552 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.490533 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96c18853-0e4e-4b1e-bf35-51c326a9fe2f-plugins-config-volume\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:30:41.490552 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.490552 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0-plugins-config-volume\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:30:41.490671 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.490562 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dfl9\" (UniqueName: \"kubernetes.io/projected/96c18853-0e4e-4b1e-bf35-51c326a9fe2f-kube-api-access-9dfl9\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:30:41.490671 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.490570 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxpx2\" (UniqueName: \"kubernetes.io/projected/3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0-kube-api-access-sxpx2\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:30:41.543215 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.543159 2572 generic.go:356] "Generic (PLEG): container finished" podID="29c11455-7c11-472b-ad6b-4d1168a62eb4" containerID="195a6e2dc3084f3998a14d6646eda63ba8ec69e62d5a2c3cb63c1136190e761e" exitCode=0 Sep 08 15:30:41.543215 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.543211 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" Sep 08 15:30:41.543367 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.543246 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" event={"ID":"29c11455-7c11-472b-ad6b-4d1168a62eb4","Type":"ContainerDied","Data":"195a6e2dc3084f3998a14d6646eda63ba8ec69e62d5a2c3cb63c1136190e761e"} Sep 08 15:30:41.543367 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.543305 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b" event={"ID":"29c11455-7c11-472b-ad6b-4d1168a62eb4","Type":"ContainerDied","Data":"5004dbb0f85da0d54bf0b34ed2b85068508f3c0f6ce5ffb1875177efbad19e09"} Sep 08 15:30:41.543367 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.543338 2572 scope.go:117] "RemoveContainer" containerID="195a6e2dc3084f3998a14d6646eda63ba8ec69e62d5a2c3cb63c1136190e761e" Sep 08 15:30:41.544696 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.544663 2572 generic.go:356] "Generic (PLEG): container finished" podID="3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0" containerID="8088edc1336ac8c63cdce115319ad9d5453abb1e6c6a2c8a6a8c099b4a027260" exitCode=0 Sep 08 15:30:41.544806 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.544707 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc" Sep 08 15:30:41.544806 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.544711 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc" event={"ID":"3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0","Type":"ContainerDied","Data":"8088edc1336ac8c63cdce115319ad9d5453abb1e6c6a2c8a6a8c099b4a027260"} Sep 08 15:30:41.544806 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.544765 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc" event={"ID":"3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0","Type":"ContainerDied","Data":"f393a15d9dcf0c471964c7ae61f34b4beaa6e4f0a5bb96ab80286401e5358c4e"} Sep 08 15:30:41.546299 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.546277 2572 generic.go:356] "Generic (PLEG): container finished" podID="96c18853-0e4e-4b1e-bf35-51c326a9fe2f" containerID="2bd1d51ed025f8cd318f8f681b336418dd05265e9a4b3e66c10eb0bb0e9356da" exitCode=0 Sep 08 15:30:41.546402 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.546333 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk" Sep 08 15:30:41.546456 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.546332 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk" event={"ID":"96c18853-0e4e-4b1e-bf35-51c326a9fe2f","Type":"ContainerDied","Data":"2bd1d51ed025f8cd318f8f681b336418dd05265e9a4b3e66c10eb0bb0e9356da"} Sep 08 15:30:41.546456 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.546449 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk" event={"ID":"96c18853-0e4e-4b1e-bf35-51c326a9fe2f","Type":"ContainerDied","Data":"ee5b07660a3f383b8d5d47057d012d19788d230b23ac15dc7eac7fd80a086e1d"} Sep 08 15:30:41.554649 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.554631 2572 scope.go:117] "RemoveContainer" containerID="195a6e2dc3084f3998a14d6646eda63ba8ec69e62d5a2c3cb63c1136190e761e" Sep 08 15:30:41.555154 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:30:41.555133 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"195a6e2dc3084f3998a14d6646eda63ba8ec69e62d5a2c3cb63c1136190e761e\": container with ID starting with 195a6e2dc3084f3998a14d6646eda63ba8ec69e62d5a2c3cb63c1136190e761e not found: ID does not exist" containerID="195a6e2dc3084f3998a14d6646eda63ba8ec69e62d5a2c3cb63c1136190e761e" Sep 08 15:30:41.555244 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.555162 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195a6e2dc3084f3998a14d6646eda63ba8ec69e62d5a2c3cb63c1136190e761e"} err="failed to get container status \"195a6e2dc3084f3998a14d6646eda63ba8ec69e62d5a2c3cb63c1136190e761e\": rpc error: code = NotFound desc = could not find container \"195a6e2dc3084f3998a14d6646eda63ba8ec69e62d5a2c3cb63c1136190e761e\": container with ID starting with 195a6e2dc3084f3998a14d6646eda63ba8ec69e62d5a2c3cb63c1136190e761e not found: ID does not exist" Sep 08 15:30:41.555244 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.555181 2572 scope.go:117] "RemoveContainer" containerID="8088edc1336ac8c63cdce115319ad9d5453abb1e6c6a2c8a6a8c099b4a027260" Sep 08 15:30:41.562882 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.562866 2572 scope.go:117] "RemoveContainer" containerID="8088edc1336ac8c63cdce115319ad9d5453abb1e6c6a2c8a6a8c099b4a027260" Sep 08 15:30:41.563145 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:30:41.563123 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8088edc1336ac8c63cdce115319ad9d5453abb1e6c6a2c8a6a8c099b4a027260\": container with ID starting with 8088edc1336ac8c63cdce115319ad9d5453abb1e6c6a2c8a6a8c099b4a027260 not found: ID does not exist" containerID="8088edc1336ac8c63cdce115319ad9d5453abb1e6c6a2c8a6a8c099b4a027260" Sep 08 15:30:41.563196 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.563151 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8088edc1336ac8c63cdce115319ad9d5453abb1e6c6a2c8a6a8c099b4a027260"} err="failed to get container status \"8088edc1336ac8c63cdce115319ad9d5453abb1e6c6a2c8a6a8c099b4a027260\": rpc error: code = NotFound desc = could not find container \"8088edc1336ac8c63cdce115319ad9d5453abb1e6c6a2c8a6a8c099b4a027260\": container with ID starting with 8088edc1336ac8c63cdce115319ad9d5453abb1e6c6a2c8a6a8c099b4a027260 not found: ID does not exist" Sep 08 15:30:41.563196 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.563174 2572 scope.go:117] "RemoveContainer" containerID="2bd1d51ed025f8cd318f8f681b336418dd05265e9a4b3e66c10eb0bb0e9356da" Sep 08 15:30:41.570244 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.570230 2572 scope.go:117] "RemoveContainer" containerID="2bd1d51ed025f8cd318f8f681b336418dd05265e9a4b3e66c10eb0bb0e9356da" Sep 08 15:30:41.570536 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:30:41.570482 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd1d51ed025f8cd318f8f681b336418dd05265e9a4b3e66c10eb0bb0e9356da\": container with ID starting with 2bd1d51ed025f8cd318f8f681b336418dd05265e9a4b3e66c10eb0bb0e9356da not found: ID does not exist" containerID="2bd1d51ed025f8cd318f8f681b336418dd05265e9a4b3e66c10eb0bb0e9356da" Sep 08 15:30:41.570601 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.570546 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd1d51ed025f8cd318f8f681b336418dd05265e9a4b3e66c10eb0bb0e9356da"} err="failed to get container status \"2bd1d51ed025f8cd318f8f681b336418dd05265e9a4b3e66c10eb0bb0e9356da\": rpc error: code = NotFound desc = could not find container \"2bd1d51ed025f8cd318f8f681b336418dd05265e9a4b3e66c10eb0bb0e9356da\": container with ID starting with 2bd1d51ed025f8cd318f8f681b336418dd05265e9a4b3e66c10eb0bb0e9356da not found: ID does not exist" Sep 08 15:30:41.614534 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.614510 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b"] Sep 08 15:30:41.627441 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.627420 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-677a6c-6c84bff4c9-hbh2b"] Sep 08 15:30:41.653471 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.653452 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk"] Sep 08 15:30:41.657763 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.657745 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-677a6c-8464b67dcd-nmgbk"] Sep 08 15:30:41.683053 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.683028 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc"] Sep 08 15:30:41.692551 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:41.691578 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-677a6c-8d578db87-ffcjc"] Sep 08 15:30:42.371915 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:42.371887 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29c11455-7c11-472b-ad6b-4d1168a62eb4" path="/var/lib/kubelet/pods/29c11455-7c11-472b-ad6b-4d1168a62eb4/volumes" Sep 08 15:30:42.372220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:42.372205 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0" path="/var/lib/kubelet/pods/3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0/volumes" Sep 08 15:30:42.372524 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:42.372509 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96c18853-0e4e-4b1e-bf35-51c326a9fe2f" path="/var/lib/kubelet/pods/96c18853-0e4e-4b1e-bf35-51c326a9fe2f/volumes" Sep 08 15:30:53.099469 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099436 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc"] Sep 08 15:30:53.099818 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099723 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e90d1787-ea44-47bb-aedf-482ea76c029a" containerName="cleanup" Sep 08 15:30:53.099818 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099733 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e90d1787-ea44-47bb-aedf-482ea76c029a" containerName="cleanup" Sep 08 15:30:53.099818 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099747 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54c5753c-7a27-44ab-8e01-41273618e0f1" containerName="payload-pre-processing" Sep 08 15:30:53.099818 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099753 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="54c5753c-7a27-44ab-8e01-41273618e0f1" containerName="payload-pre-processing" Sep 08 15:30:53.099818 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099761 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96c18853-0e4e-4b1e-bf35-51c326a9fe2f" containerName="payload-pre-processing" Sep 08 15:30:53.099818 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099766 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c18853-0e4e-4b1e-bf35-51c326a9fe2f" containerName="payload-pre-processing" Sep 08 15:30:53.099818 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099775 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6da6535b-af05-4dbb-87f5-06eb503b80f5" containerName="istio-proxy" Sep 08 15:30:53.099818 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099780 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6da6535b-af05-4dbb-87f5-06eb503b80f5" containerName="istio-proxy" Sep 08 15:30:53.099818 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099788 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c11455-7c11-472b-ad6b-4d1168a62eb4" containerName="maas-api" Sep 08 15:30:53.099818 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099792 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c11455-7c11-472b-ad6b-4d1168a62eb4" containerName="maas-api" Sep 08 15:30:53.099818 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099802 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0" containerName="payload-processing" Sep 08 15:30:53.099818 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099807 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0" containerName="payload-processing" Sep 08 15:30:53.100158 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099832 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6da6535b-af05-4dbb-87f5-06eb503b80f5" containerName="istio-proxy" Sep 08 15:30:53.100158 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099837 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6da6535b-af05-4dbb-87f5-06eb503b80f5" containerName="istio-proxy" Sep 08 15:30:53.100158 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099881 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6da6535b-af05-4dbb-87f5-06eb503b80f5" containerName="istio-proxy" Sep 08 15:30:53.100158 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099888 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="96c18853-0e4e-4b1e-bf35-51c326a9fe2f" containerName="payload-pre-processing" Sep 08 15:30:53.100158 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099894 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="54c5753c-7a27-44ab-8e01-41273618e0f1" containerName="payload-pre-processing" Sep 08 15:30:53.100158 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099899 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c11455-7c11-472b-ad6b-4d1168a62eb4" containerName="maas-api" Sep 08 15:30:53.100158 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099905 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6da6535b-af05-4dbb-87f5-06eb503b80f5" containerName="istio-proxy" Sep 08 15:30:53.100158 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099910 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e90d1787-ea44-47bb-aedf-482ea76c029a" containerName="cleanup" Sep 08 15:30:53.100158 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.099917 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf0bc11-8e1b-4ab9-b67c-9ec7571094d0" containerName="payload-processing" Sep 08 15:30:53.104310 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.104291 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc" Sep 08 15:30:53.113025 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.113006 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-49bfbc48\"" Sep 08 15:30:53.161228 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.161192 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc"] Sep 08 15:30:53.280195 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.280161 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29dfdcf5-c875-489e-9a2d-f690c60259fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc\" (UID: \"29dfdcf5-c875-489e-9a2d-f690c60259fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc" Sep 08 15:30:53.280318 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.280228 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nh8g\" (UniqueName: \"kubernetes.io/projected/29dfdcf5-c875-489e-9a2d-f690c60259fe-kube-api-access-5nh8g\") pod \"payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc\" (UID: \"29dfdcf5-c875-489e-9a2d-f690c60259fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc" Sep 08 15:30:53.364318 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.364259 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t"] Sep 08 15:30:53.367055 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.367040 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t" Sep 08 15:30:53.370122 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.370105 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-49bfbc48-dockercfg-t26dw\"" Sep 08 15:30:53.380901 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.380881 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nh8g\" (UniqueName: \"kubernetes.io/projected/29dfdcf5-c875-489e-9a2d-f690c60259fe-kube-api-access-5nh8g\") pod \"payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc\" (UID: \"29dfdcf5-c875-489e-9a2d-f690c60259fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc" Sep 08 15:30:53.380987 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.380928 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29dfdcf5-c875-489e-9a2d-f690c60259fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc\" (UID: \"29dfdcf5-c875-489e-9a2d-f690c60259fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc" Sep 08 15:30:53.381480 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.381465 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29dfdcf5-c875-489e-9a2d-f690c60259fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc\" (UID: \"29dfdcf5-c875-489e-9a2d-f690c60259fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc" Sep 08 15:30:53.389792 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.389766 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t"] Sep 08 15:30:53.414983 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.414963 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nh8g\" (UniqueName: \"kubernetes.io/projected/29dfdcf5-c875-489e-9a2d-f690c60259fe-kube-api-access-5nh8g\") pod \"payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc\" (UID: \"29dfdcf5-c875-489e-9a2d-f690c60259fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc" Sep 08 15:30:53.481442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.481416 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5fa604-6b2c-482a-8d11-88480dc58178-plugins-config-volume\") pod \"payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t\" (UID: \"0b5fa604-6b2c-482a-8d11-88480dc58178\") " pod="openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t" Sep 08 15:30:53.481555 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.481444 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pstbk\" (UniqueName: \"kubernetes.io/projected/0b5fa604-6b2c-482a-8d11-88480dc58178-kube-api-access-pstbk\") pod \"payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t\" (UID: \"0b5fa604-6b2c-482a-8d11-88480dc58178\") " pod="openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t" Sep 08 15:30:53.582400 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.582375 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5fa604-6b2c-482a-8d11-88480dc58178-plugins-config-volume\") pod \"payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t\" (UID: \"0b5fa604-6b2c-482a-8d11-88480dc58178\") " pod="openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t" Sep 08 15:30:53.582483 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.582404 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pstbk\" (UniqueName: \"kubernetes.io/projected/0b5fa604-6b2c-482a-8d11-88480dc58178-kube-api-access-pstbk\") pod \"payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t\" (UID: \"0b5fa604-6b2c-482a-8d11-88480dc58178\") " pod="openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t" Sep 08 15:30:53.582984 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.582967 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5fa604-6b2c-482a-8d11-88480dc58178-plugins-config-volume\") pod \"payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t\" (UID: \"0b5fa604-6b2c-482a-8d11-88480dc58178\") " pod="openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t" Sep 08 15:30:53.605084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.605058 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pstbk\" (UniqueName: \"kubernetes.io/projected/0b5fa604-6b2c-482a-8d11-88480dc58178-kube-api-access-pstbk\") pod \"payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t\" (UID: \"0b5fa604-6b2c-482a-8d11-88480dc58178\") " pod="openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t" Sep 08 15:30:53.675453 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.675405 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t" Sep 08 15:30:53.713911 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.713874 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc" Sep 08 15:30:53.961480 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:53.961414 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t"] Sep 08 15:30:53.967204 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:30:53.967178 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5fa604_6b2c_482a_8d11_88480dc58178.slice/crio-731ffe0f11189dd8e311f327d3457e97b5726331c1459f931a55db88940bb042 WatchSource:0}: Error finding container 731ffe0f11189dd8e311f327d3457e97b5726331c1459f931a55db88940bb042: Status 404 returned error can't find the container with id 731ffe0f11189dd8e311f327d3457e97b5726331c1459f931a55db88940bb042 Sep 08 15:30:54.025034 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:54.024998 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc"] Sep 08 15:30:54.028070 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:30:54.028051 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29dfdcf5_c875_489e_9a2d_f690c60259fe.slice/crio-5de154e39b2642ec971c775301fa4f8356fe01aedee3ceee8cb7a1ae8cd372fe WatchSource:0}: Error finding container 5de154e39b2642ec971c775301fa4f8356fe01aedee3ceee8cb7a1ae8cd372fe: Status 404 returned error can't find the container with id 5de154e39b2642ec971c775301fa4f8356fe01aedee3ceee8cb7a1ae8cd372fe Sep 08 15:30:54.605358 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:54.605328 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc" event={"ID":"29dfdcf5-c875-489e-9a2d-f690c60259fe","Type":"ContainerStarted","Data":"5ab2312fecc3c97a78afb1bd5e70c162f8dc99cb806bd9a3e3cc2df6ee066b54"} Sep 08 15:30:54.605765 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:54.605379 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc" event={"ID":"29dfdcf5-c875-489e-9a2d-f690c60259fe","Type":"ContainerStarted","Data":"5de154e39b2642ec971c775301fa4f8356fe01aedee3ceee8cb7a1ae8cd372fe"} Sep 08 15:30:54.605765 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:54.605424 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc" Sep 08 15:30:54.607119 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:54.607094 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t" event={"ID":"0b5fa604-6b2c-482a-8d11-88480dc58178","Type":"ContainerStarted","Data":"6f446f4e9206e53fc3432880d3f280fc0ca3afc471644af674e5c5f7cb54a9e7"} Sep 08 15:30:54.607253 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:54.607132 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t" event={"ID":"0b5fa604-6b2c-482a-8d11-88480dc58178","Type":"ContainerStarted","Data":"731ffe0f11189dd8e311f327d3457e97b5726331c1459f931a55db88940bb042"} Sep 08 15:30:54.607253 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:54.607188 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t" Sep 08 15:30:54.622731 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:54.622710 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6"] Sep 08 15:30:54.622870 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:54.622853 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" podUID="bccbad6b-5994-4a9b-be33-057ac78e5dc5" containerName="istio-proxy" containerID="cri-o://6d5c36f8bd8dd3a5c56249c86c1abe1d3a142453fa12edd83594ca2277884577" gracePeriod=30 Sep 08 15:30:54.641408 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:54.641368 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc" podStartSLOduration=1.641356441 podStartE2EDuration="1.641356441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:30:54.640138259 +0000 UTC m=+1316.792316862" watchObservedRunningTime="2026-09-08 15:30:54.641356441 +0000 UTC m=+1316.793535016" Sep 08 15:30:54.671569 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:54.671531 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t" podStartSLOduration=1.6715185030000002 podStartE2EDuration="1.671518503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:30:54.669707252 +0000 UTC m=+1316.821885834" watchObservedRunningTime="2026-09-08 15:30:54.671518503 +0000 UTC m=+1316.823697083" Sep 08 15:30:55.049379 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:55.049346 2572 patch_prober.go:28] interesting pod/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 15:30:55.049554 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:55.049405 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" podUID="bccbad6b-5994-4a9b-be33-057ac78e5dc5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 15:30:59.856655 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:30:59.856632 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:31:00.034520 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.034480 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-podinfo\") pod \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " Sep 08 15:31:00.034650 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.034543 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-credential-socket\") pod \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " Sep 08 15:31:00.034650 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.034571 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-workload-socket\") pod \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " Sep 08 15:31:00.034650 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.034593 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-token\") pod \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " Sep 08 15:31:00.034650 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.034607 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-workload-certs\") pod \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " Sep 08 15:31:00.034650 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.034634 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq5kx\" (UniqueName: \"kubernetes.io/projected/bccbad6b-5994-4a9b-be33-057ac78e5dc5-kube-api-access-gq5kx\") pod \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " Sep 08 15:31:00.034900 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.034659 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-data\") pod \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " Sep 08 15:31:00.034900 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.034698 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istiod-ca-cert\") pod \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " Sep 08 15:31:00.034900 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.034732 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-envoy\") pod \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\" (UID: \"bccbad6b-5994-4a9b-be33-057ac78e5dc5\") " Sep 08 15:31:00.034900 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.034854 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bccbad6b-5994-4a9b-be33-057ac78e5dc5" (UID: "bccbad6b-5994-4a9b-be33-057ac78e5dc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:31:00.035061 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.034880 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bccbad6b-5994-4a9b-be33-057ac78e5dc5" (UID: "bccbad6b-5994-4a9b-be33-057ac78e5dc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:31:00.035061 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.034931 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bccbad6b-5994-4a9b-be33-057ac78e5dc5" (UID: "bccbad6b-5994-4a9b-be33-057ac78e5dc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:31:00.035061 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.035045 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-credential-socket\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:31:00.035191 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.035065 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-workload-socket\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:31:00.035191 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.035080 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-workload-certs\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:31:00.035357 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.035326 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bccbad6b-5994-4a9b-be33-057ac78e5dc5" (UID: "bccbad6b-5994-4a9b-be33-057ac78e5dc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:31:00.035516 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.035434 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-data" (OuterVolumeSpecName: "istio-data") pod "bccbad6b-5994-4a9b-be33-057ac78e5dc5" (UID: "bccbad6b-5994-4a9b-be33-057ac78e5dc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:31:00.036833 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.036807 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bccbad6b-5994-4a9b-be33-057ac78e5dc5" (UID: "bccbad6b-5994-4a9b-be33-057ac78e5dc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 15:31:00.036833 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.036820 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-token" (OuterVolumeSpecName: "istio-token") pod "bccbad6b-5994-4a9b-be33-057ac78e5dc5" (UID: "bccbad6b-5994-4a9b-be33-057ac78e5dc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:31:00.036995 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.036979 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bccbad6b-5994-4a9b-be33-057ac78e5dc5" (UID: "bccbad6b-5994-4a9b-be33-057ac78e5dc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:31:00.037178 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.037155 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bccbad6b-5994-4a9b-be33-057ac78e5dc5-kube-api-access-gq5kx" (OuterVolumeSpecName: "kube-api-access-gq5kx") pod "bccbad6b-5994-4a9b-be33-057ac78e5dc5" (UID: "bccbad6b-5994-4a9b-be33-057ac78e5dc5"). InnerVolumeSpecName "kube-api-access-gq5kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:31:00.135925 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.135904 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-token\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:31:00.135925 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.135924 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq5kx\" (UniqueName: \"kubernetes.io/projected/bccbad6b-5994-4a9b-be33-057ac78e5dc5-kube-api-access-gq5kx\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:31:00.136030 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.135934 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-data\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:31:00.136030 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.135942 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istiod-ca-cert\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:31:00.136030 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.135949 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-envoy\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:31:00.136030 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.135956 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bccbad6b-5994-4a9b-be33-057ac78e5dc5-istio-podinfo\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:31:00.639368 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.639293 2572 generic.go:356] "Generic (PLEG): container finished" podID="bccbad6b-5994-4a9b-be33-057ac78e5dc5" containerID="6d5c36f8bd8dd3a5c56249c86c1abe1d3a142453fa12edd83594ca2277884577" exitCode=0 Sep 08 15:31:00.639368 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.639350 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" Sep 08 15:31:00.639690 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.639380 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" event={"ID":"bccbad6b-5994-4a9b-be33-057ac78e5dc5","Type":"ContainerDied","Data":"6d5c36f8bd8dd3a5c56249c86c1abe1d3a142453fa12edd83594ca2277884577"} Sep 08 15:31:00.639690 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.639461 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6" event={"ID":"bccbad6b-5994-4a9b-be33-057ac78e5dc5","Type":"ContainerDied","Data":"71c61ffeb36311f1aebdf6b3baf707af4d7101624a266bf17ec0f3eb06b984cf"} Sep 08 15:31:00.639690 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.639514 2572 scope.go:117] "RemoveContainer" containerID="6d5c36f8bd8dd3a5c56249c86c1abe1d3a142453fa12edd83594ca2277884577" Sep 08 15:31:00.640372 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.640332 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg"] Sep 08 15:31:00.640860 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.640710 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bccbad6b-5994-4a9b-be33-057ac78e5dc5" containerName="istio-proxy" Sep 08 15:31:00.640860 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.640731 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bccbad6b-5994-4a9b-be33-057ac78e5dc5" containerName="istio-proxy" Sep 08 15:31:00.640860 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.640760 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bccbad6b-5994-4a9b-be33-057ac78e5dc5" containerName="istio-proxy" Sep 08 15:31:00.640860 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.640768 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bccbad6b-5994-4a9b-be33-057ac78e5dc5" containerName="istio-proxy" Sep 08 15:31:00.640860 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.640848 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bccbad6b-5994-4a9b-be33-057ac78e5dc5" containerName="istio-proxy" Sep 08 15:31:00.640860 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.640861 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bccbad6b-5994-4a9b-be33-057ac78e5dc5" containerName="istio-proxy" Sep 08 15:31:00.644996 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.644975 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" Sep 08 15:31:00.648310 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.648294 2572 scope.go:117] "RemoveContainer" containerID="a45d366c91d1cbf581ce84e6a72178c32dce51700d304bbadd0a64dd035df463" Sep 08 15:31:00.648715 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.648694 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 15:31:00.649163 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.649143 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-2e082e-serving-cert\"" Sep 08 15:31:00.649163 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.649151 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hrbm8\"" Sep 08 15:31:00.649379 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.649292 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 08 15:31:00.650169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.650151 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 08 15:31:00.657438 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.657423 2572 scope.go:117] "RemoveContainer" containerID="6d5c36f8bd8dd3a5c56249c86c1abe1d3a142453fa12edd83594ca2277884577" Sep 08 15:31:00.657726 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:31:00.657709 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5c36f8bd8dd3a5c56249c86c1abe1d3a142453fa12edd83594ca2277884577\": container with ID starting with 6d5c36f8bd8dd3a5c56249c86c1abe1d3a142453fa12edd83594ca2277884577 not found: ID does not exist" containerID="6d5c36f8bd8dd3a5c56249c86c1abe1d3a142453fa12edd83594ca2277884577" Sep 08 15:31:00.657790 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.657736 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5c36f8bd8dd3a5c56249c86c1abe1d3a142453fa12edd83594ca2277884577"} err="failed to get container status \"6d5c36f8bd8dd3a5c56249c86c1abe1d3a142453fa12edd83594ca2277884577\": rpc error: code = NotFound desc = could not find container \"6d5c36f8bd8dd3a5c56249c86c1abe1d3a142453fa12edd83594ca2277884577\": container with ID starting with 6d5c36f8bd8dd3a5c56249c86c1abe1d3a142453fa12edd83594ca2277884577 not found: ID does not exist" Sep 08 15:31:00.657790 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.657760 2572 scope.go:117] "RemoveContainer" containerID="a45d366c91d1cbf581ce84e6a72178c32dce51700d304bbadd0a64dd035df463" Sep 08 15:31:00.658054 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:31:00.658020 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a45d366c91d1cbf581ce84e6a72178c32dce51700d304bbadd0a64dd035df463\": container with ID starting with a45d366c91d1cbf581ce84e6a72178c32dce51700d304bbadd0a64dd035df463 not found: ID does not exist" containerID="a45d366c91d1cbf581ce84e6a72178c32dce51700d304bbadd0a64dd035df463" Sep 08 15:31:00.658113 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.658058 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45d366c91d1cbf581ce84e6a72178c32dce51700d304bbadd0a64dd035df463"} err="failed to get container status \"a45d366c91d1cbf581ce84e6a72178c32dce51700d304bbadd0a64dd035df463\": rpc error: code = NotFound desc = could not find container \"a45d366c91d1cbf581ce84e6a72178c32dce51700d304bbadd0a64dd035df463\": container with ID starting with a45d366c91d1cbf581ce84e6a72178c32dce51700d304bbadd0a64dd035df463 not found: ID does not exist" Sep 08 15:31:00.668000 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.667979 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg"] Sep 08 15:31:00.740259 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.740240 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6"] Sep 08 15:31:00.744441 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.744420 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-677a6c-openshift-default-5dc78fcc76-9vln6"] Sep 08 15:31:00.841186 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.841161 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de86dadc-03ed-4180-ac3c-85216121be05-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg\" (UID: \"de86dadc-03ed-4180-ac3c-85216121be05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" Sep 08 15:31:00.841292 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.841206 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpxkc\" (UniqueName: \"kubernetes.io/projected/de86dadc-03ed-4180-ac3c-85216121be05-kube-api-access-cpxkc\") pod \"maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg\" (UID: \"de86dadc-03ed-4180-ac3c-85216121be05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" Sep 08 15:31:00.841348 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.841332 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de86dadc-03ed-4180-ac3c-85216121be05-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg\" (UID: \"de86dadc-03ed-4180-ac3c-85216121be05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" Sep 08 15:31:00.841390 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.841360 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de86dadc-03ed-4180-ac3c-85216121be05-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg\" (UID: \"de86dadc-03ed-4180-ac3c-85216121be05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" Sep 08 15:31:00.942526 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.942452 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de86dadc-03ed-4180-ac3c-85216121be05-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg\" (UID: \"de86dadc-03ed-4180-ac3c-85216121be05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" Sep 08 15:31:00.942526 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.942484 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpxkc\" (UniqueName: \"kubernetes.io/projected/de86dadc-03ed-4180-ac3c-85216121be05-kube-api-access-cpxkc\") pod \"maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg\" (UID: \"de86dadc-03ed-4180-ac3c-85216121be05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" Sep 08 15:31:00.942863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.942552 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de86dadc-03ed-4180-ac3c-85216121be05-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg\" (UID: \"de86dadc-03ed-4180-ac3c-85216121be05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" Sep 08 15:31:00.942863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.942580 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de86dadc-03ed-4180-ac3c-85216121be05-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg\" (UID: \"de86dadc-03ed-4180-ac3c-85216121be05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" Sep 08 15:31:00.942863 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:31:00.942713 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-serving-cert: secret "maas-api-e2e-shared-a-w3-2e082e-serving-cert" not found Sep 08 15:31:00.942863 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:31:00.942782 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de86dadc-03ed-4180-ac3c-85216121be05-maas-api-tls podName:de86dadc-03ed-4180-ac3c-85216121be05 nodeName:}" failed. No retries permitted until 2026-09-08 15:31:01.442761757 +0000 UTC m=+1323.594940314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de86dadc-03ed-4180-ac3c-85216121be05-maas-api-tls") pod "maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" (UID: "de86dadc-03ed-4180-ac3c-85216121be05") : secret "maas-api-e2e-shared-a-w3-2e082e-serving-cert" not found Sep 08 15:31:00.943169 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.943153 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de86dadc-03ed-4180-ac3c-85216121be05-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg\" (UID: \"de86dadc-03ed-4180-ac3c-85216121be05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" Sep 08 15:31:00.944936 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.944921 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de86dadc-03ed-4180-ac3c-85216121be05-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg\" (UID: \"de86dadc-03ed-4180-ac3c-85216121be05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" Sep 08 15:31:00.952825 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:00.952805 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpxkc\" (UniqueName: \"kubernetes.io/projected/de86dadc-03ed-4180-ac3c-85216121be05-kube-api-access-cpxkc\") pod \"maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg\" (UID: \"de86dadc-03ed-4180-ac3c-85216121be05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" Sep 08 15:31:01.447810 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:01.447778 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de86dadc-03ed-4180-ac3c-85216121be05-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg\" (UID: \"de86dadc-03ed-4180-ac3c-85216121be05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" Sep 08 15:31:01.450286 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:01.450260 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de86dadc-03ed-4180-ac3c-85216121be05-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg\" (UID: \"de86dadc-03ed-4180-ac3c-85216121be05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" Sep 08 15:31:01.554580 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:01.554553 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" Sep 08 15:31:01.818340 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:01.818308 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj"] Sep 08 15:31:01.822665 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:01.822649 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj" Sep 08 15:31:01.826744 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:01.826723 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2e082e\"" Sep 08 15:31:01.827229 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:01.827210 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2e082e-dockercfg-v76rj\"" Sep 08 15:31:01.843456 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:01.843431 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj"] Sep 08 15:31:01.872295 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:01.872277 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg"] Sep 08 15:31:01.873082 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:31:01.873064 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde86dadc_03ed_4180_ac3c_85216121be05.slice/crio-b74d2aa04d7566f541db0e118cbb4ff58562c84f4e192783a199185ebbb2f69a WatchSource:0}: Error finding container b74d2aa04d7566f541db0e118cbb4ff58562c84f4e192783a199185ebbb2f69a: Status 404 returned error can't find the container with id b74d2aa04d7566f541db0e118cbb4ff58562c84f4e192783a199185ebbb2f69a Sep 08 15:31:01.874593 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:01.874577 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 15:31:01.952306 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:01.952280 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b8901ef-372b-4fef-a6f1-92102ef628bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj\" (UID: \"6b8901ef-372b-4fef-a6f1-92102ef628bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj" Sep 08 15:31:01.952617 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:01.952319 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv9sk\" (UniqueName: \"kubernetes.io/projected/6b8901ef-372b-4fef-a6f1-92102ef628bd-kube-api-access-rv9sk\") pod \"payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj\" (UID: \"6b8901ef-372b-4fef-a6f1-92102ef628bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj" Sep 08 15:31:02.053124 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:02.053096 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b8901ef-372b-4fef-a6f1-92102ef628bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj\" (UID: \"6b8901ef-372b-4fef-a6f1-92102ef628bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj" Sep 08 15:31:02.053211 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:02.053125 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv9sk\" (UniqueName: \"kubernetes.io/projected/6b8901ef-372b-4fef-a6f1-92102ef628bd-kube-api-access-rv9sk\") pod \"payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj\" (UID: \"6b8901ef-372b-4fef-a6f1-92102ef628bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj" Sep 08 15:31:02.053735 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:02.053718 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b8901ef-372b-4fef-a6f1-92102ef628bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj\" (UID: \"6b8901ef-372b-4fef-a6f1-92102ef628bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj" Sep 08 15:31:02.065956 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:02.065935 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv9sk\" (UniqueName: \"kubernetes.io/projected/6b8901ef-372b-4fef-a6f1-92102ef628bd-kube-api-access-rv9sk\") pod \"payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj\" (UID: \"6b8901ef-372b-4fef-a6f1-92102ef628bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj" Sep 08 15:31:02.132979 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:02.132929 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj" Sep 08 15:31:02.372285 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:02.372258 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bccbad6b-5994-4a9b-be33-057ac78e5dc5" path="/var/lib/kubelet/pods/bccbad6b-5994-4a9b-be33-057ac78e5dc5/volumes" Sep 08 15:31:02.431202 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:02.431178 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj"] Sep 08 15:31:02.432771 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:31:02.432748 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8901ef_372b_4fef_a6f1_92102ef628bd.slice/crio-54fc781e1c8e5a40f12a19752e833a7a96568f4a31dd5ed3c3f71649b58a3765 WatchSource:0}: Error finding container 54fc781e1c8e5a40f12a19752e833a7a96568f4a31dd5ed3c3f71649b58a3765: Status 404 returned error can't find the container with id 54fc781e1c8e5a40f12a19752e833a7a96568f4a31dd5ed3c3f71649b58a3765 Sep 08 15:31:02.651600 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:02.651524 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" event={"ID":"de86dadc-03ed-4180-ac3c-85216121be05","Type":"ContainerStarted","Data":"ca1fd199dfb07afdc01a896b025db8ca4e992da5887043c3b84619221bf66651"} Sep 08 15:31:02.651600 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:02.651594 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" Sep 08 15:31:02.651812 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:02.651621 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" event={"ID":"de86dadc-03ed-4180-ac3c-85216121be05","Type":"ContainerStarted","Data":"b74d2aa04d7566f541db0e118cbb4ff58562c84f4e192783a199185ebbb2f69a"} Sep 08 15:31:02.653253 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:02.653229 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj" event={"ID":"6b8901ef-372b-4fef-a6f1-92102ef628bd","Type":"ContainerStarted","Data":"06d073421158ef6607bd047a1349043713bdd51429d754dde60ab1821c34d39a"} Sep 08 15:31:02.653370 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:02.653269 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj" event={"ID":"6b8901ef-372b-4fef-a6f1-92102ef628bd","Type":"ContainerStarted","Data":"54fc781e1c8e5a40f12a19752e833a7a96568f4a31dd5ed3c3f71649b58a3765"} Sep 08 15:31:02.653370 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:02.653307 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj" Sep 08 15:31:02.677948 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:02.677841 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" podStartSLOduration=2.172392579 podStartE2EDuration="2.677822748s" totalImagesPullingTime="505.430169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:31:02.677467913 +0000 UTC m=+1324.829646493" watchObservedRunningTime="2026-09-08 15:31:02.677822748 +0000 UTC m=+1324.830001328" Sep 08 15:31:02.712913 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:02.712868 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj" podStartSLOduration=1.712854406 podStartE2EDuration="1.712854406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:31:02.711398117 +0000 UTC m=+1324.863576698" watchObservedRunningTime="2026-09-08 15:31:02.712854406 +0000 UTC m=+1324.865032985" Sep 08 15:31:05.615326 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:05.615297 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t" Sep 08 15:31:05.615685 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:05.615372 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc" Sep 08 15:31:08.663230 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:08.663201 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" Sep 08 15:31:13.661058 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:13.661025 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj" Sep 08 15:31:25.055066 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.054992 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr"] Sep 08 15:31:25.059501 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.059473 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.062769 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.062743 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-bc9cdf-openshift-default-dockercfg-jz989\"" Sep 08 15:31:25.083167 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.083141 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr"] Sep 08 15:31:25.115808 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.115777 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-workload-socket\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.116000 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.115807 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-data\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.116155 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.116128 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-envoy\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.122122 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.122089 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-credential-socket\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.122285 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.122147 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-token\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.122285 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.122269 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-workload-certs\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.122422 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.122299 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-podinfo\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.122422 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.122323 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istiod-ca-cert\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.122422 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.122403 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm7p7\" (UniqueName: \"kubernetes.io/projected/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-kube-api-access-qm7p7\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.223812 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.223776 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-credential-socket\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.223812 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.223813 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-token\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.224044 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.223831 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-workload-certs\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.224044 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.223848 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-podinfo\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.224044 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.223862 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istiod-ca-cert\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.224044 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.223893 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm7p7\" (UniqueName: \"kubernetes.io/projected/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-kube-api-access-qm7p7\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.224044 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.223961 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-workload-socket\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.224044 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.223982 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-data\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.224044 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.224024 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-envoy\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.224410 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.224216 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-credential-socket\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.224410 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.224373 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-workload-certs\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.224569 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.224376 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-workload-socket\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.224569 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.224480 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-data\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.224696 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.224674 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istiod-ca-cert\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.226344 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.226323 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-envoy\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.226578 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.226559 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-podinfo\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.233806 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.233787 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-token\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.234072 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.234052 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm7p7\" (UniqueName: \"kubernetes.io/projected/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-kube-api-access-qm7p7\") pod \"e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.370250 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.370189 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:25.660470 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.660448 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr"] Sep 08 15:31:25.662287 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:31:25.662259 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b5ed25_8f05_440b_a8d2_67f56fb4b0f2.slice/crio-2a83cdd1930aea5efe871512c56e89dfe22b4fe55c595122e79fa01f5c85cbe7 WatchSource:0}: Error finding container 2a83cdd1930aea5efe871512c56e89dfe22b4fe55c595122e79fa01f5c85cbe7: Status 404 returned error can't find the container with id 2a83cdd1930aea5efe871512c56e89dfe22b4fe55c595122e79fa01f5c85cbe7 Sep 08 15:31:25.664418 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.664385 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:31:25.664505 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.664445 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:31:25.664505 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.664477 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:31:25.768300 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.768260 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" event={"ID":"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2","Type":"ContainerStarted","Data":"02f0203e33ed058fefe97a2faae4578472ac4ec131b7782da790cdfe3f8c59d2"} Sep 08 15:31:25.768462 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.768325 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" event={"ID":"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2","Type":"ContainerStarted","Data":"2a83cdd1930aea5efe871512c56e89dfe22b4fe55c595122e79fa01f5c85cbe7"} Sep 08 15:31:25.802894 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:25.802834 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" podStartSLOduration=0.802813941 podStartE2EDuration="802.813941ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:31:25.799723031 +0000 UTC m=+1347.951901609" watchObservedRunningTime="2026-09-08 15:31:25.802813941 +0000 UTC m=+1347.954992525" Sep 08 15:31:26.370912 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:26.370884 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:26.375358 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:26.375334 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:26.772455 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:26.772429 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:26.773088 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:26.773072 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:39.527146 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.527112 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t"] Sep 08 15:31:39.527801 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.527315 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t" podUID="0b5fa604-6b2c-482a-8d11-88480dc58178" containerName="payload-processing" containerID="cri-o://6f446f4e9206e53fc3432880d3f280fc0ca3afc471644af674e5c5f7cb54a9e7" gracePeriod=30 Sep 08 15:31:39.550938 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.550279 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc"] Sep 08 15:31:39.550938 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.550533 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc" podUID="29dfdcf5-c875-489e-9a2d-f690c60259fe" containerName="payload-pre-processing" containerID="cri-o://5ab2312fecc3c97a78afb1bd5e70c162f8dc99cb806bd9a3e3cc2df6ee066b54" gracePeriod=30 Sep 08 15:31:39.832477 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.832441 2572 generic.go:356] "Generic (PLEG): container finished" podID="29dfdcf5-c875-489e-9a2d-f690c60259fe" containerID="5ab2312fecc3c97a78afb1bd5e70c162f8dc99cb806bd9a3e3cc2df6ee066b54" exitCode=0 Sep 08 15:31:39.832615 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.832530 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc" event={"ID":"29dfdcf5-c875-489e-9a2d-f690c60259fe","Type":"ContainerDied","Data":"5ab2312fecc3c97a78afb1bd5e70c162f8dc99cb806bd9a3e3cc2df6ee066b54"} Sep 08 15:31:39.832615 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.832595 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc" event={"ID":"29dfdcf5-c875-489e-9a2d-f690c60259fe","Type":"ContainerDied","Data":"5de154e39b2642ec971c775301fa4f8356fe01aedee3ceee8cb7a1ae8cd372fe"} Sep 08 15:31:39.832728 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.832620 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5de154e39b2642ec971c775301fa4f8356fe01aedee3ceee8cb7a1ae8cd372fe" Sep 08 15:31:39.860138 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.860091 2572 generic.go:356] "Generic (PLEG): container finished" podID="0b5fa604-6b2c-482a-8d11-88480dc58178" containerID="6f446f4e9206e53fc3432880d3f280fc0ca3afc471644af674e5c5f7cb54a9e7" exitCode=0 Sep 08 15:31:39.860301 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.860180 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t" event={"ID":"0b5fa604-6b2c-482a-8d11-88480dc58178","Type":"ContainerDied","Data":"6f446f4e9206e53fc3432880d3f280fc0ca3afc471644af674e5c5f7cb54a9e7"} Sep 08 15:31:39.860301 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.860241 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t" event={"ID":"0b5fa604-6b2c-482a-8d11-88480dc58178","Type":"ContainerDied","Data":"731ffe0f11189dd8e311f327d3457e97b5726331c1459f931a55db88940bb042"} Sep 08 15:31:39.860301 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.860268 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="731ffe0f11189dd8e311f327d3457e97b5726331c1459f931a55db88940bb042" Sep 08 15:31:39.862366 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.862343 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t" Sep 08 15:31:39.864409 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.864394 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc" Sep 08 15:31:39.960582 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.960224 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nh8g\" (UniqueName: \"kubernetes.io/projected/29dfdcf5-c875-489e-9a2d-f690c60259fe-kube-api-access-5nh8g\") pod \"29dfdcf5-c875-489e-9a2d-f690c60259fe\" (UID: \"29dfdcf5-c875-489e-9a2d-f690c60259fe\") " Sep 08 15:31:39.960582 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.960273 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pstbk\" (UniqueName: \"kubernetes.io/projected/0b5fa604-6b2c-482a-8d11-88480dc58178-kube-api-access-pstbk\") pod \"0b5fa604-6b2c-482a-8d11-88480dc58178\" (UID: \"0b5fa604-6b2c-482a-8d11-88480dc58178\") " Sep 08 15:31:39.960582 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.960341 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29dfdcf5-c875-489e-9a2d-f690c60259fe-plugins-config-volume\") pod \"29dfdcf5-c875-489e-9a2d-f690c60259fe\" (UID: \"29dfdcf5-c875-489e-9a2d-f690c60259fe\") " Sep 08 15:31:39.960582 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.960375 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5fa604-6b2c-482a-8d11-88480dc58178-plugins-config-volume\") pod \"0b5fa604-6b2c-482a-8d11-88480dc58178\" (UID: \"0b5fa604-6b2c-482a-8d11-88480dc58178\") " Sep 08 15:31:39.960951 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.960879 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5fa604-6b2c-482a-8d11-88480dc58178-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b5fa604-6b2c-482a-8d11-88480dc58178" (UID: "0b5fa604-6b2c-482a-8d11-88480dc58178"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:31:39.961219 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.961197 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5fa604-6b2c-482a-8d11-88480dc58178-plugins-config-volume\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:31:39.961737 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.961710 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29dfdcf5-c875-489e-9a2d-f690c60259fe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29dfdcf5-c875-489e-9a2d-f690c60259fe" (UID: "29dfdcf5-c875-489e-9a2d-f690c60259fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:31:39.963684 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.963656 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29dfdcf5-c875-489e-9a2d-f690c60259fe-kube-api-access-5nh8g" (OuterVolumeSpecName: "kube-api-access-5nh8g") pod "29dfdcf5-c875-489e-9a2d-f690c60259fe" (UID: "29dfdcf5-c875-489e-9a2d-f690c60259fe"). InnerVolumeSpecName "kube-api-access-5nh8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:31:39.964359 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:39.964338 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5fa604-6b2c-482a-8d11-88480dc58178-kube-api-access-pstbk" (OuterVolumeSpecName: "kube-api-access-pstbk") pod "0b5fa604-6b2c-482a-8d11-88480dc58178" (UID: "0b5fa604-6b2c-482a-8d11-88480dc58178"). InnerVolumeSpecName "kube-api-access-pstbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:31:40.062614 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:40.062582 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29dfdcf5-c875-489e-9a2d-f690c60259fe-plugins-config-volume\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:31:40.062614 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:40.062608 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nh8g\" (UniqueName: \"kubernetes.io/projected/29dfdcf5-c875-489e-9a2d-f690c60259fe-kube-api-access-5nh8g\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:31:40.062614 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:40.062619 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pstbk\" (UniqueName: \"kubernetes.io/projected/0b5fa604-6b2c-482a-8d11-88480dc58178-kube-api-access-pstbk\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:31:40.864355 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:40.864325 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc" Sep 08 15:31:40.864762 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:40.864325 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t" Sep 08 15:31:40.902258 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:40.902230 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t"] Sep 08 15:31:40.904710 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:40.904684 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-49bfbc48-6f6bcc59fc-sdj7t"] Sep 08 15:31:40.930072 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:40.930035 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc"] Sep 08 15:31:40.948261 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:40.948225 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-49bfbc48-7ff9bb558b-psnnc"] Sep 08 15:31:42.371542 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:42.371487 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b5fa604-6b2c-482a-8d11-88480dc58178" path="/var/lib/kubelet/pods/0b5fa604-6b2c-482a-8d11-88480dc58178/volumes" Sep 08 15:31:42.371910 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:42.371870 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29dfdcf5-c875-489e-9a2d-f690c60259fe" path="/var/lib/kubelet/pods/29dfdcf5-c875-489e-9a2d-f690c60259fe/volumes" Sep 08 15:31:45.887091 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:45.887020 2572 generic.go:356] "Generic (PLEG): container finished" podID="59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" containerID="02f0203e33ed058fefe97a2faae4578472ac4ec131b7782da790cdfe3f8c59d2" exitCode=0 Sep 08 15:31:45.887508 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:45.887096 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" event={"ID":"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2","Type":"ContainerDied","Data":"02f0203e33ed058fefe97a2faae4578472ac4ec131b7782da790cdfe3f8c59d2"} Sep 08 15:31:45.887508 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:45.887432 2572 scope.go:117] "RemoveContainer" containerID="02f0203e33ed058fefe97a2faae4578472ac4ec131b7782da790cdfe3f8c59d2" Sep 08 15:31:45.888442 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:45.888413 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:31:45.888578 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:45.888482 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:31:45.888578 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:45.888540 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:31:46.894030 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:46.893994 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" event={"ID":"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2","Type":"ContainerStarted","Data":"4c8cb1338e20b4c4e8d9b056adc3f07aaa983765cfded6528b7fe98927a1b773"} Sep 08 15:31:47.371437 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:47.371391 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:47.371713 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:47.371687 2572 patch_prober.go:28] interesting pod/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 08 15:31:47.371799 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:47.371750 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" podUID="59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 08 15:31:48.370862 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:48.370832 2572 patch_prober.go:28] interesting pod/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 08 15:31:48.371291 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:48.370883 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" podUID="59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 08 15:31:49.371252 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:49.371213 2572 patch_prober.go:28] interesting pod/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 08 15:31:49.371686 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:49.371289 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" podUID="59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 08 15:31:50.371018 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:50.370984 2572 patch_prober.go:28] interesting pod/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 08 15:31:50.371194 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:50.371038 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" podUID="59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 08 15:31:51.371248 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:51.371210 2572 patch_prober.go:28] interesting pod/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 08 15:31:51.371651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:51.371274 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" podUID="59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 08 15:31:52.374328 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:52.374299 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:52.374750 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:52.374703 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:31:52.375369 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:31:52.375352 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:32:09.975441 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:09.975411 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn"] Sep 08 15:32:09.975853 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:09.975768 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29dfdcf5-c875-489e-9a2d-f690c60259fe" containerName="payload-pre-processing" Sep 08 15:32:09.975853 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:09.975781 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="29dfdcf5-c875-489e-9a2d-f690c60259fe" containerName="payload-pre-processing" Sep 08 15:32:09.975853 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:09.975791 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5fa604-6b2c-482a-8d11-88480dc58178" containerName="payload-processing" Sep 08 15:32:09.975853 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:09.975797 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5fa604-6b2c-482a-8d11-88480dc58178" containerName="payload-processing" Sep 08 15:32:09.975853 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:09.975854 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b5fa604-6b2c-482a-8d11-88480dc58178" containerName="payload-processing" Sep 08 15:32:09.976015 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:09.975863 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="29dfdcf5-c875-489e-9a2d-f690c60259fe" containerName="payload-pre-processing" Sep 08 15:32:09.980035 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:09.980014 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" Sep 08 15:32:09.985299 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:09.985280 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-149fdb-serving-cert\"" Sep 08 15:32:10.002984 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.002954 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn"] Sep 08 15:32:10.017398 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.017371 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5583bd4-423e-412a-8a6c-911422cc02c5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn\" (UID: \"b5583bd4-423e-412a-8a6c-911422cc02c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" Sep 08 15:32:10.017510 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.017433 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkh2w\" (UniqueName: \"kubernetes.io/projected/b5583bd4-423e-412a-8a6c-911422cc02c5-kube-api-access-kkh2w\") pod \"maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn\" (UID: \"b5583bd4-423e-412a-8a6c-911422cc02c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" Sep 08 15:32:10.017556 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.017515 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5583bd4-423e-412a-8a6c-911422cc02c5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn\" (UID: \"b5583bd4-423e-412a-8a6c-911422cc02c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" Sep 08 15:32:10.017597 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.017571 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5583bd4-423e-412a-8a6c-911422cc02c5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn\" (UID: \"b5583bd4-423e-412a-8a6c-911422cc02c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" Sep 08 15:32:10.118750 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.118712 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkh2w\" (UniqueName: \"kubernetes.io/projected/b5583bd4-423e-412a-8a6c-911422cc02c5-kube-api-access-kkh2w\") pod \"maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn\" (UID: \"b5583bd4-423e-412a-8a6c-911422cc02c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" Sep 08 15:32:10.118914 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.118758 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5583bd4-423e-412a-8a6c-911422cc02c5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn\" (UID: \"b5583bd4-423e-412a-8a6c-911422cc02c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" Sep 08 15:32:10.118914 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.118808 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5583bd4-423e-412a-8a6c-911422cc02c5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn\" (UID: \"b5583bd4-423e-412a-8a6c-911422cc02c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" Sep 08 15:32:10.118914 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.118853 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5583bd4-423e-412a-8a6c-911422cc02c5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn\" (UID: \"b5583bd4-423e-412a-8a6c-911422cc02c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" Sep 08 15:32:10.119085 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:32:10.118912 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-serving-cert: secret "maas-api-e2e-shared-a-w2-149fdb-serving-cert" not found Sep 08 15:32:10.119085 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:32:10.118982 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5583bd4-423e-412a-8a6c-911422cc02c5-maas-api-tls podName:b5583bd4-423e-412a-8a6c-911422cc02c5 nodeName:}" failed. No retries permitted until 2026-09-08 15:32:10.618963378 +0000 UTC m=+1392.771141937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b5583bd4-423e-412a-8a6c-911422cc02c5-maas-api-tls") pod "maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" (UID: "b5583bd4-423e-412a-8a6c-911422cc02c5") : secret "maas-api-e2e-shared-a-w2-149fdb-serving-cert" not found Sep 08 15:32:10.119556 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.119534 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5583bd4-423e-412a-8a6c-911422cc02c5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn\" (UID: \"b5583bd4-423e-412a-8a6c-911422cc02c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" Sep 08 15:32:10.121307 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.121288 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5583bd4-423e-412a-8a6c-911422cc02c5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn\" (UID: \"b5583bd4-423e-412a-8a6c-911422cc02c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" Sep 08 15:32:10.128058 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.128038 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkh2w\" (UniqueName: \"kubernetes.io/projected/b5583bd4-423e-412a-8a6c-911422cc02c5-kube-api-access-kkh2w\") pod \"maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn\" (UID: \"b5583bd4-423e-412a-8a6c-911422cc02c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" Sep 08 15:32:10.622619 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.622577 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5583bd4-423e-412a-8a6c-911422cc02c5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn\" (UID: \"b5583bd4-423e-412a-8a6c-911422cc02c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" Sep 08 15:32:10.625197 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.625165 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5583bd4-423e-412a-8a6c-911422cc02c5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn\" (UID: \"b5583bd4-423e-412a-8a6c-911422cc02c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" Sep 08 15:32:10.877154 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.877068 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt"] Sep 08 15:32:10.880478 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.880461 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt" Sep 08 15:32:10.882964 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.882942 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-149fdb-dockercfg-5rtz8\"" Sep 08 15:32:10.883076 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.882941 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-149fdb\"" Sep 08 15:32:10.889585 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.889566 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" Sep 08 15:32:10.891436 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.891416 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt"] Sep 08 15:32:10.925798 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.925761 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dksdd\" (UniqueName: \"kubernetes.io/projected/f80b5111-6cdf-4b32-8f35-81f5c0eab644-kube-api-access-dksdd\") pod \"payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt\" (UID: \"f80b5111-6cdf-4b32-8f35-81f5c0eab644\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt" Sep 08 15:32:10.926013 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:10.925865 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80b5111-6cdf-4b32-8f35-81f5c0eab644-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt\" (UID: \"f80b5111-6cdf-4b32-8f35-81f5c0eab644\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt" Sep 08 15:32:11.027034 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:11.026998 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dksdd\" (UniqueName: \"kubernetes.io/projected/f80b5111-6cdf-4b32-8f35-81f5c0eab644-kube-api-access-dksdd\") pod \"payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt\" (UID: \"f80b5111-6cdf-4b32-8f35-81f5c0eab644\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt" Sep 08 15:32:11.027475 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:11.027074 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80b5111-6cdf-4b32-8f35-81f5c0eab644-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt\" (UID: \"f80b5111-6cdf-4b32-8f35-81f5c0eab644\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt" Sep 08 15:32:11.027935 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:11.027912 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80b5111-6cdf-4b32-8f35-81f5c0eab644-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt\" (UID: \"f80b5111-6cdf-4b32-8f35-81f5c0eab644\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt" Sep 08 15:32:11.037356 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:11.037338 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dksdd\" (UniqueName: \"kubernetes.io/projected/f80b5111-6cdf-4b32-8f35-81f5c0eab644-kube-api-access-dksdd\") pod \"payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt\" (UID: \"f80b5111-6cdf-4b32-8f35-81f5c0eab644\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt" Sep 08 15:32:11.178987 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:11.178960 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn"] Sep 08 15:32:11.180609 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:32:11.180567 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5583bd4_423e_412a_8a6c_911422cc02c5.slice/crio-86256d80bb5ef71219c485e5d12a32e3088ad08cdbac07aa83109eddbc510288 WatchSource:0}: Error finding container 86256d80bb5ef71219c485e5d12a32e3088ad08cdbac07aa83109eddbc510288: Status 404 returned error can't find the container with id 86256d80bb5ef71219c485e5d12a32e3088ad08cdbac07aa83109eddbc510288 Sep 08 15:32:11.191355 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:11.191335 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt" Sep 08 15:32:11.478473 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:11.478398 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt"] Sep 08 15:32:11.482277 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:32:11.482254 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80b5111_6cdf_4b32_8f35_81f5c0eab644.slice/crio-44021ff168cdd3efab223855d8309ca8ea52654b6be3e7136dd851d9f40df42a WatchSource:0}: Error finding container 44021ff168cdd3efab223855d8309ca8ea52654b6be3e7136dd851d9f40df42a: Status 404 returned error can't find the container with id 44021ff168cdd3efab223855d8309ca8ea52654b6be3e7136dd851d9f40df42a Sep 08 15:32:12.013758 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:12.013722 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt" event={"ID":"f80b5111-6cdf-4b32-8f35-81f5c0eab644","Type":"ContainerStarted","Data":"c10446b3337c6202dcf4ed5b60835d1a802a88a4cc9be568853e57e0335298bb"} Sep 08 15:32:12.013948 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:12.013786 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt" event={"ID":"f80b5111-6cdf-4b32-8f35-81f5c0eab644","Type":"ContainerStarted","Data":"44021ff168cdd3efab223855d8309ca8ea52654b6be3e7136dd851d9f40df42a"} Sep 08 15:32:12.013948 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:12.013815 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt" Sep 08 15:32:12.015258 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:12.015232 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" event={"ID":"b5583bd4-423e-412a-8a6c-911422cc02c5","Type":"ContainerStarted","Data":"fc3bf493f164b4fed7f169e15fd249ce7dd9ff5e52bf924988cff87a2afb3e54"} Sep 08 15:32:12.015405 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:12.015270 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" event={"ID":"b5583bd4-423e-412a-8a6c-911422cc02c5","Type":"ContainerStarted","Data":"86256d80bb5ef71219c485e5d12a32e3088ad08cdbac07aa83109eddbc510288"} Sep 08 15:32:12.015405 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:12.015385 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" Sep 08 15:32:12.044939 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:12.044896 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt" podStartSLOduration=2.044882863 podStartE2EDuration="2.044882863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:32:12.044370714 +0000 UTC m=+1394.196549295" watchObservedRunningTime="2026-09-08 15:32:12.044882863 +0000 UTC m=+1394.197061442" Sep 08 15:32:12.101383 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:12.101338 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" podStartSLOduration=2.61760311 podStartE2EDuration="3.101324782s" totalImagesPullingTime="483.721672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:32:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:32:12.100568736 +0000 UTC m=+1394.252747316" watchObservedRunningTime="2026-09-08 15:32:12.101324782 +0000 UTC m=+1394.253503361" Sep 08 15:32:18.025358 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:18.025328 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" Sep 08 15:32:23.022139 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:23.022110 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt" Sep 08 15:32:34.528670 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.528640 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d"] Sep 08 15:32:34.531856 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.531833 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.534037 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.534014 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-04285b-openshift-default-dockercfg-nm989\"" Sep 08 15:32:34.547956 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.547930 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d"] Sep 08 15:32:34.626195 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.626159 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-istio-data\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.626389 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.626217 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e53b91-bb2d-44c9-ab1e-514170090912-istio-token\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.626389 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.626305 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-workload-certs\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.626389 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.626344 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-credential-socket\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.626389 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.626376 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e53b91-bb2d-44c9-ab1e-514170090912-istio-podinfo\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.626651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.626421 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln7r2\" (UniqueName: \"kubernetes.io/projected/40e53b91-bb2d-44c9-ab1e-514170090912-kube-api-access-ln7r2\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.626651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.626465 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-workload-socket\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.626651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.626538 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e53b91-bb2d-44c9-ab1e-514170090912-istiod-ca-cert\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.626651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.626578 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-istio-envoy\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.727681 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.727646 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-credential-socket\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.727848 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.727690 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e53b91-bb2d-44c9-ab1e-514170090912-istio-podinfo\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.727848 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.727709 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln7r2\" (UniqueName: \"kubernetes.io/projected/40e53b91-bb2d-44c9-ab1e-514170090912-kube-api-access-ln7r2\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.727848 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.727725 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-workload-socket\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.727848 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.727744 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e53b91-bb2d-44c9-ab1e-514170090912-istiod-ca-cert\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.727848 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.727767 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-istio-envoy\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.727848 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.727796 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-istio-data\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.727848 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.727836 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e53b91-bb2d-44c9-ab1e-514170090912-istio-token\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.728441 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.727895 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-workload-certs\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.728441 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.728098 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-credential-socket\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.728441 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.728178 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-istio-data\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.728441 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.728237 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-workload-certs\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.728441 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.728372 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-workload-socket\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.728640 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.728476 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e53b91-bb2d-44c9-ab1e-514170090912-istiod-ca-cert\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.730219 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.730194 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-istio-envoy\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.730312 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.730295 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e53b91-bb2d-44c9-ab1e-514170090912-istio-podinfo\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.738133 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.738110 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e53b91-bb2d-44c9-ab1e-514170090912-istio-token\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.738420 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.738399 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln7r2\" (UniqueName: \"kubernetes.io/projected/40e53b91-bb2d-44c9-ab1e-514170090912-kube-api-access-ln7r2\") pod \"e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:34.841850 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:34.841791 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:35.125871 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:35.125844 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d"] Sep 08 15:32:35.127534 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:32:35.127483 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e53b91_bb2d_44c9_ab1e_514170090912.slice/crio-82386de768c1db96150d1944ad8c43dd77bc31f3d2da089942c45db6b95dafba WatchSource:0}: Error finding container 82386de768c1db96150d1944ad8c43dd77bc31f3d2da089942c45db6b95dafba: Status 404 returned error can't find the container with id 82386de768c1db96150d1944ad8c43dd77bc31f3d2da089942c45db6b95dafba Sep 08 15:32:35.129700 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:35.129668 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:32:35.129795 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:35.129736 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:32:35.129795 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:35.129767 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:32:36.137594 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:36.137559 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" event={"ID":"40e53b91-bb2d-44c9-ab1e-514170090912","Type":"ContainerStarted","Data":"cc08807784904595a8970e7f0255d51f3c76973aee7c0ec22520065640159bf2"} Sep 08 15:32:36.138056 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:36.137630 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" event={"ID":"40e53b91-bb2d-44c9-ab1e-514170090912","Type":"ContainerStarted","Data":"82386de768c1db96150d1944ad8c43dd77bc31f3d2da089942c45db6b95dafba"} Sep 08 15:32:36.161685 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:36.161639 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" podStartSLOduration=2.16162275 podStartE2EDuration="2.16162275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:32:36.159768278 +0000 UTC m=+1418.311946857" watchObservedRunningTime="2026-09-08 15:32:36.16162275 +0000 UTC m=+1418.313801329" Sep 08 15:32:36.842918 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:36.842887 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:36.848374 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:36.848348 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:37.142554 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:37.142463 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:37.143558 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:37.143540 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:40.247837 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:40.247799 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk"] Sep 08 15:32:40.251151 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:40.251133 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk" Sep 08 15:32:40.254426 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:40.254404 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-04285b-dockercfg-8mktk\"" Sep 08 15:32:40.254678 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:40.254664 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-04285b\"" Sep 08 15:32:40.266754 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:40.266731 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk"] Sep 08 15:32:40.381802 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:40.381766 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcwsd\" (UniqueName: \"kubernetes.io/projected/a4a58b42-1660-4853-987e-c354a0887598-kube-api-access-qcwsd\") pod \"payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk\" (UID: \"a4a58b42-1660-4853-987e-c354a0887598\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk" Sep 08 15:32:40.381987 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:40.381828 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a58b42-1660-4853-987e-c354a0887598-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk\" (UID: \"a4a58b42-1660-4853-987e-c354a0887598\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk" Sep 08 15:32:40.482799 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:40.482766 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcwsd\" (UniqueName: \"kubernetes.io/projected/a4a58b42-1660-4853-987e-c354a0887598-kube-api-access-qcwsd\") pod \"payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk\" (UID: \"a4a58b42-1660-4853-987e-c354a0887598\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk" Sep 08 15:32:40.482974 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:40.482818 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a58b42-1660-4853-987e-c354a0887598-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk\" (UID: \"a4a58b42-1660-4853-987e-c354a0887598\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk" Sep 08 15:32:40.483481 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:40.483462 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a58b42-1660-4853-987e-c354a0887598-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk\" (UID: \"a4a58b42-1660-4853-987e-c354a0887598\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk" Sep 08 15:32:40.496982 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:40.496950 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcwsd\" (UniqueName: \"kubernetes.io/projected/a4a58b42-1660-4853-987e-c354a0887598-kube-api-access-qcwsd\") pod \"payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk\" (UID: \"a4a58b42-1660-4853-987e-c354a0887598\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk" Sep 08 15:32:40.561409 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:40.561323 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk" Sep 08 15:32:40.844734 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:40.844704 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk"] Sep 08 15:32:40.845968 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:32:40.845933 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a58b42_1660_4853_987e_c354a0887598.slice/crio-011173a2a1954771bf6e94b73ce4c30b5c940e15db1c298a19f86244b9128b0c WatchSource:0}: Error finding container 011173a2a1954771bf6e94b73ce4c30b5c940e15db1c298a19f86244b9128b0c: Status 404 returned error can't find the container with id 011173a2a1954771bf6e94b73ce4c30b5c940e15db1c298a19f86244b9128b0c Sep 08 15:32:41.161482 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:41.161383 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk" event={"ID":"a4a58b42-1660-4853-987e-c354a0887598","Type":"ContainerStarted","Data":"c6bcec7ab5d65f2951bc88d44ad06edd0cc217309a0946f5cce9afb69b279fd2"} Sep 08 15:32:41.161482 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:41.161441 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk" event={"ID":"a4a58b42-1660-4853-987e-c354a0887598","Type":"ContainerStarted","Data":"011173a2a1954771bf6e94b73ce4c30b5c940e15db1c298a19f86244b9128b0c"} Sep 08 15:32:41.161482 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:41.161473 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk" Sep 08 15:32:41.181227 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:41.181176 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk" podStartSLOduration=1.181160861 podStartE2EDuration="1.181160861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:32:41.179976861 +0000 UTC m=+1423.332155444" watchObservedRunningTime="2026-09-08 15:32:41.181160861 +0000 UTC m=+1423.333339440" Sep 08 15:32:52.167821 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:52.167738 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk" Sep 08 15:32:58.244559 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:58.244524 2572 generic.go:356] "Generic (PLEG): container finished" podID="40e53b91-bb2d-44c9-ab1e-514170090912" containerID="cc08807784904595a8970e7f0255d51f3c76973aee7c0ec22520065640159bf2" exitCode=0 Sep 08 15:32:58.244956 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:58.244599 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" event={"ID":"40e53b91-bb2d-44c9-ab1e-514170090912","Type":"ContainerDied","Data":"cc08807784904595a8970e7f0255d51f3c76973aee7c0ec22520065640159bf2"} Sep 08 15:32:58.244999 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:58.244978 2572 scope.go:117] "RemoveContainer" containerID="cc08807784904595a8970e7f0255d51f3c76973aee7c0ec22520065640159bf2" Sep 08 15:32:58.246122 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:58.246093 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:32:58.246185 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:58.246166 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:32:58.246222 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:58.246204 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:32:59.255450 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:59.255399 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" event={"ID":"40e53b91-bb2d-44c9-ab1e-514170090912","Type":"ContainerStarted","Data":"98bb40d7290b656d919e1c49fc21c9dcfac84086c3624f50b78f3fe081e0218d"} Sep 08 15:32:59.843237 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:59.842782 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:32:59.843237 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:59.842900 2572 patch_prober.go:28] interesting pod/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 08 15:32:59.843237 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:32:59.842944 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" podUID="40e53b91-bb2d-44c9-ab1e-514170090912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 08 15:33:00.842616 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:00.842572 2572 patch_prober.go:28] interesting pod/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 08 15:33:00.843128 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:00.842642 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" podUID="40e53b91-bb2d-44c9-ab1e-514170090912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 08 15:33:01.843002 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:01.842965 2572 patch_prober.go:28] interesting pod/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 08 15:33:01.843618 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:01.843585 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" podUID="40e53b91-bb2d-44c9-ab1e-514170090912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 08 15:33:02.842549 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:02.842514 2572 patch_prober.go:28] interesting pod/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 08 15:33:02.842870 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:02.842832 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" podUID="40e53b91-bb2d-44c9-ab1e-514170090912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 08 15:33:03.843208 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:03.843172 2572 patch_prober.go:28] interesting pod/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 08 15:33:03.843817 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:03.843786 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" podUID="40e53b91-bb2d-44c9-ab1e-514170090912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 08 15:33:04.846952 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:04.846921 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:33:04.847384 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:04.847190 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:33:04.848341 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:04.848317 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:33:23.083971 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:23.083937 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4"] Sep 08 15:33:23.088478 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:23.088459 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4" Sep 08 15:33:23.092303 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:23.092282 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ae34c54a\"" Sep 08 15:33:23.099533 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:23.099514 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ae34c54a-dockercfg-mdqh4\"" Sep 08 15:33:23.131418 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:23.131317 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4"] Sep 08 15:33:23.140017 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:23.139985 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb54k\" (UniqueName: \"kubernetes.io/projected/37b9fe87-dad9-40d9-8ece-50ae2bd9d208-kube-api-access-jb54k\") pod \"payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4\" (UID: \"37b9fe87-dad9-40d9-8ece-50ae2bd9d208\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4" Sep 08 15:33:23.140167 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:23.140132 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b9fe87-dad9-40d9-8ece-50ae2bd9d208-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4\" (UID: \"37b9fe87-dad9-40d9-8ece-50ae2bd9d208\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4" Sep 08 15:33:23.240975 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:23.240937 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b9fe87-dad9-40d9-8ece-50ae2bd9d208-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4\" (UID: \"37b9fe87-dad9-40d9-8ece-50ae2bd9d208\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4" Sep 08 15:33:23.241265 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:23.241015 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb54k\" (UniqueName: \"kubernetes.io/projected/37b9fe87-dad9-40d9-8ece-50ae2bd9d208-kube-api-access-jb54k\") pod \"payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4\" (UID: \"37b9fe87-dad9-40d9-8ece-50ae2bd9d208\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4" Sep 08 15:33:23.241798 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:23.241772 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b9fe87-dad9-40d9-8ece-50ae2bd9d208-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4\" (UID: \"37b9fe87-dad9-40d9-8ece-50ae2bd9d208\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4" Sep 08 15:33:23.253896 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:23.253872 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb54k\" (UniqueName: \"kubernetes.io/projected/37b9fe87-dad9-40d9-8ece-50ae2bd9d208-kube-api-access-jb54k\") pod \"payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4\" (UID: \"37b9fe87-dad9-40d9-8ece-50ae2bd9d208\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4" Sep 08 15:33:23.398476 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:23.398407 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4" Sep 08 15:33:23.673629 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:23.673606 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4"] Sep 08 15:33:23.676363 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:33:23.676306 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b9fe87_dad9_40d9_8ece_50ae2bd9d208.slice/crio-a483b432e451a5bdadad1a255b17bf8411b56de428d1aa008371990cddd9c21c WatchSource:0}: Error finding container a483b432e451a5bdadad1a255b17bf8411b56de428d1aa008371990cddd9c21c: Status 404 returned error can't find the container with id a483b432e451a5bdadad1a255b17bf8411b56de428d1aa008371990cddd9c21c Sep 08 15:33:24.383508 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:24.383454 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4" event={"ID":"37b9fe87-dad9-40d9-8ece-50ae2bd9d208","Type":"ContainerStarted","Data":"8143861e9715c9f695c4a8ceeb41ecfd7389564f17026b7ccae0351cc8545a1c"} Sep 08 15:33:24.383936 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:24.383539 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4" event={"ID":"37b9fe87-dad9-40d9-8ece-50ae2bd9d208","Type":"ContainerStarted","Data":"a483b432e451a5bdadad1a255b17bf8411b56de428d1aa008371990cddd9c21c"} Sep 08 15:33:24.383936 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:24.383562 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4" Sep 08 15:33:24.404374 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:24.404308 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4" podStartSLOduration=1.404291668 podStartE2EDuration="1.404291668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:33:24.403650582 +0000 UTC m=+1466.555829162" watchObservedRunningTime="2026-09-08 15:33:24.404291668 +0000 UTC m=+1466.556470248" Sep 08 15:33:35.390446 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:35.390416 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4" Sep 08 15:33:58.430790 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:58.430760 2572 scope.go:117] "RemoveContainer" containerID="acc541eb906b997abb315ece906b96c387de39760f4a0168acdb30e6cb7b9466" Sep 08 15:33:58.438685 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:33:58.438667 2572 scope.go:117] "RemoveContainer" containerID="d9be5bcd40dc6a66f242207da8333b9b49c369ea81f44bbcb7c51346b1169972" Sep 08 15:34:58.463596 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:34:58.458996 2572 scope.go:117] "RemoveContainer" containerID="ec5278888f8ae16a482c37fd7f4237a651b9baa2745dda2cd7c9c9d3ab890561" Sep 08 15:34:58.471470 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:34:58.471449 2572 scope.go:117] "RemoveContainer" containerID="4a9f79e55b4cdf8a3ed1031bfe848a1767693061173451f72fbe2714a7fe9b82" Sep 08 15:35:12.564648 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:35:12.564618 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4"] Sep 08 15:35:12.565104 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:35:12.564818 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4" podUID="37b9fe87-dad9-40d9-8ece-50ae2bd9d208" containerName="payload-pre-processing" containerID="cri-o://8143861e9715c9f695c4a8ceeb41ecfd7389564f17026b7ccae0351cc8545a1c" gracePeriod=30 Sep 08 15:35:12.817967 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:35:12.817913 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4" Sep 08 15:35:12.904095 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:35:12.904065 2572 generic.go:356] "Generic (PLEG): container finished" podID="37b9fe87-dad9-40d9-8ece-50ae2bd9d208" containerID="8143861e9715c9f695c4a8ceeb41ecfd7389564f17026b7ccae0351cc8545a1c" exitCode=0 Sep 08 15:35:12.904231 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:35:12.904108 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4" Sep 08 15:35:12.904231 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:35:12.904117 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4" event={"ID":"37b9fe87-dad9-40d9-8ece-50ae2bd9d208","Type":"ContainerDied","Data":"8143861e9715c9f695c4a8ceeb41ecfd7389564f17026b7ccae0351cc8545a1c"} Sep 08 15:35:12.904231 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:35:12.904156 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4" event={"ID":"37b9fe87-dad9-40d9-8ece-50ae2bd9d208","Type":"ContainerDied","Data":"a483b432e451a5bdadad1a255b17bf8411b56de428d1aa008371990cddd9c21c"} Sep 08 15:35:12.904231 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:35:12.904176 2572 scope.go:117] "RemoveContainer" containerID="8143861e9715c9f695c4a8ceeb41ecfd7389564f17026b7ccae0351cc8545a1c" Sep 08 15:35:12.920988 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:35:12.920853 2572 scope.go:117] "RemoveContainer" containerID="8143861e9715c9f695c4a8ceeb41ecfd7389564f17026b7ccae0351cc8545a1c" Sep 08 15:35:12.921668 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:35:12.921602 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8143861e9715c9f695c4a8ceeb41ecfd7389564f17026b7ccae0351cc8545a1c\": container with ID starting with 8143861e9715c9f695c4a8ceeb41ecfd7389564f17026b7ccae0351cc8545a1c not found: ID does not exist" containerID="8143861e9715c9f695c4a8ceeb41ecfd7389564f17026b7ccae0351cc8545a1c" Sep 08 15:35:12.921668 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:35:12.921639 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8143861e9715c9f695c4a8ceeb41ecfd7389564f17026b7ccae0351cc8545a1c"} err="failed to get container status \"8143861e9715c9f695c4a8ceeb41ecfd7389564f17026b7ccae0351cc8545a1c\": rpc error: code = NotFound desc = could not find container \"8143861e9715c9f695c4a8ceeb41ecfd7389564f17026b7ccae0351cc8545a1c\": container with ID starting with 8143861e9715c9f695c4a8ceeb41ecfd7389564f17026b7ccae0351cc8545a1c not found: ID does not exist" Sep 08 15:35:12.948939 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:35:12.947868 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b9fe87-dad9-40d9-8ece-50ae2bd9d208-plugins-config-volume\") pod \"37b9fe87-dad9-40d9-8ece-50ae2bd9d208\" (UID: \"37b9fe87-dad9-40d9-8ece-50ae2bd9d208\") " Sep 08 15:35:12.948939 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:35:12.947990 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jb54k\" (UniqueName: \"kubernetes.io/projected/37b9fe87-dad9-40d9-8ece-50ae2bd9d208-kube-api-access-jb54k\") pod \"37b9fe87-dad9-40d9-8ece-50ae2bd9d208\" (UID: \"37b9fe87-dad9-40d9-8ece-50ae2bd9d208\") " Sep 08 15:35:12.948939 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:35:12.948851 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b9fe87-dad9-40d9-8ece-50ae2bd9d208-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "37b9fe87-dad9-40d9-8ece-50ae2bd9d208" (UID: "37b9fe87-dad9-40d9-8ece-50ae2bd9d208"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:35:12.950937 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:35:12.950910 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b9fe87-dad9-40d9-8ece-50ae2bd9d208-kube-api-access-jb54k" (OuterVolumeSpecName: "kube-api-access-jb54k") pod "37b9fe87-dad9-40d9-8ece-50ae2bd9d208" (UID: "37b9fe87-dad9-40d9-8ece-50ae2bd9d208"). InnerVolumeSpecName "kube-api-access-jb54k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:35:13.049195 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:35:13.049170 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb54k\" (UniqueName: \"kubernetes.io/projected/37b9fe87-dad9-40d9-8ece-50ae2bd9d208-kube-api-access-jb54k\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:35:13.049195 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:35:13.049192 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b9fe87-dad9-40d9-8ece-50ae2bd9d208-plugins-config-volume\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:35:13.230457 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:35:13.226440 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4"] Sep 08 15:35:13.236082 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:35:13.236061 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae34c54a-68788d4f9d-xgrc4"] Sep 08 15:35:14.371784 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:35:14.371750 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37b9fe87-dad9-40d9-8ece-50ae2bd9d208" path="/var/lib/kubelet/pods/37b9fe87-dad9-40d9-8ece-50ae2bd9d208/volumes" Sep 08 15:36:58.502605 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:36:58.502577 2572 scope.go:117] "RemoveContainer" containerID="6f446f4e9206e53fc3432880d3f280fc0ca3afc471644af674e5c5f7cb54a9e7" Sep 08 15:36:58.510576 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:36:58.510561 2572 scope.go:117] "RemoveContainer" containerID="5ab2312fecc3c97a78afb1bd5e70c162f8dc99cb806bd9a3e3cc2df6ee066b54" Sep 08 15:37:18.580661 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:18.580625 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5"] Sep 08 15:37:18.581619 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:18.581593 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b9fe87-dad9-40d9-8ece-50ae2bd9d208" containerName="payload-pre-processing" Sep 08 15:37:18.581619 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:18.581619 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b9fe87-dad9-40d9-8ece-50ae2bd9d208" containerName="payload-pre-processing" Sep 08 15:37:18.581818 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:18.581715 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b9fe87-dad9-40d9-8ece-50ae2bd9d208" containerName="payload-pre-processing" Sep 08 15:37:18.584873 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:18.584803 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" Sep 08 15:37:18.587137 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:18.587107 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4a2c80-serving-cert\"" Sep 08 15:37:18.596358 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:18.596325 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5"] Sep 08 15:37:18.666292 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:18.666258 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cda14550-2475-4c54-8f87-cba74f09a027-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5\" (UID: \"cda14550-2475-4c54-8f87-cba74f09a027\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" Sep 08 15:37:18.666461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:18.666303 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7qk5\" (UniqueName: \"kubernetes.io/projected/cda14550-2475-4c54-8f87-cba74f09a027-kube-api-access-g7qk5\") pod \"maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5\" (UID: \"cda14550-2475-4c54-8f87-cba74f09a027\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" Sep 08 15:37:18.666461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:18.666394 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cda14550-2475-4c54-8f87-cba74f09a027-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5\" (UID: \"cda14550-2475-4c54-8f87-cba74f09a027\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" Sep 08 15:37:18.666461 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:18.666430 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cda14550-2475-4c54-8f87-cba74f09a027-metrics-certs\") pod \"maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5\" (UID: \"cda14550-2475-4c54-8f87-cba74f09a027\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" Sep 08 15:37:18.767678 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:18.767642 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cda14550-2475-4c54-8f87-cba74f09a027-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5\" (UID: \"cda14550-2475-4c54-8f87-cba74f09a027\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" Sep 08 15:37:18.767678 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:18.767674 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7qk5\" (UniqueName: \"kubernetes.io/projected/cda14550-2475-4c54-8f87-cba74f09a027-kube-api-access-g7qk5\") pod \"maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5\" (UID: \"cda14550-2475-4c54-8f87-cba74f09a027\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" Sep 08 15:37:18.767957 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:18.767935 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cda14550-2475-4c54-8f87-cba74f09a027-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5\" (UID: \"cda14550-2475-4c54-8f87-cba74f09a027\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" Sep 08 15:37:18.768024 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:18.767981 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cda14550-2475-4c54-8f87-cba74f09a027-metrics-certs\") pod \"maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5\" (UID: \"cda14550-2475-4c54-8f87-cba74f09a027\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" Sep 08 15:37:18.768084 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:37:18.768072 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-serving-cert: secret "maas-api-e2e-ipp-a-4a2c80-serving-cert" not found Sep 08 15:37:18.768146 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:37:18.768138 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cda14550-2475-4c54-8f87-cba74f09a027-maas-api-tls podName:cda14550-2475-4c54-8f87-cba74f09a027 nodeName:}" failed. No retries permitted until 2026-09-08 15:37:19.26812377 +0000 UTC m=+1701.420302328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cda14550-2475-4c54-8f87-cba74f09a027-maas-api-tls") pod "maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" (UID: "cda14550-2475-4c54-8f87-cba74f09a027") : secret "maas-api-e2e-ipp-a-4a2c80-serving-cert" not found Sep 08 15:37:18.768407 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:18.768385 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cda14550-2475-4c54-8f87-cba74f09a027-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5\" (UID: \"cda14550-2475-4c54-8f87-cba74f09a027\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" Sep 08 15:37:18.770436 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:18.770418 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cda14550-2475-4c54-8f87-cba74f09a027-metrics-certs\") pod \"maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5\" (UID: \"cda14550-2475-4c54-8f87-cba74f09a027\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" Sep 08 15:37:18.775943 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:18.775925 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7qk5\" (UniqueName: \"kubernetes.io/projected/cda14550-2475-4c54-8f87-cba74f09a027-kube-api-access-g7qk5\") pod \"maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5\" (UID: \"cda14550-2475-4c54-8f87-cba74f09a027\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" Sep 08 15:37:19.272246 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:19.272202 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cda14550-2475-4c54-8f87-cba74f09a027-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5\" (UID: \"cda14550-2475-4c54-8f87-cba74f09a027\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" Sep 08 15:37:19.274799 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:19.274774 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cda14550-2475-4c54-8f87-cba74f09a027-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5\" (UID: \"cda14550-2475-4c54-8f87-cba74f09a027\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" Sep 08 15:37:19.436844 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:19.436810 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb"] Sep 08 15:37:19.440240 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:19.440224 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb" Sep 08 15:37:19.450577 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:19.450557 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4a2c80-dockercfg-jlm6b\"" Sep 08 15:37:19.455484 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:19.455464 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4a2c80\"" Sep 08 15:37:19.473648 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:19.473623 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb"] Sep 08 15:37:19.495277 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:19.495249 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" Sep 08 15:37:19.575778 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:19.575195 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ebed04-c936-4a96-8585-8a6885549752-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb\" (UID: \"c9ebed04-c936-4a96-8585-8a6885549752\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb" Sep 08 15:37:19.575778 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:19.575253 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-646nj\" (UniqueName: \"kubernetes.io/projected/c9ebed04-c936-4a96-8585-8a6885549752-kube-api-access-646nj\") pod \"payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb\" (UID: \"c9ebed04-c936-4a96-8585-8a6885549752\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb" Sep 08 15:37:19.676682 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:19.676645 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ebed04-c936-4a96-8585-8a6885549752-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb\" (UID: \"c9ebed04-c936-4a96-8585-8a6885549752\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb" Sep 08 15:37:19.677061 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:19.676694 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-646nj\" (UniqueName: \"kubernetes.io/projected/c9ebed04-c936-4a96-8585-8a6885549752-kube-api-access-646nj\") pod \"payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb\" (UID: \"c9ebed04-c936-4a96-8585-8a6885549752\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb" Sep 08 15:37:19.677538 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:19.677516 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ebed04-c936-4a96-8585-8a6885549752-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb\" (UID: \"c9ebed04-c936-4a96-8585-8a6885549752\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb" Sep 08 15:37:19.686047 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:19.686022 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-646nj\" (UniqueName: \"kubernetes.io/projected/c9ebed04-c936-4a96-8585-8a6885549752-kube-api-access-646nj\") pod \"payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb\" (UID: \"c9ebed04-c936-4a96-8585-8a6885549752\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb" Sep 08 15:37:19.750288 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:19.750258 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb" Sep 08 15:37:19.773282 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:19.773243 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5"] Sep 08 15:37:19.774753 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:37:19.774722 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda14550_2475_4c54_8f87_cba74f09a027.slice/crio-e4b92e4faafeedecc604ec98d5d7379f4210caa4f7debaa01f0f9dc2e586a917 WatchSource:0}: Error finding container e4b92e4faafeedecc604ec98d5d7379f4210caa4f7debaa01f0f9dc2e586a917: Status 404 returned error can't find the container with id e4b92e4faafeedecc604ec98d5d7379f4210caa4f7debaa01f0f9dc2e586a917 Sep 08 15:37:19.776431 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:19.776410 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 15:37:20.032760 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:20.032733 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb"] Sep 08 15:37:20.035582 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:37:20.035537 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ebed04_c936_4a96_8585_8a6885549752.slice/crio-01ec947aabba2b9abe9f2b601980dfa318802aebfe6bee8823c875cd4b2bf342 WatchSource:0}: Error finding container 01ec947aabba2b9abe9f2b601980dfa318802aebfe6bee8823c875cd4b2bf342: Status 404 returned error can't find the container with id 01ec947aabba2b9abe9f2b601980dfa318802aebfe6bee8823c875cd4b2bf342 Sep 08 15:37:20.541264 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:20.541229 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb" event={"ID":"c9ebed04-c936-4a96-8585-8a6885549752","Type":"ContainerStarted","Data":"3f1f479f55a76b842460f123cbc5a96e91474262ee8d9ec78cb6e0256cfc0b37"} Sep 08 15:37:20.541550 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:20.541522 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb" event={"ID":"c9ebed04-c936-4a96-8585-8a6885549752","Type":"ContainerStarted","Data":"01ec947aabba2b9abe9f2b601980dfa318802aebfe6bee8823c875cd4b2bf342"} Sep 08 15:37:20.541687 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:20.541565 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb" Sep 08 15:37:20.542704 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:20.542679 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" event={"ID":"cda14550-2475-4c54-8f87-cba74f09a027","Type":"ContainerStarted","Data":"e4b92e4faafeedecc604ec98d5d7379f4210caa4f7debaa01f0f9dc2e586a917"} Sep 08 15:37:20.562104 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:20.562011 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb" podStartSLOduration=1.5619999020000002 podStartE2EDuration="1.561999902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:37:20.561178614 +0000 UTC m=+1702.713357195" watchObservedRunningTime="2026-09-08 15:37:20.561999902 +0000 UTC m=+1702.714178481" Sep 08 15:37:21.548386 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:21.548354 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" event={"ID":"cda14550-2475-4c54-8f87-cba74f09a027","Type":"ContainerStarted","Data":"f6ed5b51ba40478bfc663f236266a841cff2a5a2ffae7264b033d95d5c95a31f"} Sep 08 15:37:21.548848 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:21.548448 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" Sep 08 15:37:21.570543 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:21.570483 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" podStartSLOduration=2.157215049 podStartE2EDuration="3.570467231s" totalImagesPullingTime="1.413252182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:37:21.567665894 +0000 UTC m=+1703.719844473" watchObservedRunningTime="2026-09-08 15:37:21.570467231 +0000 UTC m=+1703.722645855" Sep 08 15:37:27.558047 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:27.558020 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" Sep 08 15:37:31.550829 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:31.550797 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb" Sep 08 15:37:43.328276 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.328236 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb"] Sep 08 15:37:43.331964 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.331938 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb" Sep 08 15:37:43.334696 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.334670 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ee0da5-dockercfg-xs289\"" Sep 08 15:37:43.334696 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.334690 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ee0da5\"" Sep 08 15:37:43.345897 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.345869 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb"] Sep 08 15:37:43.492401 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.492370 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd385e6-8d84-44be-89cb-84e1a6635ab3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb\" (UID: \"3bd385e6-8d84-44be-89cb-84e1a6635ab3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb" Sep 08 15:37:43.492593 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.492544 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grr8x\" (UniqueName: \"kubernetes.io/projected/3bd385e6-8d84-44be-89cb-84e1a6635ab3-kube-api-access-grr8x\") pod \"payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb\" (UID: \"3bd385e6-8d84-44be-89cb-84e1a6635ab3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb" Sep 08 15:37:43.513980 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.513955 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6"] Sep 08 15:37:43.517262 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.517243 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.519837 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.519819 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9dea65-openshift-default-dockercfg-rp2bm\"" Sep 08 15:37:43.531095 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.531072 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6"] Sep 08 15:37:43.593822 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.593751 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd385e6-8d84-44be-89cb-84e1a6635ab3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb\" (UID: \"3bd385e6-8d84-44be-89cb-84e1a6635ab3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb" Sep 08 15:37:43.593822 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.593813 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grr8x\" (UniqueName: \"kubernetes.io/projected/3bd385e6-8d84-44be-89cb-84e1a6635ab3-kube-api-access-grr8x\") pod \"payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb\" (UID: \"3bd385e6-8d84-44be-89cb-84e1a6635ab3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb" Sep 08 15:37:43.594431 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.594414 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd385e6-8d84-44be-89cb-84e1a6635ab3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb\" (UID: \"3bd385e6-8d84-44be-89cb-84e1a6635ab3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb" Sep 08 15:37:43.609592 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.609558 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grr8x\" (UniqueName: \"kubernetes.io/projected/3bd385e6-8d84-44be-89cb-84e1a6635ab3-kube-api-access-grr8x\") pod \"payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb\" (UID: \"3bd385e6-8d84-44be-89cb-84e1a6635ab3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb" Sep 08 15:37:43.643375 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.643327 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb" Sep 08 15:37:43.694857 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.694800 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-istio-envoy\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.695131 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.694895 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-istio-data\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.695131 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.694928 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a792ae0a-d83e-434f-9812-7d0c60892622-istiod-ca-cert\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.695131 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.694970 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-workload-socket\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.695131 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.694993 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a792ae0a-d83e-434f-9812-7d0c60892622-istio-token\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.695131 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.695030 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-workload-certs\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.695131 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.695069 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-credential-socket\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.695131 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.695093 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a792ae0a-d83e-434f-9812-7d0c60892622-istio-podinfo\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.695131 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.695117 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m47gc\" (UniqueName: \"kubernetes.io/projected/a792ae0a-d83e-434f-9812-7d0c60892622-kube-api-access-m47gc\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.796440 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.796396 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-workload-socket\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.796440 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.796437 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a792ae0a-d83e-434f-9812-7d0c60892622-istio-token\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.796738 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.796577 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-workload-certs\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.796738 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.796650 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-credential-socket\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.796738 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.796679 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a792ae0a-d83e-434f-9812-7d0c60892622-istio-podinfo\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.796738 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.796706 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m47gc\" (UniqueName: \"kubernetes.io/projected/a792ae0a-d83e-434f-9812-7d0c60892622-kube-api-access-m47gc\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.796936 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.796848 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-workload-socket\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.796936 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.796859 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-istio-envoy\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.796936 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.796897 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-workload-certs\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.796936 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.796927 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-credential-socket\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.797117 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.796977 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-istio-data\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.797117 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.797011 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a792ae0a-d83e-434f-9812-7d0c60892622-istiod-ca-cert\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.797261 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.797244 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-istio-data\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.797597 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.797538 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a792ae0a-d83e-434f-9812-7d0c60892622-istiod-ca-cert\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.799226 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.799200 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-istio-envoy\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.799504 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.799469 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a792ae0a-d83e-434f-9812-7d0c60892622-istio-podinfo\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.819674 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.819648 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m47gc\" (UniqueName: \"kubernetes.io/projected/a792ae0a-d83e-434f-9812-7d0c60892622-kube-api-access-m47gc\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.819880 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.819861 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a792ae0a-d83e-434f-9812-7d0c60892622-istio-token\") pod \"e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.827909 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.827883 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:43.935205 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:43.935166 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb"] Sep 08 15:37:43.938657 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:37:43.938610 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd385e6_8d84_44be_89cb_84e1a6635ab3.slice/crio-755db5372c7505c8092042a65b58af6d7d52c96f129754a3ceb034470d194d6d WatchSource:0}: Error finding container 755db5372c7505c8092042a65b58af6d7d52c96f129754a3ceb034470d194d6d: Status 404 returned error can't find the container with id 755db5372c7505c8092042a65b58af6d7d52c96f129754a3ceb034470d194d6d Sep 08 15:37:44.119612 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:44.119576 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6"] Sep 08 15:37:44.123673 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:37:44.123645 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda792ae0a_d83e_434f_9812_7d0c60892622.slice/crio-e5fc0590de5ff1881e6834c68573352ff432cff7c1a39ebaf872cb325ca683e8 WatchSource:0}: Error finding container e5fc0590de5ff1881e6834c68573352ff432cff7c1a39ebaf872cb325ca683e8: Status 404 returned error can't find the container with id e5fc0590de5ff1881e6834c68573352ff432cff7c1a39ebaf872cb325ca683e8 Sep 08 15:37:44.126074 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:44.126039 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:37:44.126171 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:44.126108 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:37:44.126171 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:44.126136 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:37:44.663586 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:44.663539 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" event={"ID":"a792ae0a-d83e-434f-9812-7d0c60892622","Type":"ContainerStarted","Data":"0effbc8e01efe382488400290bc498f17c1c7972aaca98b23d5562c7f9edf4a9"} Sep 08 15:37:44.664060 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:44.663609 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" event={"ID":"a792ae0a-d83e-434f-9812-7d0c60892622","Type":"ContainerStarted","Data":"e5fc0590de5ff1881e6834c68573352ff432cff7c1a39ebaf872cb325ca683e8"} Sep 08 15:37:44.665335 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:44.665305 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb" event={"ID":"3bd385e6-8d84-44be-89cb-84e1a6635ab3","Type":"ContainerStarted","Data":"0215f5d74010a26703891be40ceca7b24eeff2bcf0bb3cf0ee71c9530d076f30"} Sep 08 15:37:44.665455 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:44.665359 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb" Sep 08 15:37:44.665455 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:44.665385 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb" event={"ID":"3bd385e6-8d84-44be-89cb-84e1a6635ab3","Type":"ContainerStarted","Data":"755db5372c7505c8092042a65b58af6d7d52c96f129754a3ceb034470d194d6d"} Sep 08 15:37:44.690772 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:44.690707 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" podStartSLOduration=1.690686387 podStartE2EDuration="1.690686387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:37:44.690524981 +0000 UTC m=+1726.842703559" watchObservedRunningTime="2026-09-08 15:37:44.690686387 +0000 UTC m=+1726.842864965" Sep 08 15:37:44.719825 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:44.719762 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb" podStartSLOduration=1.719737549 podStartE2EDuration="1.719737549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:37:44.718845196 +0000 UTC m=+1726.871023775" watchObservedRunningTime="2026-09-08 15:37:44.719737549 +0000 UTC m=+1726.871916130" Sep 08 15:37:44.828433 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:44.828382 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:45.833539 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:45.833513 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:46.676624 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:46.676594 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:46.677732 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:46.677713 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:37:55.673327 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:37:55.673292 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb" Sep 08 15:38:05.795579 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:05.795540 2572 generic.go:356] "Generic (PLEG): container finished" podID="a792ae0a-d83e-434f-9812-7d0c60892622" containerID="0effbc8e01efe382488400290bc498f17c1c7972aaca98b23d5562c7f9edf4a9" exitCode=0 Sep 08 15:38:05.796057 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:05.795621 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" event={"ID":"a792ae0a-d83e-434f-9812-7d0c60892622","Type":"ContainerDied","Data":"0effbc8e01efe382488400290bc498f17c1c7972aaca98b23d5562c7f9edf4a9"} Sep 08 15:38:05.796145 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:05.796114 2572 scope.go:117] "RemoveContainer" containerID="0effbc8e01efe382488400290bc498f17c1c7972aaca98b23d5562c7f9edf4a9" Sep 08 15:38:05.797451 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:05.797425 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:38:05.797547 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:05.797532 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:38:05.797596 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:05.797569 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 08 15:38:06.802734 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:06.802700 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" event={"ID":"a792ae0a-d83e-434f-9812-7d0c60892622","Type":"ContainerStarted","Data":"4e9de8912f4c9885d939f50d1520b59532e0237999735beb3a5393c4a35e9d15"} Sep 08 15:38:06.828374 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:06.828333 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:38:06.828686 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:06.828663 2572 patch_prober.go:28] interesting pod/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 08 15:38:06.828765 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:06.828711 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" podUID="a792ae0a-d83e-434f-9812-7d0c60892622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 08 15:38:07.829095 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:07.829054 2572 patch_prober.go:28] interesting pod/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 08 15:38:07.829636 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:07.829114 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" podUID="a792ae0a-d83e-434f-9812-7d0c60892622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 08 15:38:08.828880 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:08.828838 2572 patch_prober.go:28] interesting pod/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 08 15:38:08.829066 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:08.828900 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" podUID="a792ae0a-d83e-434f-9812-7d0c60892622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 08 15:38:09.828578 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:09.828539 2572 patch_prober.go:28] interesting pod/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 08 15:38:09.829055 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:09.828618 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" podUID="a792ae0a-d83e-434f-9812-7d0c60892622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 08 15:38:10.828928 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:10.828891 2572 patch_prober.go:28] interesting pod/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 08 15:38:10.829334 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:10.828946 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" podUID="a792ae0a-d83e-434f-9812-7d0c60892622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 08 15:38:11.829236 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:11.829198 2572 patch_prober.go:28] interesting pod/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 08 15:38:11.829651 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:11.829256 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" podUID="a792ae0a-d83e-434f-9812-7d0c60892622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 08 15:38:12.833588 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:12.833557 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:38:12.833986 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:12.833856 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:38:12.834625 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:12.834607 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:38:48.720670 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:48.720634 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb"] Sep 08 15:38:48.721207 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:48.720884 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb" podUID="3bd385e6-8d84-44be-89cb-84e1a6635ab3" containerName="payload-pre-processing" containerID="cri-o://0215f5d74010a26703891be40ceca7b24eeff2bcf0bb3cf0ee71c9530d076f30" gracePeriod=30 Sep 08 15:38:48.967590 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:48.967560 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb" Sep 08 15:38:49.010570 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:49.010533 2572 generic.go:356] "Generic (PLEG): container finished" podID="3bd385e6-8d84-44be-89cb-84e1a6635ab3" containerID="0215f5d74010a26703891be40ceca7b24eeff2bcf0bb3cf0ee71c9530d076f30" exitCode=0 Sep 08 15:38:49.010724 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:49.010621 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb" Sep 08 15:38:49.010724 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:49.010651 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb" event={"ID":"3bd385e6-8d84-44be-89cb-84e1a6635ab3","Type":"ContainerDied","Data":"0215f5d74010a26703891be40ceca7b24eeff2bcf0bb3cf0ee71c9530d076f30"} Sep 08 15:38:49.010724 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:49.010707 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb" event={"ID":"3bd385e6-8d84-44be-89cb-84e1a6635ab3","Type":"ContainerDied","Data":"755db5372c7505c8092042a65b58af6d7d52c96f129754a3ceb034470d194d6d"} Sep 08 15:38:49.010885 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:49.010731 2572 scope.go:117] "RemoveContainer" containerID="0215f5d74010a26703891be40ceca7b24eeff2bcf0bb3cf0ee71c9530d076f30" Sep 08 15:38:49.019554 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:49.019536 2572 scope.go:117] "RemoveContainer" containerID="0215f5d74010a26703891be40ceca7b24eeff2bcf0bb3cf0ee71c9530d076f30" Sep 08 15:38:49.019847 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:38:49.019831 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0215f5d74010a26703891be40ceca7b24eeff2bcf0bb3cf0ee71c9530d076f30\": container with ID starting with 0215f5d74010a26703891be40ceca7b24eeff2bcf0bb3cf0ee71c9530d076f30 not found: ID does not exist" containerID="0215f5d74010a26703891be40ceca7b24eeff2bcf0bb3cf0ee71c9530d076f30" Sep 08 15:38:49.019904 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:49.019855 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0215f5d74010a26703891be40ceca7b24eeff2bcf0bb3cf0ee71c9530d076f30"} err="failed to get container status \"0215f5d74010a26703891be40ceca7b24eeff2bcf0bb3cf0ee71c9530d076f30\": rpc error: code = NotFound desc = could not find container \"0215f5d74010a26703891be40ceca7b24eeff2bcf0bb3cf0ee71c9530d076f30\": container with ID starting with 0215f5d74010a26703891be40ceca7b24eeff2bcf0bb3cf0ee71c9530d076f30 not found: ID does not exist" Sep 08 15:38:49.119611 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:49.119581 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grr8x\" (UniqueName: \"kubernetes.io/projected/3bd385e6-8d84-44be-89cb-84e1a6635ab3-kube-api-access-grr8x\") pod \"3bd385e6-8d84-44be-89cb-84e1a6635ab3\" (UID: \"3bd385e6-8d84-44be-89cb-84e1a6635ab3\") " Sep 08 15:38:49.119762 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:49.119719 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd385e6-8d84-44be-89cb-84e1a6635ab3-plugins-config-volume\") pod \"3bd385e6-8d84-44be-89cb-84e1a6635ab3\" (UID: \"3bd385e6-8d84-44be-89cb-84e1a6635ab3\") " Sep 08 15:38:49.120034 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:49.120004 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd385e6-8d84-44be-89cb-84e1a6635ab3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3bd385e6-8d84-44be-89cb-84e1a6635ab3" (UID: "3bd385e6-8d84-44be-89cb-84e1a6635ab3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:38:49.121891 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:49.121866 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd385e6-8d84-44be-89cb-84e1a6635ab3-kube-api-access-grr8x" (OuterVolumeSpecName: "kube-api-access-grr8x") pod "3bd385e6-8d84-44be-89cb-84e1a6635ab3" (UID: "3bd385e6-8d84-44be-89cb-84e1a6635ab3"). InnerVolumeSpecName "kube-api-access-grr8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:38:49.220754 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:49.220704 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grr8x\" (UniqueName: \"kubernetes.io/projected/3bd385e6-8d84-44be-89cb-84e1a6635ab3-kube-api-access-grr8x\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:38:49.220754 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:49.220748 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd385e6-8d84-44be-89cb-84e1a6635ab3-plugins-config-volume\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:38:49.334410 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:49.334379 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb"] Sep 08 15:38:49.338867 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:49.338845 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ee0da5-5d95dd7654-bvjqb"] Sep 08 15:38:50.370996 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:38:50.370962 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bd385e6-8d84-44be-89cb-84e1a6635ab3" path="/var/lib/kubelet/pods/3bd385e6-8d84-44be-89cb-84e1a6635ab3/volumes" Sep 08 15:39:07.461588 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.461553 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn"] Sep 08 15:39:07.462005 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.461756 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" podUID="b5583bd4-423e-412a-8a6c-911422cc02c5" containerName="maas-api" containerID="cri-o://fc3bf493f164b4fed7f169e15fd249ce7dd9ff5e52bf924988cff87a2afb3e54" gracePeriod=30 Sep 08 15:39:07.544532 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.544466 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt"] Sep 08 15:39:07.545116 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.545064 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt" podUID="f80b5111-6cdf-4b32-8f35-81f5c0eab644" containerName="payload-pre-processing" containerID="cri-o://c10446b3337c6202dcf4ed5b60835d1a802a88a4cc9be568853e57e0335298bb" gracePeriod=30 Sep 08 15:39:07.820147 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.817218 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" Sep 08 15:39:07.858086 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.858055 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt" Sep 08 15:39:07.887437 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.887409 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5583bd4-423e-412a-8a6c-911422cc02c5-metrics-certs\") pod \"b5583bd4-423e-412a-8a6c-911422cc02c5\" (UID: \"b5583bd4-423e-412a-8a6c-911422cc02c5\") " Sep 08 15:39:07.887437 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.887441 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5583bd4-423e-412a-8a6c-911422cc02c5-maas-api-tls\") pod \"b5583bd4-423e-412a-8a6c-911422cc02c5\" (UID: \"b5583bd4-423e-412a-8a6c-911422cc02c5\") " Sep 08 15:39:07.887639 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.887464 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80b5111-6cdf-4b32-8f35-81f5c0eab644-plugins-config-volume\") pod \"f80b5111-6cdf-4b32-8f35-81f5c0eab644\" (UID: \"f80b5111-6cdf-4b32-8f35-81f5c0eab644\") " Sep 08 15:39:07.887639 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.887487 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5583bd4-423e-412a-8a6c-911422cc02c5-openshift-service-ca\") pod \"b5583bd4-423e-412a-8a6c-911422cc02c5\" (UID: \"b5583bd4-423e-412a-8a6c-911422cc02c5\") " Sep 08 15:39:07.887639 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.887585 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dksdd\" (UniqueName: \"kubernetes.io/projected/f80b5111-6cdf-4b32-8f35-81f5c0eab644-kube-api-access-dksdd\") pod \"f80b5111-6cdf-4b32-8f35-81f5c0eab644\" (UID: \"f80b5111-6cdf-4b32-8f35-81f5c0eab644\") " Sep 08 15:39:07.887639 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.887624 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkh2w\" (UniqueName: \"kubernetes.io/projected/b5583bd4-423e-412a-8a6c-911422cc02c5-kube-api-access-kkh2w\") pod \"b5583bd4-423e-412a-8a6c-911422cc02c5\" (UID: \"b5583bd4-423e-412a-8a6c-911422cc02c5\") " Sep 08 15:39:07.887888 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.887861 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5583bd4-423e-412a-8a6c-911422cc02c5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b5583bd4-423e-412a-8a6c-911422cc02c5" (UID: "b5583bd4-423e-412a-8a6c-911422cc02c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:39:07.887950 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.887886 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f80b5111-6cdf-4b32-8f35-81f5c0eab644-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f80b5111-6cdf-4b32-8f35-81f5c0eab644" (UID: "f80b5111-6cdf-4b32-8f35-81f5c0eab644"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:39:07.889801 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.889779 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5583bd4-423e-412a-8a6c-911422cc02c5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b5583bd4-423e-412a-8a6c-911422cc02c5" (UID: "b5583bd4-423e-412a-8a6c-911422cc02c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 15:39:07.889886 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.889790 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5583bd4-423e-412a-8a6c-911422cc02c5-kube-api-access-kkh2w" (OuterVolumeSpecName: "kube-api-access-kkh2w") pod "b5583bd4-423e-412a-8a6c-911422cc02c5" (UID: "b5583bd4-423e-412a-8a6c-911422cc02c5"). InnerVolumeSpecName "kube-api-access-kkh2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:39:07.889886 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.889813 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5583bd4-423e-412a-8a6c-911422cc02c5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b5583bd4-423e-412a-8a6c-911422cc02c5" (UID: "b5583bd4-423e-412a-8a6c-911422cc02c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 15:39:07.889886 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.889820 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f80b5111-6cdf-4b32-8f35-81f5c0eab644-kube-api-access-dksdd" (OuterVolumeSpecName: "kube-api-access-dksdd") pod "f80b5111-6cdf-4b32-8f35-81f5c0eab644" (UID: "f80b5111-6cdf-4b32-8f35-81f5c0eab644"). InnerVolumeSpecName "kube-api-access-dksdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:39:07.988903 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.988814 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dksdd\" (UniqueName: \"kubernetes.io/projected/f80b5111-6cdf-4b32-8f35-81f5c0eab644-kube-api-access-dksdd\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:39:07.988903 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.988844 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkh2w\" (UniqueName: \"kubernetes.io/projected/b5583bd4-423e-412a-8a6c-911422cc02c5-kube-api-access-kkh2w\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:39:07.988903 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.988854 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5583bd4-423e-412a-8a6c-911422cc02c5-metrics-certs\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:39:07.988903 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.988863 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5583bd4-423e-412a-8a6c-911422cc02c5-maas-api-tls\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:39:07.988903 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.988871 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80b5111-6cdf-4b32-8f35-81f5c0eab644-plugins-config-volume\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:39:07.988903 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:07.988879 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5583bd4-423e-412a-8a6c-911422cc02c5-openshift-service-ca\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:39:08.104340 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.104307 2572 generic.go:356] "Generic (PLEG): container finished" podID="b5583bd4-423e-412a-8a6c-911422cc02c5" containerID="fc3bf493f164b4fed7f169e15fd249ce7dd9ff5e52bf924988cff87a2afb3e54" exitCode=0 Sep 08 15:39:08.104556 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.104354 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" Sep 08 15:39:08.104556 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.104392 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" event={"ID":"b5583bd4-423e-412a-8a6c-911422cc02c5","Type":"ContainerDied","Data":"fc3bf493f164b4fed7f169e15fd249ce7dd9ff5e52bf924988cff87a2afb3e54"} Sep 08 15:39:08.104556 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.104451 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn" event={"ID":"b5583bd4-423e-412a-8a6c-911422cc02c5","Type":"ContainerDied","Data":"86256d80bb5ef71219c485e5d12a32e3088ad08cdbac07aa83109eddbc510288"} Sep 08 15:39:08.104556 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.104483 2572 scope.go:117] "RemoveContainer" containerID="fc3bf493f164b4fed7f169e15fd249ce7dd9ff5e52bf924988cff87a2afb3e54" Sep 08 15:39:08.105875 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.105850 2572 generic.go:356] "Generic (PLEG): container finished" podID="f80b5111-6cdf-4b32-8f35-81f5c0eab644" containerID="c10446b3337c6202dcf4ed5b60835d1a802a88a4cc9be568853e57e0335298bb" exitCode=0 Sep 08 15:39:08.106010 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.105898 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt" Sep 08 15:39:08.106010 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.105893 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt" event={"ID":"f80b5111-6cdf-4b32-8f35-81f5c0eab644","Type":"ContainerDied","Data":"c10446b3337c6202dcf4ed5b60835d1a802a88a4cc9be568853e57e0335298bb"} Sep 08 15:39:08.106010 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.105955 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt" event={"ID":"f80b5111-6cdf-4b32-8f35-81f5c0eab644","Type":"ContainerDied","Data":"44021ff168cdd3efab223855d8309ca8ea52654b6be3e7136dd851d9f40df42a"} Sep 08 15:39:08.114664 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.114647 2572 scope.go:117] "RemoveContainer" containerID="fc3bf493f164b4fed7f169e15fd249ce7dd9ff5e52bf924988cff87a2afb3e54" Sep 08 15:39:08.114998 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:39:08.114979 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3bf493f164b4fed7f169e15fd249ce7dd9ff5e52bf924988cff87a2afb3e54\": container with ID starting with fc3bf493f164b4fed7f169e15fd249ce7dd9ff5e52bf924988cff87a2afb3e54 not found: ID does not exist" containerID="fc3bf493f164b4fed7f169e15fd249ce7dd9ff5e52bf924988cff87a2afb3e54" Sep 08 15:39:08.115063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.115005 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3bf493f164b4fed7f169e15fd249ce7dd9ff5e52bf924988cff87a2afb3e54"} err="failed to get container status \"fc3bf493f164b4fed7f169e15fd249ce7dd9ff5e52bf924988cff87a2afb3e54\": rpc error: code = NotFound desc = could not find container \"fc3bf493f164b4fed7f169e15fd249ce7dd9ff5e52bf924988cff87a2afb3e54\": container with ID starting with fc3bf493f164b4fed7f169e15fd249ce7dd9ff5e52bf924988cff87a2afb3e54 not found: ID does not exist" Sep 08 15:39:08.115063 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.115022 2572 scope.go:117] "RemoveContainer" containerID="c10446b3337c6202dcf4ed5b60835d1a802a88a4cc9be568853e57e0335298bb" Sep 08 15:39:08.123180 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.123165 2572 scope.go:117] "RemoveContainer" containerID="c10446b3337c6202dcf4ed5b60835d1a802a88a4cc9be568853e57e0335298bb" Sep 08 15:39:08.123450 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:39:08.123432 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c10446b3337c6202dcf4ed5b60835d1a802a88a4cc9be568853e57e0335298bb\": container with ID starting with c10446b3337c6202dcf4ed5b60835d1a802a88a4cc9be568853e57e0335298bb not found: ID does not exist" containerID="c10446b3337c6202dcf4ed5b60835d1a802a88a4cc9be568853e57e0335298bb" Sep 08 15:39:08.123536 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.123458 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c10446b3337c6202dcf4ed5b60835d1a802a88a4cc9be568853e57e0335298bb"} err="failed to get container status \"c10446b3337c6202dcf4ed5b60835d1a802a88a4cc9be568853e57e0335298bb\": rpc error: code = NotFound desc = could not find container \"c10446b3337c6202dcf4ed5b60835d1a802a88a4cc9be568853e57e0335298bb\": container with ID starting with c10446b3337c6202dcf4ed5b60835d1a802a88a4cc9be568853e57e0335298bb not found: ID does not exist" Sep 08 15:39:08.128109 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.128072 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn"] Sep 08 15:39:08.129836 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.129816 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-149fdb-6466bb7968-fjrzn"] Sep 08 15:39:08.144815 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.144794 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt"] Sep 08 15:39:08.148829 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.148805 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-149fdb-6bc584bf4b-p4lgt"] Sep 08 15:39:08.371886 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.371851 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5583bd4-423e-412a-8a6c-911422cc02c5" path="/var/lib/kubelet/pods/b5583bd4-423e-412a-8a6c-911422cc02c5/volumes" Sep 08 15:39:08.372214 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:08.372201 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f80b5111-6cdf-4b32-8f35-81f5c0eab644" path="/var/lib/kubelet/pods/f80b5111-6cdf-4b32-8f35-81f5c0eab644/volumes" Sep 08 15:39:28.211091 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:28.211054 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb_c9ebed04-c936-4a96-8585-8a6885549752/payload-processing/0.log" Sep 08 15:39:35.200439 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:35.200402 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk"] Sep 08 15:39:35.200896 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:35.200634 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk" podUID="a4a58b42-1660-4853-987e-c354a0887598" containerName="payload-processing" containerID="cri-o://c6bcec7ab5d65f2951bc88d44ad06edd0cc217309a0946f5cce9afb69b279fd2" gracePeriod=30 Sep 08 15:39:35.464614 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:35.464552 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk" Sep 08 15:39:35.542647 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:35.542615 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcwsd\" (UniqueName: \"kubernetes.io/projected/a4a58b42-1660-4853-987e-c354a0887598-kube-api-access-qcwsd\") pod \"a4a58b42-1660-4853-987e-c354a0887598\" (UID: \"a4a58b42-1660-4853-987e-c354a0887598\") " Sep 08 15:39:35.542857 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:35.542691 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a58b42-1660-4853-987e-c354a0887598-plugins-config-volume\") pod \"a4a58b42-1660-4853-987e-c354a0887598\" (UID: \"a4a58b42-1660-4853-987e-c354a0887598\") " Sep 08 15:39:35.543084 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:35.543058 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a58b42-1660-4853-987e-c354a0887598-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a4a58b42-1660-4853-987e-c354a0887598" (UID: "a4a58b42-1660-4853-987e-c354a0887598"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:39:35.544981 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:35.544958 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a58b42-1660-4853-987e-c354a0887598-kube-api-access-qcwsd" (OuterVolumeSpecName: "kube-api-access-qcwsd") pod "a4a58b42-1660-4853-987e-c354a0887598" (UID: "a4a58b42-1660-4853-987e-c354a0887598"). InnerVolumeSpecName "kube-api-access-qcwsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:39:35.643823 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:35.643785 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a58b42-1660-4853-987e-c354a0887598-plugins-config-volume\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:39:35.643823 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:35.643814 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcwsd\" (UniqueName: \"kubernetes.io/projected/a4a58b42-1660-4853-987e-c354a0887598-kube-api-access-qcwsd\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:39:36.250594 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:36.250560 2572 generic.go:356] "Generic (PLEG): container finished" podID="a4a58b42-1660-4853-987e-c354a0887598" containerID="c6bcec7ab5d65f2951bc88d44ad06edd0cc217309a0946f5cce9afb69b279fd2" exitCode=0 Sep 08 15:39:36.251044 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:36.250604 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk" Sep 08 15:39:36.251044 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:36.250619 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk" event={"ID":"a4a58b42-1660-4853-987e-c354a0887598","Type":"ContainerDied","Data":"c6bcec7ab5d65f2951bc88d44ad06edd0cc217309a0946f5cce9afb69b279fd2"} Sep 08 15:39:36.251044 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:36.250671 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk" event={"ID":"a4a58b42-1660-4853-987e-c354a0887598","Type":"ContainerDied","Data":"011173a2a1954771bf6e94b73ce4c30b5c940e15db1c298a19f86244b9128b0c"} Sep 08 15:39:36.251044 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:36.250696 2572 scope.go:117] "RemoveContainer" containerID="c6bcec7ab5d65f2951bc88d44ad06edd0cc217309a0946f5cce9afb69b279fd2" Sep 08 15:39:36.261831 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:36.261810 2572 scope.go:117] "RemoveContainer" containerID="c6bcec7ab5d65f2951bc88d44ad06edd0cc217309a0946f5cce9afb69b279fd2" Sep 08 15:39:36.262131 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:39:36.262110 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6bcec7ab5d65f2951bc88d44ad06edd0cc217309a0946f5cce9afb69b279fd2\": container with ID starting with c6bcec7ab5d65f2951bc88d44ad06edd0cc217309a0946f5cce9afb69b279fd2 not found: ID does not exist" containerID="c6bcec7ab5d65f2951bc88d44ad06edd0cc217309a0946f5cce9afb69b279fd2" Sep 08 15:39:36.262200 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:36.262139 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6bcec7ab5d65f2951bc88d44ad06edd0cc217309a0946f5cce9afb69b279fd2"} err="failed to get container status \"c6bcec7ab5d65f2951bc88d44ad06edd0cc217309a0946f5cce9afb69b279fd2\": rpc error: code = NotFound desc = could not find container \"c6bcec7ab5d65f2951bc88d44ad06edd0cc217309a0946f5cce9afb69b279fd2\": container with ID starting with c6bcec7ab5d65f2951bc88d44ad06edd0cc217309a0946f5cce9afb69b279fd2 not found: ID does not exist" Sep 08 15:39:36.274061 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:36.274039 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk"] Sep 08 15:39:36.281791 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:36.281770 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-04285b-864b5bc48f-rd4jk"] Sep 08 15:39:36.378125 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:36.378082 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4a58b42-1660-4853-987e-c354a0887598" path="/var/lib/kubelet/pods/a4a58b42-1660-4853-987e-c354a0887598/volumes" Sep 08 15:39:42.144099 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:42.144067 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d"] Sep 08 15:39:42.144547 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:42.144294 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" podUID="40e53b91-bb2d-44c9-ab1e-514170090912" containerName="istio-proxy" containerID="cri-o://98bb40d7290b656d919e1c49fc21c9dcfac84086c3624f50b78f3fe081e0218d" gracePeriod=30 Sep 08 15:39:47.303682 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.303642 2572 generic.go:356] "Generic (PLEG): container finished" podID="40e53b91-bb2d-44c9-ab1e-514170090912" containerID="98bb40d7290b656d919e1c49fc21c9dcfac84086c3624f50b78f3fe081e0218d" exitCode=0 Sep 08 15:39:47.304041 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.303710 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" event={"ID":"40e53b91-bb2d-44c9-ab1e-514170090912","Type":"ContainerDied","Data":"98bb40d7290b656d919e1c49fc21c9dcfac84086c3624f50b78f3fe081e0218d"} Sep 08 15:39:47.304041 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.303782 2572 scope.go:117] "RemoveContainer" containerID="cc08807784904595a8970e7f0255d51f3c76973aee7c0ec22520065640159bf2" Sep 08 15:39:47.389775 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.389746 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:39:47.554460 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.554430 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-istio-data\") pod \"40e53b91-bb2d-44c9-ab1e-514170090912\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " Sep 08 15:39:47.554678 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.554501 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-istio-envoy\") pod \"40e53b91-bb2d-44c9-ab1e-514170090912\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " Sep 08 15:39:47.554678 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.554559 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln7r2\" (UniqueName: \"kubernetes.io/projected/40e53b91-bb2d-44c9-ab1e-514170090912-kube-api-access-ln7r2\") pod \"40e53b91-bb2d-44c9-ab1e-514170090912\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " Sep 08 15:39:47.554678 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.554576 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-workload-socket\") pod \"40e53b91-bb2d-44c9-ab1e-514170090912\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " Sep 08 15:39:47.554678 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.554624 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e53b91-bb2d-44c9-ab1e-514170090912-istio-podinfo\") pod \"40e53b91-bb2d-44c9-ab1e-514170090912\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " Sep 08 15:39:47.554678 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.554652 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e53b91-bb2d-44c9-ab1e-514170090912-istio-token\") pod \"40e53b91-bb2d-44c9-ab1e-514170090912\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " Sep 08 15:39:47.554958 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.554698 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-credential-socket\") pod \"40e53b91-bb2d-44c9-ab1e-514170090912\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " Sep 08 15:39:47.554958 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.554723 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-workload-certs\") pod \"40e53b91-bb2d-44c9-ab1e-514170090912\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " Sep 08 15:39:47.554958 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.554747 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e53b91-bb2d-44c9-ab1e-514170090912-istiod-ca-cert\") pod \"40e53b91-bb2d-44c9-ab1e-514170090912\" (UID: \"40e53b91-bb2d-44c9-ab1e-514170090912\") " Sep 08 15:39:47.554958 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.554757 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-istio-data" (OuterVolumeSpecName: "istio-data") pod "40e53b91-bb2d-44c9-ab1e-514170090912" (UID: "40e53b91-bb2d-44c9-ab1e-514170090912"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:39:47.554958 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.554911 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "40e53b91-bb2d-44c9-ab1e-514170090912" (UID: "40e53b91-bb2d-44c9-ab1e-514170090912"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:39:47.555239 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.555043 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-istio-data\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:39:47.555239 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.555092 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-workload-socket\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:39:47.555239 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.555187 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e53b91-bb2d-44c9-ab1e-514170090912-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "40e53b91-bb2d-44c9-ab1e-514170090912" (UID: "40e53b91-bb2d-44c9-ab1e-514170090912"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:39:47.555590 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.555544 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "40e53b91-bb2d-44c9-ab1e-514170090912" (UID: "40e53b91-bb2d-44c9-ab1e-514170090912"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:39:47.555766 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.555730 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "40e53b91-bb2d-44c9-ab1e-514170090912" (UID: "40e53b91-bb2d-44c9-ab1e-514170090912"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:39:47.557031 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.556989 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "40e53b91-bb2d-44c9-ab1e-514170090912" (UID: "40e53b91-bb2d-44c9-ab1e-514170090912"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:39:47.557451 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.557428 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e53b91-bb2d-44c9-ab1e-514170090912-kube-api-access-ln7r2" (OuterVolumeSpecName: "kube-api-access-ln7r2") pod "40e53b91-bb2d-44c9-ab1e-514170090912" (UID: "40e53b91-bb2d-44c9-ab1e-514170090912"). InnerVolumeSpecName "kube-api-access-ln7r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:39:47.557573 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.557471 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40e53b91-bb2d-44c9-ab1e-514170090912-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "40e53b91-bb2d-44c9-ab1e-514170090912" (UID: "40e53b91-bb2d-44c9-ab1e-514170090912"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 15:39:47.557670 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.557654 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e53b91-bb2d-44c9-ab1e-514170090912-istio-token" (OuterVolumeSpecName: "istio-token") pod "40e53b91-bb2d-44c9-ab1e-514170090912" (UID: "40e53b91-bb2d-44c9-ab1e-514170090912"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:39:47.656606 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.656562 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e53b91-bb2d-44c9-ab1e-514170090912-istio-podinfo\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:39:47.656606 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.656599 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e53b91-bb2d-44c9-ab1e-514170090912-istio-token\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:39:47.656606 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.656608 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-credential-socket\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:39:47.656606 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.656619 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-workload-certs\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:39:47.656876 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.656627 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e53b91-bb2d-44c9-ab1e-514170090912-istiod-ca-cert\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:39:47.656876 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.656635 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e53b91-bb2d-44c9-ab1e-514170090912-istio-envoy\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:39:47.656876 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:47.656643 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln7r2\" (UniqueName: \"kubernetes.io/projected/40e53b91-bb2d-44c9-ab1e-514170090912-kube-api-access-ln7r2\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:39:48.309691 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:48.309665 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" Sep 08 15:39:48.309691 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:48.309672 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d" event={"ID":"40e53b91-bb2d-44c9-ab1e-514170090912","Type":"ContainerDied","Data":"82386de768c1db96150d1944ad8c43dd77bc31f3d2da089942c45db6b95dafba"} Sep 08 15:39:48.310202 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:48.309728 2572 scope.go:117] "RemoveContainer" containerID="98bb40d7290b656d919e1c49fc21c9dcfac84086c3624f50b78f3fe081e0218d" Sep 08 15:39:48.332579 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:48.332546 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d"] Sep 08 15:39:48.340061 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:48.340036 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-04285b-openshift-default-7c595df64b-v4g9d"] Sep 08 15:39:48.370882 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:39:48.370854 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40e53b91-bb2d-44c9-ab1e-514170090912" path="/var/lib/kubelet/pods/40e53b91-bb2d-44c9-ab1e-514170090912/volumes" Sep 08 15:40:30.685704 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:30.685622 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5"] Sep 08 15:40:30.686238 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:30.685907 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" podUID="cda14550-2475-4c54-8f87-cba74f09a027" containerName="maas-api" containerID="cri-o://f6ed5b51ba40478bfc663f236266a841cff2a5a2ffae7264b033d95d5c95a31f" gracePeriod=30 Sep 08 15:40:30.773420 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:30.773386 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb"] Sep 08 15:40:30.773691 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:30.773639 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb" podUID="c9ebed04-c936-4a96-8585-8a6885549752" containerName="payload-processing" containerID="cri-o://3f1f479f55a76b842460f123cbc5a96e91474262ee8d9ec78cb6e0256cfc0b37" gracePeriod=30 Sep 08 15:40:31.027152 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.027126 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" Sep 08 15:40:31.036863 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.036842 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb" Sep 08 15:40:31.124237 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.124204 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cda14550-2475-4c54-8f87-cba74f09a027-openshift-service-ca\") pod \"cda14550-2475-4c54-8f87-cba74f09a027\" (UID: \"cda14550-2475-4c54-8f87-cba74f09a027\") " Sep 08 15:40:31.124402 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.124257 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cda14550-2475-4c54-8f87-cba74f09a027-maas-api-tls\") pod \"cda14550-2475-4c54-8f87-cba74f09a027\" (UID: \"cda14550-2475-4c54-8f87-cba74f09a027\") " Sep 08 15:40:31.124402 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.124279 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ebed04-c936-4a96-8585-8a6885549752-plugins-config-volume\") pod \"c9ebed04-c936-4a96-8585-8a6885549752\" (UID: \"c9ebed04-c936-4a96-8585-8a6885549752\") " Sep 08 15:40:31.124402 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.124318 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-646nj\" (UniqueName: \"kubernetes.io/projected/c9ebed04-c936-4a96-8585-8a6885549752-kube-api-access-646nj\") pod \"c9ebed04-c936-4a96-8585-8a6885549752\" (UID: \"c9ebed04-c936-4a96-8585-8a6885549752\") " Sep 08 15:40:31.124402 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.124397 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7qk5\" (UniqueName: \"kubernetes.io/projected/cda14550-2475-4c54-8f87-cba74f09a027-kube-api-access-g7qk5\") pod \"cda14550-2475-4c54-8f87-cba74f09a027\" (UID: \"cda14550-2475-4c54-8f87-cba74f09a027\") " Sep 08 15:40:31.124697 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.124420 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cda14550-2475-4c54-8f87-cba74f09a027-metrics-certs\") pod \"cda14550-2475-4c54-8f87-cba74f09a027\" (UID: \"cda14550-2475-4c54-8f87-cba74f09a027\") " Sep 08 15:40:31.124697 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.124647 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda14550-2475-4c54-8f87-cba74f09a027-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cda14550-2475-4c54-8f87-cba74f09a027" (UID: "cda14550-2475-4c54-8f87-cba74f09a027"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:40:31.124808 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.124730 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ebed04-c936-4a96-8585-8a6885549752-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c9ebed04-c936-4a96-8585-8a6885549752" (UID: "c9ebed04-c936-4a96-8585-8a6885549752"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:40:31.124808 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.124791 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cda14550-2475-4c54-8f87-cba74f09a027-openshift-service-ca\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:40:31.126683 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.126658 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cda14550-2475-4c54-8f87-cba74f09a027-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cda14550-2475-4c54-8f87-cba74f09a027" (UID: "cda14550-2475-4c54-8f87-cba74f09a027"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 15:40:31.126683 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.126674 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ebed04-c936-4a96-8585-8a6885549752-kube-api-access-646nj" (OuterVolumeSpecName: "kube-api-access-646nj") pod "c9ebed04-c936-4a96-8585-8a6885549752" (UID: "c9ebed04-c936-4a96-8585-8a6885549752"). InnerVolumeSpecName "kube-api-access-646nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:40:31.127064 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.127038 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda14550-2475-4c54-8f87-cba74f09a027-kube-api-access-g7qk5" (OuterVolumeSpecName: "kube-api-access-g7qk5") pod "cda14550-2475-4c54-8f87-cba74f09a027" (UID: "cda14550-2475-4c54-8f87-cba74f09a027"). InnerVolumeSpecName "kube-api-access-g7qk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:40:31.127064 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.127043 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cda14550-2475-4c54-8f87-cba74f09a027-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cda14550-2475-4c54-8f87-cba74f09a027" (UID: "cda14550-2475-4c54-8f87-cba74f09a027"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 15:40:31.225817 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.225739 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7qk5\" (UniqueName: \"kubernetes.io/projected/cda14550-2475-4c54-8f87-cba74f09a027-kube-api-access-g7qk5\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:40:31.225817 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.225771 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cda14550-2475-4c54-8f87-cba74f09a027-metrics-certs\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:40:31.225817 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.225780 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cda14550-2475-4c54-8f87-cba74f09a027-maas-api-tls\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:40:31.225817 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.225788 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ebed04-c936-4a96-8585-8a6885549752-plugins-config-volume\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:40:31.225817 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.225797 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-646nj\" (UniqueName: \"kubernetes.io/projected/c9ebed04-c936-4a96-8585-8a6885549752-kube-api-access-646nj\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:40:31.511637 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.511605 2572 generic.go:356] "Generic (PLEG): container finished" podID="cda14550-2475-4c54-8f87-cba74f09a027" containerID="f6ed5b51ba40478bfc663f236266a841cff2a5a2ffae7264b033d95d5c95a31f" exitCode=0 Sep 08 15:40:31.511804 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.511651 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" Sep 08 15:40:31.511804 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.511690 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" event={"ID":"cda14550-2475-4c54-8f87-cba74f09a027","Type":"ContainerDied","Data":"f6ed5b51ba40478bfc663f236266a841cff2a5a2ffae7264b033d95d5c95a31f"} Sep 08 15:40:31.511804 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.511749 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5" event={"ID":"cda14550-2475-4c54-8f87-cba74f09a027","Type":"ContainerDied","Data":"e4b92e4faafeedecc604ec98d5d7379f4210caa4f7debaa01f0f9dc2e586a917"} Sep 08 15:40:31.511804 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.511783 2572 scope.go:117] "RemoveContainer" containerID="f6ed5b51ba40478bfc663f236266a841cff2a5a2ffae7264b033d95d5c95a31f" Sep 08 15:40:31.513053 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.513026 2572 generic.go:356] "Generic (PLEG): container finished" podID="c9ebed04-c936-4a96-8585-8a6885549752" containerID="3f1f479f55a76b842460f123cbc5a96e91474262ee8d9ec78cb6e0256cfc0b37" exitCode=0 Sep 08 15:40:31.513203 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.513073 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb" Sep 08 15:40:31.513203 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.513123 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb" event={"ID":"c9ebed04-c936-4a96-8585-8a6885549752","Type":"ContainerDied","Data":"3f1f479f55a76b842460f123cbc5a96e91474262ee8d9ec78cb6e0256cfc0b37"} Sep 08 15:40:31.513203 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.513173 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb" event={"ID":"c9ebed04-c936-4a96-8585-8a6885549752","Type":"ContainerDied","Data":"01ec947aabba2b9abe9f2b601980dfa318802aebfe6bee8823c875cd4b2bf342"} Sep 08 15:40:31.521628 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.521612 2572 scope.go:117] "RemoveContainer" containerID="f6ed5b51ba40478bfc663f236266a841cff2a5a2ffae7264b033d95d5c95a31f" Sep 08 15:40:31.521917 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:40:31.521899 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ed5b51ba40478bfc663f236266a841cff2a5a2ffae7264b033d95d5c95a31f\": container with ID starting with f6ed5b51ba40478bfc663f236266a841cff2a5a2ffae7264b033d95d5c95a31f not found: ID does not exist" containerID="f6ed5b51ba40478bfc663f236266a841cff2a5a2ffae7264b033d95d5c95a31f" Sep 08 15:40:31.521980 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.521923 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ed5b51ba40478bfc663f236266a841cff2a5a2ffae7264b033d95d5c95a31f"} err="failed to get container status \"f6ed5b51ba40478bfc663f236266a841cff2a5a2ffae7264b033d95d5c95a31f\": rpc error: code = NotFound desc = could not find container \"f6ed5b51ba40478bfc663f236266a841cff2a5a2ffae7264b033d95d5c95a31f\": container with ID starting with f6ed5b51ba40478bfc663f236266a841cff2a5a2ffae7264b033d95d5c95a31f not found: ID does not exist" Sep 08 15:40:31.521980 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.521940 2572 scope.go:117] "RemoveContainer" containerID="3f1f479f55a76b842460f123cbc5a96e91474262ee8d9ec78cb6e0256cfc0b37" Sep 08 15:40:31.530905 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.530888 2572 scope.go:117] "RemoveContainer" containerID="3f1f479f55a76b842460f123cbc5a96e91474262ee8d9ec78cb6e0256cfc0b37" Sep 08 15:40:31.531175 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:40:31.531158 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f1f479f55a76b842460f123cbc5a96e91474262ee8d9ec78cb6e0256cfc0b37\": container with ID starting with 3f1f479f55a76b842460f123cbc5a96e91474262ee8d9ec78cb6e0256cfc0b37 not found: ID does not exist" containerID="3f1f479f55a76b842460f123cbc5a96e91474262ee8d9ec78cb6e0256cfc0b37" Sep 08 15:40:31.531218 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.531182 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f1f479f55a76b842460f123cbc5a96e91474262ee8d9ec78cb6e0256cfc0b37"} err="failed to get container status \"3f1f479f55a76b842460f123cbc5a96e91474262ee8d9ec78cb6e0256cfc0b37\": rpc error: code = NotFound desc = could not find container \"3f1f479f55a76b842460f123cbc5a96e91474262ee8d9ec78cb6e0256cfc0b37\": container with ID starting with 3f1f479f55a76b842460f123cbc5a96e91474262ee8d9ec78cb6e0256cfc0b37 not found: ID does not exist" Sep 08 15:40:31.546746 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.546712 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb"] Sep 08 15:40:31.550047 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.550023 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a2c80-58f55dffcc-pczlb"] Sep 08 15:40:31.580409 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.580384 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5"] Sep 08 15:40:31.585397 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:31.585375 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a2c80-6b44697846-xvwl5"] Sep 08 15:40:32.370656 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:32.370616 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9ebed04-c936-4a96-8585-8a6885549752" path="/var/lib/kubelet/pods/c9ebed04-c936-4a96-8585-8a6885549752/volumes" Sep 08 15:40:32.371179 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:40:32.371160 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cda14550-2475-4c54-8f87-cba74f09a027" path="/var/lib/kubelet/pods/cda14550-2475-4c54-8f87-cba74f09a027/volumes" Sep 08 15:41:35.956220 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956177 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9dea65-xcxhj"] Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956534 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f80b5111-6cdf-4b32-8f35-81f5c0eab644" containerName="payload-pre-processing" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956549 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f80b5111-6cdf-4b32-8f35-81f5c0eab644" containerName="payload-pre-processing" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956557 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5583bd4-423e-412a-8a6c-911422cc02c5" containerName="maas-api" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956563 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5583bd4-423e-412a-8a6c-911422cc02c5" containerName="maas-api" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956572 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd385e6-8d84-44be-89cb-84e1a6635ab3" containerName="payload-pre-processing" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956577 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd385e6-8d84-44be-89cb-84e1a6635ab3" containerName="payload-pre-processing" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956590 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cda14550-2475-4c54-8f87-cba74f09a027" containerName="maas-api" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956596 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cda14550-2475-4c54-8f87-cba74f09a027" containerName="maas-api" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956602 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4a58b42-1660-4853-987e-c354a0887598" containerName="payload-processing" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956607 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4a58b42-1660-4853-987e-c354a0887598" containerName="payload-processing" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956616 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9ebed04-c936-4a96-8585-8a6885549752" containerName="payload-processing" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956622 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9ebed04-c936-4a96-8585-8a6885549752" containerName="payload-processing" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956633 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40e53b91-bb2d-44c9-ab1e-514170090912" containerName="istio-proxy" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956638 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="40e53b91-bb2d-44c9-ab1e-514170090912" containerName="istio-proxy" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956644 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40e53b91-bb2d-44c9-ab1e-514170090912" containerName="istio-proxy" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956648 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="40e53b91-bb2d-44c9-ab1e-514170090912" containerName="istio-proxy" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956698 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cda14550-2475-4c54-8f87-cba74f09a027" containerName="maas-api" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956705 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bd385e6-8d84-44be-89cb-84e1a6635ab3" containerName="payload-pre-processing" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956711 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9ebed04-c936-4a96-8585-8a6885549752" containerName="payload-processing" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956717 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f80b5111-6cdf-4b32-8f35-81f5c0eab644" containerName="payload-pre-processing" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956723 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="40e53b91-bb2d-44c9-ab1e-514170090912" containerName="istio-proxy" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956729 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="40e53b91-bb2d-44c9-ab1e-514170090912" containerName="istio-proxy" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956737 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4a58b42-1660-4853-987e-c354a0887598" containerName="payload-processing" Sep 08 15:41:35.956748 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.956743 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5583bd4-423e-412a-8a6c-911422cc02c5" containerName="maas-api" Sep 08 15:41:35.959793 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.959770 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9dea65-xcxhj" Sep 08 15:41:35.962567 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.962540 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8qqvg\"" Sep 08 15:41:35.968351 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.968317 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9dea65-xcxhj"] Sep 08 15:41:35.999614 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:35.999560 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/435ddd50-3461-4e22-b8c7-fee775605714-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9dea65-xcxhj\" (UID: \"435ddd50-3461-4e22-b8c7-fee775605714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9dea65-xcxhj" Sep 08 15:41:36.100352 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:36.100286 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/435ddd50-3461-4e22-b8c7-fee775605714-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9dea65-xcxhj\" (UID: \"435ddd50-3461-4e22-b8c7-fee775605714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9dea65-xcxhj" Sep 08 15:41:36.101179 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:36.101151 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/435ddd50-3461-4e22-b8c7-fee775605714-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9dea65-xcxhj\" (UID: \"435ddd50-3461-4e22-b8c7-fee775605714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9dea65-xcxhj" Sep 08 15:41:36.271947 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:36.271909 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9dea65-xcxhj" Sep 08 15:41:36.550438 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:36.550408 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9dea65-xcxhj"] Sep 08 15:41:36.553256 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:41:36.553211 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435ddd50_3461_4e22_b8c7_fee775605714.slice/crio-26f7c3d81a1f152b9525cfa73d38014575719b6af182a563ec0598becb689863 WatchSource:0}: Error finding container 26f7c3d81a1f152b9525cfa73d38014575719b6af182a563ec0598becb689863: Status 404 returned error can't find the container with id 26f7c3d81a1f152b9525cfa73d38014575719b6af182a563ec0598becb689863 Sep 08 15:41:36.824621 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:36.824526 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9dea65-xcxhj" event={"ID":"435ddd50-3461-4e22-b8c7-fee775605714","Type":"ContainerStarted","Data":"26f7c3d81a1f152b9525cfa73d38014575719b6af182a563ec0598becb689863"} Sep 08 15:41:37.831164 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:37.831129 2572 generic.go:356] "Generic (PLEG): container finished" podID="435ddd50-3461-4e22-b8c7-fee775605714" containerID="f296360dd9b894dcd858ca91be3d79f48e40011a0c1ba279e78ce214e3416fb5" exitCode=0 Sep 08 15:41:37.831587 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:37.831194 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9dea65-xcxhj" event={"ID":"435ddd50-3461-4e22-b8c7-fee775605714","Type":"ContainerDied","Data":"f296360dd9b894dcd858ca91be3d79f48e40011a0c1ba279e78ce214e3416fb5"} Sep 08 15:41:38.981738 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:38.981705 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9dea65-xcxhj" Sep 08 15:41:39.028661 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:39.028623 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/435ddd50-3461-4e22-b8c7-fee775605714-maas-api-service-ca\") pod \"435ddd50-3461-4e22-b8c7-fee775605714\" (UID: \"435ddd50-3461-4e22-b8c7-fee775605714\") " Sep 08 15:41:39.028993 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:39.028970 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/435ddd50-3461-4e22-b8c7-fee775605714-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "435ddd50-3461-4e22-b8c7-fee775605714" (UID: "435ddd50-3461-4e22-b8c7-fee775605714"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:41:39.129558 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:39.129464 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/435ddd50-3461-4e22-b8c7-fee775605714-maas-api-service-ca\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:41:39.840750 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:39.840711 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9dea65-xcxhj" event={"ID":"435ddd50-3461-4e22-b8c7-fee775605714","Type":"ContainerDied","Data":"26f7c3d81a1f152b9525cfa73d38014575719b6af182a563ec0598becb689863"} Sep 08 15:41:39.840750 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:39.840738 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9dea65-xcxhj" Sep 08 15:41:39.840965 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:39.840766 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26f7c3d81a1f152b9525cfa73d38014575719b6af182a563ec0598becb689863" Sep 08 15:41:46.806767 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:46.806730 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9dea65-xcxhj"] Sep 08 15:41:46.810308 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:46.810263 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9dea65-xcxhj"] Sep 08 15:41:48.372885 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:48.372842 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="435ddd50-3461-4e22-b8c7-fee775605714" path="/var/lib/kubelet/pods/435ddd50-3461-4e22-b8c7-fee775605714/volumes" Sep 08 15:41:54.373505 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:54.373412 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6"] Sep 08 15:41:54.373921 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:54.373680 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" podUID="a792ae0a-d83e-434f-9812-7d0c60892622" containerName="istio-proxy" containerID="cri-o://4e9de8912f4c9885d939f50d1520b59532e0237999735beb3a5393c4a35e9d15" gracePeriod=30 Sep 08 15:41:57.836766 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:57.836723 2572 patch_prober.go:28] interesting pod/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 15:41:57.837171 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:57.836802 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" podUID="a792ae0a-d83e-434f-9812-7d0c60892622" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 15:41:59.620609 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.620579 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:41:59.723730 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.723627 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-istio-envoy\") pod \"a792ae0a-d83e-434f-9812-7d0c60892622\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " Sep 08 15:41:59.723730 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.723663 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-workload-socket\") pod \"a792ae0a-d83e-434f-9812-7d0c60892622\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " Sep 08 15:41:59.723730 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.723688 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m47gc\" (UniqueName: \"kubernetes.io/projected/a792ae0a-d83e-434f-9812-7d0c60892622-kube-api-access-m47gc\") pod \"a792ae0a-d83e-434f-9812-7d0c60892622\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " Sep 08 15:41:59.723730 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.723703 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-workload-certs\") pod \"a792ae0a-d83e-434f-9812-7d0c60892622\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " Sep 08 15:41:59.723730 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.723738 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a792ae0a-d83e-434f-9812-7d0c60892622-istio-podinfo\") pod \"a792ae0a-d83e-434f-9812-7d0c60892622\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " Sep 08 15:41:59.724116 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.723777 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-credential-socket\") pod \"a792ae0a-d83e-434f-9812-7d0c60892622\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " Sep 08 15:41:59.724116 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.723797 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-istio-data\") pod \"a792ae0a-d83e-434f-9812-7d0c60892622\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " Sep 08 15:41:59.724116 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.723820 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a792ae0a-d83e-434f-9812-7d0c60892622-istiod-ca-cert\") pod \"a792ae0a-d83e-434f-9812-7d0c60892622\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " Sep 08 15:41:59.724116 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.723855 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a792ae0a-d83e-434f-9812-7d0c60892622-istio-token\") pod \"a792ae0a-d83e-434f-9812-7d0c60892622\" (UID: \"a792ae0a-d83e-434f-9812-7d0c60892622\") " Sep 08 15:41:59.724116 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.724029 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a792ae0a-d83e-434f-9812-7d0c60892622" (UID: "a792ae0a-d83e-434f-9812-7d0c60892622"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:41:59.724116 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.724095 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a792ae0a-d83e-434f-9812-7d0c60892622" (UID: "a792ae0a-d83e-434f-9812-7d0c60892622"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:41:59.724435 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.724122 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-istio-data" (OuterVolumeSpecName: "istio-data") pod "a792ae0a-d83e-434f-9812-7d0c60892622" (UID: "a792ae0a-d83e-434f-9812-7d0c60892622"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:41:59.724435 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.724158 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-workload-socket\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:41:59.724435 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.724388 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a792ae0a-d83e-434f-9812-7d0c60892622" (UID: "a792ae0a-d83e-434f-9812-7d0c60892622"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:41:59.724621 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.724460 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a792ae0a-d83e-434f-9812-7d0c60892622-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a792ae0a-d83e-434f-9812-7d0c60892622" (UID: "a792ae0a-d83e-434f-9812-7d0c60892622"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:41:59.726207 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.726180 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a792ae0a-d83e-434f-9812-7d0c60892622-kube-api-access-m47gc" (OuterVolumeSpecName: "kube-api-access-m47gc") pod "a792ae0a-d83e-434f-9812-7d0c60892622" (UID: "a792ae0a-d83e-434f-9812-7d0c60892622"). InnerVolumeSpecName "kube-api-access-m47gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:41:59.726324 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.726234 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a792ae0a-d83e-434f-9812-7d0c60892622-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a792ae0a-d83e-434f-9812-7d0c60892622" (UID: "a792ae0a-d83e-434f-9812-7d0c60892622"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 15:41:59.726681 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.726663 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a792ae0a-d83e-434f-9812-7d0c60892622" (UID: "a792ae0a-d83e-434f-9812-7d0c60892622"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:41:59.726739 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.726694 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a792ae0a-d83e-434f-9812-7d0c60892622-istio-token" (OuterVolumeSpecName: "istio-token") pod "a792ae0a-d83e-434f-9812-7d0c60892622" (UID: "a792ae0a-d83e-434f-9812-7d0c60892622"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:41:59.825061 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.825015 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m47gc\" (UniqueName: \"kubernetes.io/projected/a792ae0a-d83e-434f-9812-7d0c60892622-kube-api-access-m47gc\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:41:59.825061 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.825053 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-workload-certs\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:41:59.825266 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.825079 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a792ae0a-d83e-434f-9812-7d0c60892622-istio-podinfo\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:41:59.825266 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.825088 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-credential-socket\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:41:59.825266 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.825096 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-istio-data\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:41:59.825266 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.825104 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a792ae0a-d83e-434f-9812-7d0c60892622-istiod-ca-cert\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:41:59.825266 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.825113 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a792ae0a-d83e-434f-9812-7d0c60892622-istio-token\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:41:59.825266 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.825120 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a792ae0a-d83e-434f-9812-7d0c60892622-istio-envoy\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:41:59.941299 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.941258 2572 generic.go:356] "Generic (PLEG): container finished" podID="a792ae0a-d83e-434f-9812-7d0c60892622" containerID="4e9de8912f4c9885d939f50d1520b59532e0237999735beb3a5393c4a35e9d15" exitCode=0 Sep 08 15:41:59.941471 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.941318 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" Sep 08 15:41:59.941471 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.941337 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" event={"ID":"a792ae0a-d83e-434f-9812-7d0c60892622","Type":"ContainerDied","Data":"4e9de8912f4c9885d939f50d1520b59532e0237999735beb3a5393c4a35e9d15"} Sep 08 15:41:59.941471 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.941384 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6" event={"ID":"a792ae0a-d83e-434f-9812-7d0c60892622","Type":"ContainerDied","Data":"e5fc0590de5ff1881e6834c68573352ff432cff7c1a39ebaf872cb325ca683e8"} Sep 08 15:41:59.941471 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.941406 2572 scope.go:117] "RemoveContainer" containerID="4e9de8912f4c9885d939f50d1520b59532e0237999735beb3a5393c4a35e9d15" Sep 08 15:41:59.950840 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.950810 2572 scope.go:117] "RemoveContainer" containerID="0effbc8e01efe382488400290bc498f17c1c7972aaca98b23d5562c7f9edf4a9" Sep 08 15:41:59.959511 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.959470 2572 scope.go:117] "RemoveContainer" containerID="4e9de8912f4c9885d939f50d1520b59532e0237999735beb3a5393c4a35e9d15" Sep 08 15:41:59.959892 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:41:59.959869 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e9de8912f4c9885d939f50d1520b59532e0237999735beb3a5393c4a35e9d15\": container with ID starting with 4e9de8912f4c9885d939f50d1520b59532e0237999735beb3a5393c4a35e9d15 not found: ID does not exist" containerID="4e9de8912f4c9885d939f50d1520b59532e0237999735beb3a5393c4a35e9d15" Sep 08 15:41:59.959951 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.959903 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e9de8912f4c9885d939f50d1520b59532e0237999735beb3a5393c4a35e9d15"} err="failed to get container status \"4e9de8912f4c9885d939f50d1520b59532e0237999735beb3a5393c4a35e9d15\": rpc error: code = NotFound desc = could not find container \"4e9de8912f4c9885d939f50d1520b59532e0237999735beb3a5393c4a35e9d15\": container with ID starting with 4e9de8912f4c9885d939f50d1520b59532e0237999735beb3a5393c4a35e9d15 not found: ID does not exist" Sep 08 15:41:59.959951 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.959926 2572 scope.go:117] "RemoveContainer" containerID="0effbc8e01efe382488400290bc498f17c1c7972aaca98b23d5562c7f9edf4a9" Sep 08 15:41:59.960229 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:41:59.960210 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0effbc8e01efe382488400290bc498f17c1c7972aaca98b23d5562c7f9edf4a9\": container with ID starting with 0effbc8e01efe382488400290bc498f17c1c7972aaca98b23d5562c7f9edf4a9 not found: ID does not exist" containerID="0effbc8e01efe382488400290bc498f17c1c7972aaca98b23d5562c7f9edf4a9" Sep 08 15:41:59.960301 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.960241 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0effbc8e01efe382488400290bc498f17c1c7972aaca98b23d5562c7f9edf4a9"} err="failed to get container status \"0effbc8e01efe382488400290bc498f17c1c7972aaca98b23d5562c7f9edf4a9\": rpc error: code = NotFound desc = could not find container \"0effbc8e01efe382488400290bc498f17c1c7972aaca98b23d5562c7f9edf4a9\": container with ID starting with 0effbc8e01efe382488400290bc498f17c1c7972aaca98b23d5562c7f9edf4a9 not found: ID does not exist" Sep 08 15:41:59.966944 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.966866 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6"] Sep 08 15:41:59.968944 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:41:59.968921 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9dea65-openshift-default-77cd79d5f6-v6jg6"] Sep 08 15:42:00.373859 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:00.373824 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a792ae0a-d83e-434f-9812-7d0c60892622" path="/var/lib/kubelet/pods/a792ae0a-d83e-434f-9812-7d0c60892622/volumes" Sep 08 15:42:05.169464 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.169423 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg"] Sep 08 15:42:05.169990 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.169737 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" podUID="de86dadc-03ed-4180-ac3c-85216121be05" containerName="maas-api" containerID="cri-o://ca1fd199dfb07afdc01a896b025db8ca4e992da5887043c3b84619221bf66651" gracePeriod=30 Sep 08 15:42:05.229454 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.229411 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj"] Sep 08 15:42:05.229768 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.229731 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj" podUID="6b8901ef-372b-4fef-a6f1-92102ef628bd" containerName="payload-processing" containerID="cri-o://06d073421158ef6607bd047a1349043713bdd51429d754dde60ab1821c34d39a" gracePeriod=30 Sep 08 15:42:05.492686 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.492617 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" Sep 08 15:42:05.523727 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.523695 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj" Sep 08 15:42:05.681116 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.681078 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpxkc\" (UniqueName: \"kubernetes.io/projected/de86dadc-03ed-4180-ac3c-85216121be05-kube-api-access-cpxkc\") pod \"de86dadc-03ed-4180-ac3c-85216121be05\" (UID: \"de86dadc-03ed-4180-ac3c-85216121be05\") " Sep 08 15:42:05.681325 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.681148 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b8901ef-372b-4fef-a6f1-92102ef628bd-plugins-config-volume\") pod \"6b8901ef-372b-4fef-a6f1-92102ef628bd\" (UID: \"6b8901ef-372b-4fef-a6f1-92102ef628bd\") " Sep 08 15:42:05.681325 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.681171 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de86dadc-03ed-4180-ac3c-85216121be05-metrics-certs\") pod \"de86dadc-03ed-4180-ac3c-85216121be05\" (UID: \"de86dadc-03ed-4180-ac3c-85216121be05\") " Sep 08 15:42:05.681325 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.681218 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de86dadc-03ed-4180-ac3c-85216121be05-maas-api-tls\") pod \"de86dadc-03ed-4180-ac3c-85216121be05\" (UID: \"de86dadc-03ed-4180-ac3c-85216121be05\") " Sep 08 15:42:05.681325 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.681267 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de86dadc-03ed-4180-ac3c-85216121be05-openshift-service-ca\") pod \"de86dadc-03ed-4180-ac3c-85216121be05\" (UID: \"de86dadc-03ed-4180-ac3c-85216121be05\") " Sep 08 15:42:05.681325 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.681305 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv9sk\" (UniqueName: \"kubernetes.io/projected/6b8901ef-372b-4fef-a6f1-92102ef628bd-kube-api-access-rv9sk\") pod \"6b8901ef-372b-4fef-a6f1-92102ef628bd\" (UID: \"6b8901ef-372b-4fef-a6f1-92102ef628bd\") " Sep 08 15:42:05.681642 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.681608 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b8901ef-372b-4fef-a6f1-92102ef628bd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b8901ef-372b-4fef-a6f1-92102ef628bd" (UID: "6b8901ef-372b-4fef-a6f1-92102ef628bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:42:05.681699 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.681682 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de86dadc-03ed-4180-ac3c-85216121be05-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "de86dadc-03ed-4180-ac3c-85216121be05" (UID: "de86dadc-03ed-4180-ac3c-85216121be05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:42:05.683722 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.683647 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de86dadc-03ed-4180-ac3c-85216121be05-kube-api-access-cpxkc" (OuterVolumeSpecName: "kube-api-access-cpxkc") pod "de86dadc-03ed-4180-ac3c-85216121be05" (UID: "de86dadc-03ed-4180-ac3c-85216121be05"). InnerVolumeSpecName "kube-api-access-cpxkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:42:05.683722 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.683647 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de86dadc-03ed-4180-ac3c-85216121be05-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "de86dadc-03ed-4180-ac3c-85216121be05" (UID: "de86dadc-03ed-4180-ac3c-85216121be05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 15:42:05.683722 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.683654 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b8901ef-372b-4fef-a6f1-92102ef628bd-kube-api-access-rv9sk" (OuterVolumeSpecName: "kube-api-access-rv9sk") pod "6b8901ef-372b-4fef-a6f1-92102ef628bd" (UID: "6b8901ef-372b-4fef-a6f1-92102ef628bd"). InnerVolumeSpecName "kube-api-access-rv9sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:42:05.683916 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.683724 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de86dadc-03ed-4180-ac3c-85216121be05-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "de86dadc-03ed-4180-ac3c-85216121be05" (UID: "de86dadc-03ed-4180-ac3c-85216121be05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 15:42:05.782519 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.782444 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b8901ef-372b-4fef-a6f1-92102ef628bd-plugins-config-volume\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:42:05.782519 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.782480 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de86dadc-03ed-4180-ac3c-85216121be05-metrics-certs\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:42:05.782519 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.782521 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de86dadc-03ed-4180-ac3c-85216121be05-maas-api-tls\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:42:05.782519 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.782533 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de86dadc-03ed-4180-ac3c-85216121be05-openshift-service-ca\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:42:05.782818 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.782542 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv9sk\" (UniqueName: \"kubernetes.io/projected/6b8901ef-372b-4fef-a6f1-92102ef628bd-kube-api-access-rv9sk\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:42:05.782818 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.782553 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpxkc\" (UniqueName: \"kubernetes.io/projected/de86dadc-03ed-4180-ac3c-85216121be05-kube-api-access-cpxkc\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:42:05.972019 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.971922 2572 generic.go:356] "Generic (PLEG): container finished" podID="6b8901ef-372b-4fef-a6f1-92102ef628bd" containerID="06d073421158ef6607bd047a1349043713bdd51429d754dde60ab1821c34d39a" exitCode=0 Sep 08 15:42:05.972019 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.971971 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj" Sep 08 15:42:05.972019 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.972006 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj" event={"ID":"6b8901ef-372b-4fef-a6f1-92102ef628bd","Type":"ContainerDied","Data":"06d073421158ef6607bd047a1349043713bdd51429d754dde60ab1821c34d39a"} Sep 08 15:42:05.972334 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.972061 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj" event={"ID":"6b8901ef-372b-4fef-a6f1-92102ef628bd","Type":"ContainerDied","Data":"54fc781e1c8e5a40f12a19752e833a7a96568f4a31dd5ed3c3f71649b58a3765"} Sep 08 15:42:05.972334 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.972085 2572 scope.go:117] "RemoveContainer" containerID="06d073421158ef6607bd047a1349043713bdd51429d754dde60ab1821c34d39a" Sep 08 15:42:05.973648 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.973625 2572 generic.go:356] "Generic (PLEG): container finished" podID="de86dadc-03ed-4180-ac3c-85216121be05" containerID="ca1fd199dfb07afdc01a896b025db8ca4e992da5887043c3b84619221bf66651" exitCode=0 Sep 08 15:42:05.973771 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.973662 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" event={"ID":"de86dadc-03ed-4180-ac3c-85216121be05","Type":"ContainerDied","Data":"ca1fd199dfb07afdc01a896b025db8ca4e992da5887043c3b84619221bf66651"} Sep 08 15:42:05.973771 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.973675 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" Sep 08 15:42:05.973771 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.973709 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg" event={"ID":"de86dadc-03ed-4180-ac3c-85216121be05","Type":"ContainerDied","Data":"b74d2aa04d7566f541db0e118cbb4ff58562c84f4e192783a199185ebbb2f69a"} Sep 08 15:42:05.982441 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.982421 2572 scope.go:117] "RemoveContainer" containerID="06d073421158ef6607bd047a1349043713bdd51429d754dde60ab1821c34d39a" Sep 08 15:42:05.982806 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:42:05.982785 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d073421158ef6607bd047a1349043713bdd51429d754dde60ab1821c34d39a\": container with ID starting with 06d073421158ef6607bd047a1349043713bdd51429d754dde60ab1821c34d39a not found: ID does not exist" containerID="06d073421158ef6607bd047a1349043713bdd51429d754dde60ab1821c34d39a" Sep 08 15:42:05.982865 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.982814 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d073421158ef6607bd047a1349043713bdd51429d754dde60ab1821c34d39a"} err="failed to get container status \"06d073421158ef6607bd047a1349043713bdd51429d754dde60ab1821c34d39a\": rpc error: code = NotFound desc = could not find container \"06d073421158ef6607bd047a1349043713bdd51429d754dde60ab1821c34d39a\": container with ID starting with 06d073421158ef6607bd047a1349043713bdd51429d754dde60ab1821c34d39a not found: ID does not exist" Sep 08 15:42:05.982865 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.982832 2572 scope.go:117] "RemoveContainer" containerID="ca1fd199dfb07afdc01a896b025db8ca4e992da5887043c3b84619221bf66651" Sep 08 15:42:05.991672 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.991591 2572 scope.go:117] "RemoveContainer" containerID="ca1fd199dfb07afdc01a896b025db8ca4e992da5887043c3b84619221bf66651" Sep 08 15:42:05.991982 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:42:05.991958 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca1fd199dfb07afdc01a896b025db8ca4e992da5887043c3b84619221bf66651\": container with ID starting with ca1fd199dfb07afdc01a896b025db8ca4e992da5887043c3b84619221bf66651 not found: ID does not exist" containerID="ca1fd199dfb07afdc01a896b025db8ca4e992da5887043c3b84619221bf66651" Sep 08 15:42:05.992089 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.991988 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca1fd199dfb07afdc01a896b025db8ca4e992da5887043c3b84619221bf66651"} err="failed to get container status \"ca1fd199dfb07afdc01a896b025db8ca4e992da5887043c3b84619221bf66651\": rpc error: code = NotFound desc = could not find container \"ca1fd199dfb07afdc01a896b025db8ca4e992da5887043c3b84619221bf66651\": container with ID starting with ca1fd199dfb07afdc01a896b025db8ca4e992da5887043c3b84619221bf66651 not found: ID does not exist" Sep 08 15:42:05.997934 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:05.997903 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg"] Sep 08 15:42:06.001538 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:06.001512 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e082e-7898cd7bc5-9swrg"] Sep 08 15:42:06.012550 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:06.012518 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj"] Sep 08 15:42:06.016259 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:06.016219 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2e082e-5659fb8659-twtlj"] Sep 08 15:42:06.371107 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:06.371070 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b8901ef-372b-4fef-a6f1-92102ef628bd" path="/var/lib/kubelet/pods/6b8901ef-372b-4fef-a6f1-92102ef628bd/volumes" Sep 08 15:42:06.371525 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:06.371474 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de86dadc-03ed-4180-ac3c-85216121be05" path="/var/lib/kubelet/pods/de86dadc-03ed-4180-ac3c-85216121be05/volumes" Sep 08 15:42:12.689604 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.689568 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bc9cdf-xt72f"] Sep 08 15:42:12.690048 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.689849 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="435ddd50-3461-4e22-b8c7-fee775605714" containerName="revoke-keys" Sep 08 15:42:12.690048 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.689859 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="435ddd50-3461-4e22-b8c7-fee775605714" containerName="revoke-keys" Sep 08 15:42:12.690048 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.689873 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a792ae0a-d83e-434f-9812-7d0c60892622" containerName="istio-proxy" Sep 08 15:42:12.690048 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.689878 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a792ae0a-d83e-434f-9812-7d0c60892622" containerName="istio-proxy" Sep 08 15:42:12.690048 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.689890 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b8901ef-372b-4fef-a6f1-92102ef628bd" containerName="payload-processing" Sep 08 15:42:12.690048 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.689895 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b8901ef-372b-4fef-a6f1-92102ef628bd" containerName="payload-processing" Sep 08 15:42:12.690048 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.689904 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a792ae0a-d83e-434f-9812-7d0c60892622" containerName="istio-proxy" Sep 08 15:42:12.690048 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.689909 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a792ae0a-d83e-434f-9812-7d0c60892622" containerName="istio-proxy" Sep 08 15:42:12.690048 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.689915 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de86dadc-03ed-4180-ac3c-85216121be05" containerName="maas-api" Sep 08 15:42:12.690048 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.689920 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="de86dadc-03ed-4180-ac3c-85216121be05" containerName="maas-api" Sep 08 15:42:12.690048 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.689966 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a792ae0a-d83e-434f-9812-7d0c60892622" containerName="istio-proxy" Sep 08 15:42:12.690048 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.689974 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="435ddd50-3461-4e22-b8c7-fee775605714" containerName="revoke-keys" Sep 08 15:42:12.690048 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.689981 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b8901ef-372b-4fef-a6f1-92102ef628bd" containerName="payload-processing" Sep 08 15:42:12.690048 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.689987 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a792ae0a-d83e-434f-9812-7d0c60892622" containerName="istio-proxy" Sep 08 15:42:12.690048 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.689993 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="de86dadc-03ed-4180-ac3c-85216121be05" containerName="maas-api" Sep 08 15:42:12.694202 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.694185 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bc9cdf-xt72f" Sep 08 15:42:12.697617 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.697594 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 15:42:12.697731 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.697643 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8qqvg\"" Sep 08 15:42:12.703164 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.703144 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bc9cdf-xt72f"] Sep 08 15:42:12.739263 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.739237 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49687f3c-2a33-44af-b64c-142941ab0f7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bc9cdf-xt72f\" (UID: \"49687f3c-2a33-44af-b64c-142941ab0f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bc9cdf-xt72f" Sep 08 15:42:12.839872 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.839844 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49687f3c-2a33-44af-b64c-142941ab0f7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bc9cdf-xt72f\" (UID: \"49687f3c-2a33-44af-b64c-142941ab0f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bc9cdf-xt72f" Sep 08 15:42:12.840395 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:12.840377 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49687f3c-2a33-44af-b64c-142941ab0f7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bc9cdf-xt72f\" (UID: \"49687f3c-2a33-44af-b64c-142941ab0f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bc9cdf-xt72f" Sep 08 15:42:13.005218 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:13.005194 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bc9cdf-xt72f" Sep 08 15:42:13.276137 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:13.276095 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bc9cdf-xt72f"] Sep 08 15:42:13.278435 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:42:13.278406 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49687f3c_2a33_44af_b64c_142941ab0f7d.slice/crio-25fe3ebc360be2822e228698e04ef152c539942f31558f9cb2a1ff00a5819032 WatchSource:0}: Error finding container 25fe3ebc360be2822e228698e04ef152c539942f31558f9cb2a1ff00a5819032: Status 404 returned error can't find the container with id 25fe3ebc360be2822e228698e04ef152c539942f31558f9cb2a1ff00a5819032 Sep 08 15:42:14.013923 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:14.013892 2572 generic.go:356] "Generic (PLEG): container finished" podID="49687f3c-2a33-44af-b64c-142941ab0f7d" containerID="b633bd153118cf75142c65d323969cabe34e12fdf681ab736efb3bfcbe9eb612" exitCode=0 Sep 08 15:42:14.014249 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:14.013949 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bc9cdf-xt72f" event={"ID":"49687f3c-2a33-44af-b64c-142941ab0f7d","Type":"ContainerDied","Data":"b633bd153118cf75142c65d323969cabe34e12fdf681ab736efb3bfcbe9eb612"} Sep 08 15:42:14.014249 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:14.014003 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bc9cdf-xt72f" event={"ID":"49687f3c-2a33-44af-b64c-142941ab0f7d","Type":"ContainerStarted","Data":"25fe3ebc360be2822e228698e04ef152c539942f31558f9cb2a1ff00a5819032"} Sep 08 15:42:15.149170 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:15.149144 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bc9cdf-xt72f" Sep 08 15:42:15.157861 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:15.157842 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49687f3c-2a33-44af-b64c-142941ab0f7d-maas-api-service-ca\") pod \"49687f3c-2a33-44af-b64c-142941ab0f7d\" (UID: \"49687f3c-2a33-44af-b64c-142941ab0f7d\") " Sep 08 15:42:15.158146 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:15.158125 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49687f3c-2a33-44af-b64c-142941ab0f7d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "49687f3c-2a33-44af-b64c-142941ab0f7d" (UID: "49687f3c-2a33-44af-b64c-142941ab0f7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:42:15.258888 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:15.258853 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49687f3c-2a33-44af-b64c-142941ab0f7d-maas-api-service-ca\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:42:16.026109 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:16.026071 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bc9cdf-xt72f" Sep 08 15:42:16.026341 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:16.026072 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bc9cdf-xt72f" event={"ID":"49687f3c-2a33-44af-b64c-142941ab0f7d","Type":"ContainerDied","Data":"25fe3ebc360be2822e228698e04ef152c539942f31558f9cb2a1ff00a5819032"} Sep 08 15:42:16.026341 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:16.026212 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25fe3ebc360be2822e228698e04ef152c539942f31558f9cb2a1ff00a5819032" Sep 08 15:42:23.700975 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:23.700930 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bc9cdf-xt72f"] Sep 08 15:42:23.707673 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:23.707639 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bc9cdf-xt72f"] Sep 08 15:42:24.371195 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:24.371157 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49687f3c-2a33-44af-b64c-142941ab0f7d" path="/var/lib/kubelet/pods/49687f3c-2a33-44af-b64c-142941ab0f7d/volumes" Sep 08 15:42:31.313486 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:31.313451 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr"] Sep 08 15:42:31.313920 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:31.313693 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" podUID="59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" containerName="istio-proxy" containerID="cri-o://4c8cb1338e20b4c4e8d9b056adc3f07aaa983765cfded6528b7fe98927a1b773" gracePeriod=30 Sep 08 15:42:36.562144 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.562116 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:42:36.654171 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.654071 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm7p7\" (UniqueName: \"kubernetes.io/projected/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-kube-api-access-qm7p7\") pod \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " Sep 08 15:42:36.654171 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.654132 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-credential-socket\") pod \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " Sep 08 15:42:36.654171 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.654151 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-token\") pod \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " Sep 08 15:42:36.654171 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.654168 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-podinfo\") pod \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " Sep 08 15:42:36.654551 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.654202 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-workload-certs\") pod \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " Sep 08 15:42:36.654551 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.654228 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-workload-socket\") pod \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " Sep 08 15:42:36.654551 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.654257 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-data\") pod \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " Sep 08 15:42:36.654551 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.654317 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-envoy\") pod \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " Sep 08 15:42:36.654551 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.654346 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istiod-ca-cert\") pod \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\" (UID: \"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2\") " Sep 08 15:42:36.654819 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.654586 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" (UID: "59b5ed25-8f05-440b-a8d2-67f56fb4b0f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:42:36.654819 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.654736 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-data" (OuterVolumeSpecName: "istio-data") pod "59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" (UID: "59b5ed25-8f05-440b-a8d2-67f56fb4b0f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:42:36.654819 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.654801 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" (UID: "59b5ed25-8f05-440b-a8d2-67f56fb4b0f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:42:36.654982 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.654900 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" (UID: "59b5ed25-8f05-440b-a8d2-67f56fb4b0f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:42:36.655045 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.654486 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" (UID: "59b5ed25-8f05-440b-a8d2-67f56fb4b0f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:42:36.656728 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.656693 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" (UID: "59b5ed25-8f05-440b-a8d2-67f56fb4b0f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 15:42:36.656986 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.656953 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-token" (OuterVolumeSpecName: "istio-token") pod "59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" (UID: "59b5ed25-8f05-440b-a8d2-67f56fb4b0f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:42:36.656986 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.656957 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-kube-api-access-qm7p7" (OuterVolumeSpecName: "kube-api-access-qm7p7") pod "59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" (UID: "59b5ed25-8f05-440b-a8d2-67f56fb4b0f2"). InnerVolumeSpecName "kube-api-access-qm7p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:42:36.657223 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.657194 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" (UID: "59b5ed25-8f05-440b-a8d2-67f56fb4b0f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 15:42:36.755010 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.754963 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm7p7\" (UniqueName: \"kubernetes.io/projected/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-kube-api-access-qm7p7\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:42:36.755010 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.755001 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-credential-socket\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:42:36.755010 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.755012 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-token\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:42:36.755010 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.755020 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-podinfo\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:42:36.755301 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.755028 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-workload-certs\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:42:36.755301 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.755037 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-workload-socket\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:42:36.755301 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.755045 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-data\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:42:36.755301 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.755053 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istio-envoy\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:42:36.755301 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:36.755060 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2-istiod-ca-cert\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:42:37.122954 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:37.122912 2572 generic.go:356] "Generic (PLEG): container finished" podID="59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" containerID="4c8cb1338e20b4c4e8d9b056adc3f07aaa983765cfded6528b7fe98927a1b773" exitCode=0 Sep 08 15:42:37.123151 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:37.122979 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" event={"ID":"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2","Type":"ContainerDied","Data":"4c8cb1338e20b4c4e8d9b056adc3f07aaa983765cfded6528b7fe98927a1b773"} Sep 08 15:42:37.123151 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:37.122988 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" Sep 08 15:42:37.123151 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:37.123067 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr" event={"ID":"59b5ed25-8f05-440b-a8d2-67f56fb4b0f2","Type":"ContainerDied","Data":"2a83cdd1930aea5efe871512c56e89dfe22b4fe55c595122e79fa01f5c85cbe7"} Sep 08 15:42:37.123151 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:37.123098 2572 scope.go:117] "RemoveContainer" containerID="4c8cb1338e20b4c4e8d9b056adc3f07aaa983765cfded6528b7fe98927a1b773" Sep 08 15:42:37.132263 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:37.132230 2572 scope.go:117] "RemoveContainer" containerID="02f0203e33ed058fefe97a2faae4578472ac4ec131b7782da790cdfe3f8c59d2" Sep 08 15:42:37.142534 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:37.142486 2572 scope.go:117] "RemoveContainer" containerID="4c8cb1338e20b4c4e8d9b056adc3f07aaa983765cfded6528b7fe98927a1b773" Sep 08 15:42:37.142914 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:42:37.142892 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c8cb1338e20b4c4e8d9b056adc3f07aaa983765cfded6528b7fe98927a1b773\": container with ID starting with 4c8cb1338e20b4c4e8d9b056adc3f07aaa983765cfded6528b7fe98927a1b773 not found: ID does not exist" containerID="4c8cb1338e20b4c4e8d9b056adc3f07aaa983765cfded6528b7fe98927a1b773" Sep 08 15:42:37.142962 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:37.142925 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8cb1338e20b4c4e8d9b056adc3f07aaa983765cfded6528b7fe98927a1b773"} err="failed to get container status \"4c8cb1338e20b4c4e8d9b056adc3f07aaa983765cfded6528b7fe98927a1b773\": rpc error: code = NotFound desc = could not find container \"4c8cb1338e20b4c4e8d9b056adc3f07aaa983765cfded6528b7fe98927a1b773\": container with ID starting with 4c8cb1338e20b4c4e8d9b056adc3f07aaa983765cfded6528b7fe98927a1b773 not found: ID does not exist" Sep 08 15:42:37.142962 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:37.142945 2572 scope.go:117] "RemoveContainer" containerID="02f0203e33ed058fefe97a2faae4578472ac4ec131b7782da790cdfe3f8c59d2" Sep 08 15:42:37.143261 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:42:37.143243 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02f0203e33ed058fefe97a2faae4578472ac4ec131b7782da790cdfe3f8c59d2\": container with ID starting with 02f0203e33ed058fefe97a2faae4578472ac4ec131b7782da790cdfe3f8c59d2 not found: ID does not exist" containerID="02f0203e33ed058fefe97a2faae4578472ac4ec131b7782da790cdfe3f8c59d2" Sep 08 15:42:37.143322 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:37.143266 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f0203e33ed058fefe97a2faae4578472ac4ec131b7782da790cdfe3f8c59d2"} err="failed to get container status \"02f0203e33ed058fefe97a2faae4578472ac4ec131b7782da790cdfe3f8c59d2\": rpc error: code = NotFound desc = could not find container \"02f0203e33ed058fefe97a2faae4578472ac4ec131b7782da790cdfe3f8c59d2\": container with ID starting with 02f0203e33ed058fefe97a2faae4578472ac4ec131b7782da790cdfe3f8c59d2 not found: ID does not exist" Sep 08 15:42:37.146662 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:37.146635 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr"] Sep 08 15:42:37.150360 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:37.150327 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bc9cdf-openshift-default-575d495f84-9chlr"] Sep 08 15:42:38.371306 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:42:38.371261 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" path="/var/lib/kubelet/pods/59b5ed25-8f05-440b-a8d2-67f56fb4b0f2/volumes" Sep 08 15:45:00.134383 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:00.134343 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814705-fkwgv"] Sep 08 15:45:00.134840 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:00.134695 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" containerName="istio-proxy" Sep 08 15:45:00.134840 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:00.134707 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" containerName="istio-proxy" Sep 08 15:45:00.134840 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:00.134717 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" containerName="istio-proxy" Sep 08 15:45:00.134840 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:00.134723 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" containerName="istio-proxy" Sep 08 15:45:00.134840 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:00.134729 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49687f3c-2a33-44af-b64c-142941ab0f7d" containerName="revoke-keys" Sep 08 15:45:00.134840 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:00.134736 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="49687f3c-2a33-44af-b64c-142941ab0f7d" containerName="revoke-keys" Sep 08 15:45:00.134840 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:00.134781 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" containerName="istio-proxy" Sep 08 15:45:00.134840 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:00.134791 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="49687f3c-2a33-44af-b64c-142941ab0f7d" containerName="revoke-keys" Sep 08 15:45:00.137660 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:00.137643 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814705-fkwgv" Sep 08 15:45:00.141753 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:00.141733 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8qqvg\"" Sep 08 15:45:00.148089 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:00.148062 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814705-fkwgv" Sep 08 15:45:00.148467 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:00.148441 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814705-fkwgv"] Sep 08 15:45:00.423968 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:00.423863 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814705-fkwgv"] Sep 08 15:45:00.426781 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:45:00.426752 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69a2a3e_ebce_4aa8_9525_536a88f2b551.slice/crio-9fc7175f63c4c0cc4b05aa96705aed23c57101e02242c8085512c92d92b24b29 WatchSource:0}: Error finding container 9fc7175f63c4c0cc4b05aa96705aed23c57101e02242c8085512c92d92b24b29: Status 404 returned error can't find the container with id 9fc7175f63c4c0cc4b05aa96705aed23c57101e02242c8085512c92d92b24b29 Sep 08 15:45:00.749719 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:00.749676 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814705-fkwgv" event={"ID":"b69a2a3e-ebce-4aa8-9525-536a88f2b551","Type":"ContainerStarted","Data":"fc9d7d58abe57d3008a0684d5a48962430beee0590b479d7b660b53216c2b042"} Sep 08 15:45:00.749943 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:00.749747 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814705-fkwgv" event={"ID":"b69a2a3e-ebce-4aa8-9525-536a88f2b551","Type":"ContainerStarted","Data":"9fc7175f63c4c0cc4b05aa96705aed23c57101e02242c8085512c92d92b24b29"} Sep 08 15:45:00.767071 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:00.767011 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814705-fkwgv" podStartSLOduration=0.766994362 podStartE2EDuration="766.994362ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:45:00.765370386 +0000 UTC m=+2162.917548965" watchObservedRunningTime="2026-09-08 15:45:00.766994362 +0000 UTC m=+2162.919172943" Sep 08 15:45:01.761515 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:01.761456 2572 generic.go:356] "Generic (PLEG): container finished" podID="b69a2a3e-ebce-4aa8-9525-536a88f2b551" containerID="fc9d7d58abe57d3008a0684d5a48962430beee0590b479d7b660b53216c2b042" exitCode=0 Sep 08 15:45:01.761927 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:01.761532 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814705-fkwgv" event={"ID":"b69a2a3e-ebce-4aa8-9525-536a88f2b551","Type":"ContainerDied","Data":"fc9d7d58abe57d3008a0684d5a48962430beee0590b479d7b660b53216c2b042"} Sep 08 15:45:02.894364 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:02.894338 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814705-fkwgv" Sep 08 15:45:03.771698 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:03.771662 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814705-fkwgv" Sep 08 15:45:03.771698 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:03.771684 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814705-fkwgv" event={"ID":"b69a2a3e-ebce-4aa8-9525-536a88f2b551","Type":"ContainerDied","Data":"9fc7175f63c4c0cc4b05aa96705aed23c57101e02242c8085512c92d92b24b29"} Sep 08 15:45:03.771935 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:45:03.771737 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fc7175f63c4c0cc4b05aa96705aed23c57101e02242c8085512c92d92b24b29" Sep 08 15:47:54.562507 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.562406 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-b858864dd-5xx5h"] Sep 08 15:47:54.562983 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.562740 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b69a2a3e-ebce-4aa8-9525-536a88f2b551" containerName="cleanup" Sep 08 15:47:54.562983 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.562751 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b69a2a3e-ebce-4aa8-9525-536a88f2b551" containerName="cleanup" Sep 08 15:47:54.562983 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.562800 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b69a2a3e-ebce-4aa8-9525-536a88f2b551" containerName="cleanup" Sep 08 15:47:54.562983 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.562808 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b5ed25-8f05-440b-a8d2-67f56fb4b0f2" containerName="istio-proxy" Sep 08 15:47:54.565864 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.565842 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" Sep 08 15:47:54.568232 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.568205 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 15:47:54.568917 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.568898 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 08 15:47:54.569190 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.569160 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 08 15:47:54.569190 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.569204 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hrbm8\"" Sep 08 15:47:54.569439 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.569239 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 08 15:47:54.578155 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.578129 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b858864dd-5xx5h"] Sep 08 15:47:54.693746 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.693704 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-metrics-certs\") pod \"maas-api-b858864dd-5xx5h\" (UID: \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\") " pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" Sep 08 15:47:54.693746 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.693745 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-openshift-service-ca\") pod \"maas-api-b858864dd-5xx5h\" (UID: \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\") " pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" Sep 08 15:47:54.693967 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.693781 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-maas-api-tls\") pod \"maas-api-b858864dd-5xx5h\" (UID: \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\") " pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" Sep 08 15:47:54.693967 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.693796 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84rp7\" (UniqueName: \"kubernetes.io/projected/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-kube-api-access-84rp7\") pod \"maas-api-b858864dd-5xx5h\" (UID: \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\") " pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" Sep 08 15:47:54.794686 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.794641 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-metrics-certs\") pod \"maas-api-b858864dd-5xx5h\" (UID: \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\") " pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" Sep 08 15:47:54.794686 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.794690 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-openshift-service-ca\") pod \"maas-api-b858864dd-5xx5h\" (UID: \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\") " pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" Sep 08 15:47:54.794917 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.794728 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-maas-api-tls\") pod \"maas-api-b858864dd-5xx5h\" (UID: \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\") " pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" Sep 08 15:47:54.794917 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.794744 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84rp7\" (UniqueName: \"kubernetes.io/projected/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-kube-api-access-84rp7\") pod \"maas-api-b858864dd-5xx5h\" (UID: \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\") " pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" Sep 08 15:47:54.795399 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.795325 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-openshift-service-ca\") pod \"maas-api-b858864dd-5xx5h\" (UID: \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\") " pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" Sep 08 15:47:54.797247 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.797222 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-metrics-certs\") pod \"maas-api-b858864dd-5xx5h\" (UID: \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\") " pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" Sep 08 15:47:54.797384 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.797362 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-maas-api-tls\") pod \"maas-api-b858864dd-5xx5h\" (UID: \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\") " pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" Sep 08 15:47:54.802750 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.802726 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84rp7\" (UniqueName: \"kubernetes.io/projected/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-kube-api-access-84rp7\") pod \"maas-api-b858864dd-5xx5h\" (UID: \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\") " pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" Sep 08 15:47:54.877041 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:54.876954 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" Sep 08 15:47:55.149011 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:55.148980 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b858864dd-5xx5h"] Sep 08 15:47:55.151611 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:47:55.151579 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a7ec92_3692_4f0d_844f_8dce03e7d41b.slice/crio-c405a3d88616957d6e0e7c81fb5313a40b5231cf9df6d0cf80c01effd07b8bef WatchSource:0}: Error finding container c405a3d88616957d6e0e7c81fb5313a40b5231cf9df6d0cf80c01effd07b8bef: Status 404 returned error can't find the container with id c405a3d88616957d6e0e7c81fb5313a40b5231cf9df6d0cf80c01effd07b8bef Sep 08 15:47:55.152917 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:55.152898 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 15:47:55.512578 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:55.512541 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" event={"ID":"d1a7ec92-3692-4f0d-844f-8dce03e7d41b","Type":"ContainerStarted","Data":"c405a3d88616957d6e0e7c81fb5313a40b5231cf9df6d0cf80c01effd07b8bef"} Sep 08 15:47:56.518572 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:56.518533 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" event={"ID":"d1a7ec92-3692-4f0d-844f-8dce03e7d41b","Type":"ContainerStarted","Data":"527fb13c57451c97834658330789afdf148cbb0a7bdbb0f664a1842645bc8f78"} Sep 08 15:47:56.518971 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:56.518626 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" Sep 08 15:47:56.536692 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:56.536632 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" podStartSLOduration=2.106095212 podStartE2EDuration="2.536614963s" totalImagesPullingTime="430.519751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:47:56.535335595 +0000 UTC m=+2338.687514176" watchObservedRunningTime="2026-09-08 15:47:56.536614963 +0000 UTC m=+2338.688793652" Sep 08 15:47:58.668086 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:47:58.668054 2572 scope.go:117] "RemoveContainer" containerID="f296360dd9b894dcd858ca91be3d79f48e40011a0c1ba279e78ce214e3416fb5" Sep 08 15:48:01.899509 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:01.899466 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-b858864dd-5xx5h"] Sep 08 15:48:01.899927 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:01.899713 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" podUID="d1a7ec92-3692-4f0d-844f-8dce03e7d41b" containerName="maas-api" containerID="cri-o://527fb13c57451c97834658330789afdf148cbb0a7bdbb0f664a1842645bc8f78" gracePeriod=30 Sep 08 15:48:01.904681 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:01.904657 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" Sep 08 15:48:01.939555 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:01.939514 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f797f8588-wffvp"] Sep 08 15:48:01.942906 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:01.942881 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f797f8588-wffvp" Sep 08 15:48:01.952907 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:01.952874 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f797f8588-wffvp"] Sep 08 15:48:02.062232 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.062193 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25752849-2d55-4624-a8e4-f25dbba671e8-openshift-service-ca\") pod \"maas-api-7f797f8588-wffvp\" (UID: \"25752849-2d55-4624-a8e4-f25dbba671e8\") " pod="odh-ai-gateway-infra/maas-api-7f797f8588-wffvp" Sep 08 15:48:02.062232 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.062234 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbtst\" (UniqueName: \"kubernetes.io/projected/25752849-2d55-4624-a8e4-f25dbba671e8-kube-api-access-lbtst\") pod \"maas-api-7f797f8588-wffvp\" (UID: \"25752849-2d55-4624-a8e4-f25dbba671e8\") " pod="odh-ai-gateway-infra/maas-api-7f797f8588-wffvp" Sep 08 15:48:02.062434 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.062256 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25752849-2d55-4624-a8e4-f25dbba671e8-maas-api-tls\") pod \"maas-api-7f797f8588-wffvp\" (UID: \"25752849-2d55-4624-a8e4-f25dbba671e8\") " pod="odh-ai-gateway-infra/maas-api-7f797f8588-wffvp" Sep 08 15:48:02.062434 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.062370 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25752849-2d55-4624-a8e4-f25dbba671e8-metrics-certs\") pod \"maas-api-7f797f8588-wffvp\" (UID: \"25752849-2d55-4624-a8e4-f25dbba671e8\") " pod="odh-ai-gateway-infra/maas-api-7f797f8588-wffvp" Sep 08 15:48:02.146539 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.146510 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" Sep 08 15:48:02.163398 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.163313 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25752849-2d55-4624-a8e4-f25dbba671e8-metrics-certs\") pod \"maas-api-7f797f8588-wffvp\" (UID: \"25752849-2d55-4624-a8e4-f25dbba671e8\") " pod="odh-ai-gateway-infra/maas-api-7f797f8588-wffvp" Sep 08 15:48:02.163398 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.163381 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25752849-2d55-4624-a8e4-f25dbba671e8-openshift-service-ca\") pod \"maas-api-7f797f8588-wffvp\" (UID: \"25752849-2d55-4624-a8e4-f25dbba671e8\") " pod="odh-ai-gateway-infra/maas-api-7f797f8588-wffvp" Sep 08 15:48:02.163599 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.163401 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbtst\" (UniqueName: \"kubernetes.io/projected/25752849-2d55-4624-a8e4-f25dbba671e8-kube-api-access-lbtst\") pod \"maas-api-7f797f8588-wffvp\" (UID: \"25752849-2d55-4624-a8e4-f25dbba671e8\") " pod="odh-ai-gateway-infra/maas-api-7f797f8588-wffvp" Sep 08 15:48:02.163599 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.163423 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25752849-2d55-4624-a8e4-f25dbba671e8-maas-api-tls\") pod \"maas-api-7f797f8588-wffvp\" (UID: \"25752849-2d55-4624-a8e4-f25dbba671e8\") " pod="odh-ai-gateway-infra/maas-api-7f797f8588-wffvp" Sep 08 15:48:02.165013 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.164974 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25752849-2d55-4624-a8e4-f25dbba671e8-openshift-service-ca\") pod \"maas-api-7f797f8588-wffvp\" (UID: \"25752849-2d55-4624-a8e4-f25dbba671e8\") " pod="odh-ai-gateway-infra/maas-api-7f797f8588-wffvp" Sep 08 15:48:02.166642 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.166618 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25752849-2d55-4624-a8e4-f25dbba671e8-metrics-certs\") pod \"maas-api-7f797f8588-wffvp\" (UID: \"25752849-2d55-4624-a8e4-f25dbba671e8\") " pod="odh-ai-gateway-infra/maas-api-7f797f8588-wffvp" Sep 08 15:48:02.166773 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.166655 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25752849-2d55-4624-a8e4-f25dbba671e8-maas-api-tls\") pod \"maas-api-7f797f8588-wffvp\" (UID: \"25752849-2d55-4624-a8e4-f25dbba671e8\") " pod="odh-ai-gateway-infra/maas-api-7f797f8588-wffvp" Sep 08 15:48:02.172369 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.172342 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbtst\" (UniqueName: \"kubernetes.io/projected/25752849-2d55-4624-a8e4-f25dbba671e8-kube-api-access-lbtst\") pod \"maas-api-7f797f8588-wffvp\" (UID: \"25752849-2d55-4624-a8e4-f25dbba671e8\") " pod="odh-ai-gateway-infra/maas-api-7f797f8588-wffvp" Sep 08 15:48:02.263760 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.263722 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-openshift-service-ca\") pod \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\" (UID: \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\") " Sep 08 15:48:02.263966 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.263784 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-maas-api-tls\") pod \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\" (UID: \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\") " Sep 08 15:48:02.263966 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.263820 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-metrics-certs\") pod \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\" (UID: \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\") " Sep 08 15:48:02.263966 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.263862 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84rp7\" (UniqueName: \"kubernetes.io/projected/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-kube-api-access-84rp7\") pod \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\" (UID: \"d1a7ec92-3692-4f0d-844f-8dce03e7d41b\") " Sep 08 15:48:02.264196 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.264169 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d1a7ec92-3692-4f0d-844f-8dce03e7d41b" (UID: "d1a7ec92-3692-4f0d-844f-8dce03e7d41b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 15:48:02.266111 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.266080 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d1a7ec92-3692-4f0d-844f-8dce03e7d41b" (UID: "d1a7ec92-3692-4f0d-844f-8dce03e7d41b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 15:48:02.266217 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.266123 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d1a7ec92-3692-4f0d-844f-8dce03e7d41b" (UID: "d1a7ec92-3692-4f0d-844f-8dce03e7d41b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 15:48:02.266217 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.266145 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-kube-api-access-84rp7" (OuterVolumeSpecName: "kube-api-access-84rp7") pod "d1a7ec92-3692-4f0d-844f-8dce03e7d41b" (UID: "d1a7ec92-3692-4f0d-844f-8dce03e7d41b"). InnerVolumeSpecName "kube-api-access-84rp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 15:48:02.276191 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.276165 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f797f8588-wffvp" Sep 08 15:48:02.365071 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.365030 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84rp7\" (UniqueName: \"kubernetes.io/projected/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-kube-api-access-84rp7\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:48:02.365071 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.365064 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-openshift-service-ca\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:48:02.365071 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.365075 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-maas-api-tls\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:48:02.365309 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.365084 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1a7ec92-3692-4f0d-844f-8dce03e7d41b-metrics-certs\") on node \"ip-10-0-136-159.ec2.internal\" DevicePath \"\"" Sep 08 15:48:02.547447 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.547411 2572 generic.go:356] "Generic (PLEG): container finished" podID="d1a7ec92-3692-4f0d-844f-8dce03e7d41b" containerID="527fb13c57451c97834658330789afdf148cbb0a7bdbb0f664a1842645bc8f78" exitCode=0 Sep 08 15:48:02.547645 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.547460 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" Sep 08 15:48:02.547645 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.547482 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" event={"ID":"d1a7ec92-3692-4f0d-844f-8dce03e7d41b","Type":"ContainerDied","Data":"527fb13c57451c97834658330789afdf148cbb0a7bdbb0f664a1842645bc8f78"} Sep 08 15:48:02.547645 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.547540 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b858864dd-5xx5h" event={"ID":"d1a7ec92-3692-4f0d-844f-8dce03e7d41b","Type":"ContainerDied","Data":"c405a3d88616957d6e0e7c81fb5313a40b5231cf9df6d0cf80c01effd07b8bef"} Sep 08 15:48:02.547645 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.547560 2572 scope.go:117] "RemoveContainer" containerID="527fb13c57451c97834658330789afdf148cbb0a7bdbb0f664a1842645bc8f78" Sep 08 15:48:02.551262 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.551243 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f797f8588-wffvp"] Sep 08 15:48:02.553743 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:48:02.553714 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25752849_2d55_4624_a8e4_f25dbba671e8.slice/crio-e78eca814ae7284aa228c003b81461f4d19d3a560af81d28711530177137dd59 WatchSource:0}: Error finding container e78eca814ae7284aa228c003b81461f4d19d3a560af81d28711530177137dd59: Status 404 returned error can't find the container with id e78eca814ae7284aa228c003b81461f4d19d3a560af81d28711530177137dd59 Sep 08 15:48:02.557083 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.557062 2572 scope.go:117] "RemoveContainer" containerID="527fb13c57451c97834658330789afdf148cbb0a7bdbb0f664a1842645bc8f78" Sep 08 15:48:02.557353 ip-10-0-136-159 kubenswrapper[2572]: E0908 15:48:02.557337 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"527fb13c57451c97834658330789afdf148cbb0a7bdbb0f664a1842645bc8f78\": container with ID starting with 527fb13c57451c97834658330789afdf148cbb0a7bdbb0f664a1842645bc8f78 not found: ID does not exist" containerID="527fb13c57451c97834658330789afdf148cbb0a7bdbb0f664a1842645bc8f78" Sep 08 15:48:02.557412 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.557361 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"527fb13c57451c97834658330789afdf148cbb0a7bdbb0f664a1842645bc8f78"} err="failed to get container status \"527fb13c57451c97834658330789afdf148cbb0a7bdbb0f664a1842645bc8f78\": rpc error: code = NotFound desc = could not find container \"527fb13c57451c97834658330789afdf148cbb0a7bdbb0f664a1842645bc8f78\": container with ID starting with 527fb13c57451c97834658330789afdf148cbb0a7bdbb0f664a1842645bc8f78 not found: ID does not exist" Sep 08 15:48:02.563343 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.563320 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-b858864dd-5xx5h"] Sep 08 15:48:02.567276 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:02.567256 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-b858864dd-5xx5h"] Sep 08 15:48:03.554559 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:03.554523 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f797f8588-wffvp" event={"ID":"25752849-2d55-4624-a8e4-f25dbba671e8","Type":"ContainerStarted","Data":"ad93de412f33975c25d588b7df85c1129aa0486dce1c352fd2f869a4b091c72c"} Sep 08 15:48:03.554982 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:03.554577 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f797f8588-wffvp" event={"ID":"25752849-2d55-4624-a8e4-f25dbba671e8","Type":"ContainerStarted","Data":"e78eca814ae7284aa228c003b81461f4d19d3a560af81d28711530177137dd59"} Sep 08 15:48:03.554982 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:03.554626 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f797f8588-wffvp" Sep 08 15:48:03.574868 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:03.574794 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f797f8588-wffvp" podStartSLOduration=2.233395518 podStartE2EDuration="2.574778383s" totalImagesPullingTime="341.382865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:48:03.572969692 +0000 UTC m=+2345.725148297" watchObservedRunningTime="2026-09-08 15:48:03.574778383 +0000 UTC m=+2345.726956962" Sep 08 15:48:04.371165 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:04.371127 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1a7ec92-3692-4f0d-844f-8dce03e7d41b" path="/var/lib/kubelet/pods/d1a7ec92-3692-4f0d-844f-8dce03e7d41b/volumes" Sep 08 15:48:09.564409 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:09.564381 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f797f8588-wffvp" Sep 08 15:48:36.219572 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:36.219537 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-thdk7_76339afb-f89c-41bd-83b2-5eebc48cf3bb/server/0.log" Sep 08 15:48:36.335635 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:36.335603 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-p8s2s_b9b71d4e-2efc-4e58-8863-5cb9c53a071e/manager/0.log" Sep 08 15:48:36.448714 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:36.448685 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66db8bdb84-4lrw9_3eaa0c78-d939-42c9-b831-0f4784f26ba9/copy-manifests/0.log" Sep 08 15:48:36.455568 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:36.455545 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66db8bdb84-4lrw9_3eaa0c78-d939-42c9-b831-0f4784f26ba9/manager/0.log" Sep 08 15:48:37.129595 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:37.129554 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7f797f8588-wffvp_25752849-2d55-4624-a8e4-f25dbba671e8/maas-api/0.log" Sep 08 15:48:37.235504 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:37.235473 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814690-dm49t_e90d1787-ea44-47bb-aedf-482ea76c029a/cleanup/0.log" Sep 08 15:48:37.342105 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:37.342080 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814705-fkwgv_b69a2a3e-ebce-4aa8-9525-536a88f2b551/cleanup/0.log" Sep 08 15:48:39.255934 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:39.255897 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn_e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a/istio-proxy/0.log" Sep 08 15:48:39.363855 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:39.363829 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9mh8t_581e98bb-746d-47a3-ab34-19e4d7755fe5/discovery/0.log" Sep 08 15:48:40.400012 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:40.399982 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-pc2kg_3f32b63c-aeb3-4ec5-9801-c95895130a34/main/0.log" Sep 08 15:48:40.737520 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:40.737419 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-xrnlz_52110219-9ed0-45a1-ab44-521f68d30b3f/main/0.log" Sep 08 15:48:40.853536 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:40.853504 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffwkrz_e02c78ba-cc89-455a-a661-997905e79abe/main/0.log" Sep 08 15:48:50.988038 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:50.988000 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9b44p_f755fdd9-db6c-43ec-bd91-5d0d861aae73/global-pull-secret-syncer/0.log" Sep 08 15:48:51.155914 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:51.155880 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j97ml_c83abab5-7e60-48e3-b075-bbd45a2a68ed/konnectivity-agent/0.log" Sep 08 15:48:51.236054 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:51.236023 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-159.ec2.internal_f326de9abe5883a85f6af224dafa6b15/haproxy/0.log" Sep 08 15:48:58.027512 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:58.027461 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2rlks_1c2d30a3-ba64-418d-8d60-947b151bd033/node-exporter/0.log" Sep 08 15:48:58.045697 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:58.045670 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2rlks_1c2d30a3-ba64-418d-8d60-947b151bd033/kube-rbac-proxy/0.log" Sep 08 15:48:58.063535 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:58.063478 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2rlks_1c2d30a3-ba64-418d-8d60-947b151bd033/init-textfile/0.log" Sep 08 15:48:58.688194 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:48:58.688165 2572 scope.go:117] "RemoveContainer" containerID="b633bd153118cf75142c65d323969cabe34e12fdf681ab736efb3bfcbe9eb612" Sep 08 15:49:01.968970 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:01.968935 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n"] Sep 08 15:49:01.969479 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:01.969447 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1a7ec92-3692-4f0d-844f-8dce03e7d41b" containerName="maas-api" Sep 08 15:49:01.969479 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:01.969475 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1a7ec92-3692-4f0d-844f-8dce03e7d41b" containerName="maas-api" Sep 08 15:49:01.969724 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:01.969557 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1a7ec92-3692-4f0d-844f-8dce03e7d41b" containerName="maas-api" Sep 08 15:49:01.973746 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:01.973730 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" Sep 08 15:49:01.976248 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:01.976228 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lskt4\"/\"openshift-service-ca.crt\"" Sep 08 15:49:01.977430 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:01.977324 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lskt4\"/\"default-dockercfg-psrfj\"" Sep 08 15:49:01.977430 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:01.977350 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lskt4\"/\"kube-root-ca.crt\"" Sep 08 15:49:01.985437 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:01.985409 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n"] Sep 08 15:49:02.033946 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.033913 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8aefebe9-5543-4e90-825f-27a5d288d06a-proc\") pod \"perf-node-gather-daemonset-dqs8n\" (UID: \"8aefebe9-5543-4e90-825f-27a5d288d06a\") " pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" Sep 08 15:49:02.033946 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.033948 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8aefebe9-5543-4e90-825f-27a5d288d06a-sys\") pod \"perf-node-gather-daemonset-dqs8n\" (UID: \"8aefebe9-5543-4e90-825f-27a5d288d06a\") " pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" Sep 08 15:49:02.034132 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.033973 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8aefebe9-5543-4e90-825f-27a5d288d06a-lib-modules\") pod \"perf-node-gather-daemonset-dqs8n\" (UID: \"8aefebe9-5543-4e90-825f-27a5d288d06a\") " pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" Sep 08 15:49:02.034132 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.034033 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7zkh\" (UniqueName: \"kubernetes.io/projected/8aefebe9-5543-4e90-825f-27a5d288d06a-kube-api-access-q7zkh\") pod \"perf-node-gather-daemonset-dqs8n\" (UID: \"8aefebe9-5543-4e90-825f-27a5d288d06a\") " pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" Sep 08 15:49:02.034132 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.034091 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8aefebe9-5543-4e90-825f-27a5d288d06a-podres\") pod \"perf-node-gather-daemonset-dqs8n\" (UID: \"8aefebe9-5543-4e90-825f-27a5d288d06a\") " pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" Sep 08 15:49:02.134780 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.134734 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8aefebe9-5543-4e90-825f-27a5d288d06a-proc\") pod \"perf-node-gather-daemonset-dqs8n\" (UID: \"8aefebe9-5543-4e90-825f-27a5d288d06a\") " pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" Sep 08 15:49:02.134956 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.134801 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8aefebe9-5543-4e90-825f-27a5d288d06a-sys\") pod \"perf-node-gather-daemonset-dqs8n\" (UID: \"8aefebe9-5543-4e90-825f-27a5d288d06a\") " pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" Sep 08 15:49:02.134956 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.134837 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8aefebe9-5543-4e90-825f-27a5d288d06a-lib-modules\") pod \"perf-node-gather-daemonset-dqs8n\" (UID: \"8aefebe9-5543-4e90-825f-27a5d288d06a\") " pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" Sep 08 15:49:02.134956 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.134847 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8aefebe9-5543-4e90-825f-27a5d288d06a-proc\") pod \"perf-node-gather-daemonset-dqs8n\" (UID: \"8aefebe9-5543-4e90-825f-27a5d288d06a\") " pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" Sep 08 15:49:02.134956 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.134857 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7zkh\" (UniqueName: \"kubernetes.io/projected/8aefebe9-5543-4e90-825f-27a5d288d06a-kube-api-access-q7zkh\") pod \"perf-node-gather-daemonset-dqs8n\" (UID: \"8aefebe9-5543-4e90-825f-27a5d288d06a\") " pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" Sep 08 15:49:02.134956 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.134917 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8aefebe9-5543-4e90-825f-27a5d288d06a-sys\") pod \"perf-node-gather-daemonset-dqs8n\" (UID: \"8aefebe9-5543-4e90-825f-27a5d288d06a\") " pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" Sep 08 15:49:02.134956 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.134921 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8aefebe9-5543-4e90-825f-27a5d288d06a-podres\") pod \"perf-node-gather-daemonset-dqs8n\" (UID: \"8aefebe9-5543-4e90-825f-27a5d288d06a\") " pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" Sep 08 15:49:02.134956 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.134956 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8aefebe9-5543-4e90-825f-27a5d288d06a-lib-modules\") pod \"perf-node-gather-daemonset-dqs8n\" (UID: \"8aefebe9-5543-4e90-825f-27a5d288d06a\") " pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" Sep 08 15:49:02.135186 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.134987 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8aefebe9-5543-4e90-825f-27a5d288d06a-podres\") pod \"perf-node-gather-daemonset-dqs8n\" (UID: \"8aefebe9-5543-4e90-825f-27a5d288d06a\") " pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" Sep 08 15:49:02.148027 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.147998 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7zkh\" (UniqueName: \"kubernetes.io/projected/8aefebe9-5543-4e90-825f-27a5d288d06a-kube-api-access-q7zkh\") pod \"perf-node-gather-daemonset-dqs8n\" (UID: \"8aefebe9-5543-4e90-825f-27a5d288d06a\") " pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" Sep 08 15:49:02.289186 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.289140 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" Sep 08 15:49:02.563051 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.563024 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n"] Sep 08 15:49:02.565605 ip-10-0-136-159 kubenswrapper[2572]: W0908 15:49:02.565574 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8aefebe9_5543_4e90_825f_27a5d288d06a.slice/crio-15a16af298f56de86c0adf7083a5d33c9cd92cf63f15bb802fd2beca8f308abf WatchSource:0}: Error finding container 15a16af298f56de86c0adf7083a5d33c9cd92cf63f15bb802fd2beca8f308abf: Status 404 returned error can't find the container with id 15a16af298f56de86c0adf7083a5d33c9cd92cf63f15bb802fd2beca8f308abf Sep 08 15:49:02.815085 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.814993 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" event={"ID":"8aefebe9-5543-4e90-825f-27a5d288d06a","Type":"ContainerStarted","Data":"efc90b411d97ddafcc8c80111a2fc90cc292903de442b79a073ee86d2034180d"} Sep 08 15:49:02.815085 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.815048 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" event={"ID":"8aefebe9-5543-4e90-825f-27a5d288d06a","Type":"ContainerStarted","Data":"15a16af298f56de86c0adf7083a5d33c9cd92cf63f15bb802fd2beca8f308abf"} Sep 08 15:49:02.815313 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.815151 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" Sep 08 15:49:02.831919 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:02.831872 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" podStartSLOduration=1.831857531 podStartE2EDuration="1.831857531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:49:02.830694814 +0000 UTC m=+2404.982873394" watchObservedRunningTime="2026-09-08 15:49:02.831857531 +0000 UTC m=+2404.984036111" Sep 08 15:49:03.328999 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:03.328966 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2hz45_266dc55e-c863-43b4-93ee-e4b5fc671954/dns/0.log" Sep 08 15:49:03.351814 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:03.351784 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2hz45_266dc55e-c863-43b4-93ee-e4b5fc671954/kube-rbac-proxy/0.log" Sep 08 15:49:03.544241 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:03.544214 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zxbnk_1dfce943-0703-4e7d-b479-a7616aa67993/dns-node-resolver/0.log" Sep 08 15:49:04.127761 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:04.127719 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wjg4s_90fade35-fbac-48dd-834c-919ef54da378/node-ca/0.log" Sep 08 15:49:05.041361 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:05.041327 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn_e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a/istio-proxy/0.log" Sep 08 15:49:05.337885 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:05.337800 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9mh8t_581e98bb-746d-47a3-ab34-19e4d7755fe5/discovery/0.log" Sep 08 15:49:06.429994 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:06.429964 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pj4qx_ee735889-613b-4a94-a579-596fabddb360/serve-healthcheck-canary/0.log" Sep 08 15:49:07.016129 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:07.016103 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-65p5d_69ccff15-f07b-4c80-8f00-96b2d7cd2888/kube-rbac-proxy/0.log" Sep 08 15:49:07.034209 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:07.034182 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-65p5d_69ccff15-f07b-4c80-8f00-96b2d7cd2888/exporter/0.log" Sep 08 15:49:07.053271 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:07.053243 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-65p5d_69ccff15-f07b-4c80-8f00-96b2d7cd2888/extractor/0.log" Sep 08 15:49:08.830058 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:08.830024 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lskt4/perf-node-gather-daemonset-dqs8n" Sep 08 15:49:09.664576 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:09.664545 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-thdk7_76339afb-f89c-41bd-83b2-5eebc48cf3bb/server/0.log" Sep 08 15:49:09.744949 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:09.744910 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-p8s2s_b9b71d4e-2efc-4e58-8863-5cb9c53a071e/manager/0.log" Sep 08 15:49:09.769470 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:09.769441 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66db8bdb84-4lrw9_3eaa0c78-d939-42c9-b831-0f4784f26ba9/manager/0.log" Sep 08 15:49:09.788097 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:09.788066 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66db8bdb84-4lrw9_3eaa0c78-d939-42c9-b831-0f4784f26ba9/copy-manifests/0.log" Sep 08 15:49:11.075001 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:11.074978 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn_e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a/istio-proxy/0.log" Sep 08 15:49:11.364979 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:11.364884 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9mh8t_581e98bb-746d-47a3-ab34-19e4d7755fe5/discovery/0.log" Sep 08 15:49:11.439243 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:11.439214 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7f7779588f-v9r6v_09f2130e-c158-4405-a1dc-55dd88b8502c/manager/0.log" Sep 08 15:49:13.076194 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:13.076085 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-jk4qn_e63bb5ac-88a9-4477-8e9e-9f0abd3eaf7a/istio-proxy/0.log" Sep 08 15:49:13.358383 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:13.358288 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9mh8t_581e98bb-746d-47a3-ab34-19e4d7755fe5/discovery/0.log" Sep 08 15:49:17.955399 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:17.955351 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mhvv_4d45089b-2c7d-45d7-883b-417a0bcb9c1b/kube-multus-additional-cni-plugins/0.log" Sep 08 15:49:17.975979 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:17.975947 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mhvv_4d45089b-2c7d-45d7-883b-417a0bcb9c1b/egress-router-binary-copy/0.log" Sep 08 15:49:17.995841 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:17.995811 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mhvv_4d45089b-2c7d-45d7-883b-417a0bcb9c1b/cni-plugins/0.log" Sep 08 15:49:18.022988 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:18.022959 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mhvv_4d45089b-2c7d-45d7-883b-417a0bcb9c1b/bond-cni-plugin/0.log" Sep 08 15:49:18.041376 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:18.041344 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mhvv_4d45089b-2c7d-45d7-883b-417a0bcb9c1b/routeoverride-cni/0.log" Sep 08 15:49:18.059073 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:18.059043 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mhvv_4d45089b-2c7d-45d7-883b-417a0bcb9c1b/whereabouts-cni-bincopy/0.log" Sep 08 15:49:18.076632 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:18.076605 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mhvv_4d45089b-2c7d-45d7-883b-417a0bcb9c1b/whereabouts-cni/0.log" Sep 08 15:49:18.487523 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:18.487473 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pxfh4_7662e1ba-5c64-4348-844c-1f5fd285ce6f/kube-multus/0.log" Sep 08 15:49:18.632735 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:18.632703 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n5x8x_1dafedb8-50fc-4eff-a5a8-ee1be3e3f751/network-metrics-daemon/0.log" Sep 08 15:49:18.652528 ip-10-0-136-159 kubenswrapper[2572]: I0908 15:49:18.652481 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n5x8x_1dafedb8-50fc-4eff-a5a8-ee1be3e3f751/kube-rbac-proxy/0.log"