Sep 17 01:22:14.671208 ip-10-0-138-83 systemd[1]: Starting Kubernetes Kubelet... Sep 17 01:22:14.982759 ip-10-0-138-83 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 01:22:14.982759 ip-10-0-138-83 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 17 01:22:14.982759 ip-10-0-138-83 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 01:22:14.982759 ip-10-0-138-83 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 17 01:22:14.982759 ip-10-0-138-83 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 01:22:14.983537 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.983451 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 17 01:22:14.986788 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986766 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 01:22:14.986788 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986785 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 01:22:14.986788 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986790 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 01:22:14.986788 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986796 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 01:22:14.987113 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986802 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 01:22:14.987113 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986806 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 01:22:14.987113 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986810 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 01:22:14.987113 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986814 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 01:22:14.987113 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986818 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 01:22:14.987113 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986821 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 01:22:14.987113 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986825 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 01:22:14.987113 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986829 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 01:22:14.987113 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986833 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 01:22:14.987113 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986838 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 01:22:14.987113 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986841 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 01:22:14.987113 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986847 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 01:22:14.987113 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986852 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 01:22:14.987113 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986857 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 01:22:14.987113 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986861 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 01:22:14.987113 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986865 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 17 01:22:14.987113 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986870 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 01:22:14.987113 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986881 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 01:22:14.987113 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986886 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 01:22:14.987891 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986891 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 01:22:14.987891 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986896 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 01:22:14.987891 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986901 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 01:22:14.987891 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986905 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 01:22:14.987891 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986909 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 01:22:14.987891 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986914 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 01:22:14.987891 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986921 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 01:22:14.987891 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986925 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 01:22:14.987891 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986946 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 01:22:14.987891 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986950 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 01:22:14.987891 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986954 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 01:22:14.987891 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986958 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 01:22:14.987891 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986962 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 01:22:14.987891 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986966 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 01:22:14.987891 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986970 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 01:22:14.987891 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986974 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 01:22:14.987891 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986978 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 01:22:14.987891 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986982 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 01:22:14.987891 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986986 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986991 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986995 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.986999 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987004 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987008 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987012 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987017 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987020 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987025 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987029 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987033 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987038 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987042 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987047 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987051 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987055 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987059 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987064 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987068 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 01:22:14.988716 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987072 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987076 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987080 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987085 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987089 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987093 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987097 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987103 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987107 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987111 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987116 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987120 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987124 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987128 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987132 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987137 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987141 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987145 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987149 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987154 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 01:22:14.989567 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987161 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987165 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987169 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987173 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987177 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987182 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987186 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987191 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987195 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987199 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987212 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987217 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987221 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987226 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987230 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987234 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987239 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.987243 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988369 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988381 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 01:22:14.990446 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988386 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 01:22:14.991329 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988390 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 01:22:14.991329 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988395 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 01:22:14.991329 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988400 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 01:22:14.991329 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988405 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 01:22:14.991329 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988409 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 01:22:14.991329 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988413 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 01:22:14.991329 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988417 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 01:22:14.991329 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988422 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 01:22:14.991329 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988427 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 01:22:14.991329 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988431 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 01:22:14.991329 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988435 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 01:22:14.991329 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988440 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 01:22:14.991329 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988447 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 01:22:14.991329 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988453 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 01:22:14.991329 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988458 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 01:22:14.991329 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988463 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 01:22:14.991329 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988468 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 01:22:14.991329 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988473 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 01:22:14.991329 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988477 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 01:22:14.992020 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988482 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 01:22:14.992020 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988486 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 01:22:14.992020 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988492 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 01:22:14.992020 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988496 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 01:22:14.992020 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988500 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 01:22:14.992020 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988505 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 01:22:14.992020 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988509 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 01:22:14.992020 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988513 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 01:22:14.992020 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988517 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 17 01:22:14.992020 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988521 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 01:22:14.992020 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988525 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 01:22:14.992020 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988530 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 01:22:14.992020 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988535 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 01:22:14.992020 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988540 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 01:22:14.992020 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988545 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 01:22:14.992020 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988549 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 01:22:14.992020 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988556 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 01:22:14.992020 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988562 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 01:22:14.992020 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988567 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988571 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988576 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988580 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988585 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988590 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988595 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988604 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988608 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988613 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988617 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988621 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988625 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988629 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988633 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988638 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988643 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988647 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988651 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988655 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 01:22:14.992526 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988660 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988664 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988668 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988672 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988678 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988683 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988687 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988691 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988695 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988699 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988703 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988707 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988711 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988715 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988719 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988723 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988727 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988733 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988738 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988742 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 01:22:14.993074 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988747 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988751 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988755 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988759 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988763 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988767 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988772 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988776 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988780 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988785 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988789 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988794 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988798 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988802 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988806 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988811 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988816 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988820 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.988824 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989429 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 01:22:14.993667 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989437 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989442 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989447 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989451 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989456 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989460 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989465 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989469 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989473 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989478 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989482 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989487 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989492 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989497 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989501 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989505 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989510 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989514 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989519 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989523 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 01:22:14.994247 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989528 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 01:22:14.994881 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989532 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 01:22:14.994881 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989539 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 01:22:14.994881 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989545 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 01:22:14.994881 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989550 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 01:22:14.994881 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989555 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 01:22:14.994881 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989559 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 01:22:14.994881 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989563 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 01:22:14.994881 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989568 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 01:22:14.994881 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989573 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 01:22:14.994881 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989578 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 01:22:14.994881 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989583 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 01:22:14.994881 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989588 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 01:22:14.994881 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989592 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 01:22:14.994881 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989597 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 01:22:14.994881 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989601 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 01:22:14.994881 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989605 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 01:22:14.994881 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989609 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 01:22:14.994881 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989614 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 01:22:14.994881 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989618 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 01:22:14.995383 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989623 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 01:22:14.995383 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989627 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 01:22:14.995383 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989631 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 01:22:14.995383 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989635 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 01:22:14.995383 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989640 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 01:22:14.995383 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989645 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 01:22:14.995383 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989649 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 01:22:14.995383 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989653 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 01:22:14.995383 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989658 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 01:22:14.995383 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989662 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 01:22:14.995383 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989666 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 01:22:14.995383 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989670 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 01:22:14.995383 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989675 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 01:22:14.995383 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989679 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 01:22:14.995383 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989683 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 01:22:14.995383 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989687 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 01:22:14.995383 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989695 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 01:22:14.995383 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989700 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 01:22:14.995383 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989705 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989710 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989715 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989722 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989726 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989730 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989734 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989738 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989743 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989747 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989751 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989755 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989759 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989763 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989767 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989771 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989775 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989780 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989784 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989789 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 01:22:14.995975 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989794 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989798 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989802 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989806 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989810 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989816 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989822 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989826 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989831 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989835 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989839 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989843 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989847 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989851 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989855 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989861 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989866 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989870 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989874 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.989878 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 01:22:14.996467 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990002 2577 flags.go:64] FLAG: --address="0.0.0.0" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990051 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990062 2577 flags.go:64] FLAG: --anonymous-auth="true" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990069 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990076 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990081 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990087 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990100 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990105 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990110 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990115 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990120 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990125 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990132 2577 flags.go:64] FLAG: --cgroup-root="" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990136 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990141 2577 flags.go:64] FLAG: --client-ca-file="" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990146 2577 flags.go:64] FLAG: --cloud-provider="external" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990150 2577 flags.go:64] FLAG: --cluster-dns="[]" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990159 2577 flags.go:64] FLAG: --cluster-domain="" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990164 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990170 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990175 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990181 2577 flags.go:64] FLAG: --container-log-max-files="5" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990187 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 17 01:22:14.996983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990191 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990197 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990202 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990209 2577 flags.go:64] FLAG: --contention-profiling="false" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990214 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990218 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990224 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990229 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990235 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990240 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990245 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990250 2577 flags.go:64] FLAG: --enable-load-reader="false" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990255 2577 flags.go:64] FLAG: --enable-server="true" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990260 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990267 2577 flags.go:64] FLAG: --event-burst="100" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990272 2577 flags.go:64] FLAG: --event-qps="50" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990277 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990282 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990287 2577 flags.go:64] FLAG: --eviction-hard="" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990292 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990297 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990302 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990308 2577 flags.go:64] FLAG: --eviction-soft="" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990313 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990318 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 17 01:22:14.997583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990323 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990329 2577 flags.go:64] FLAG: --experimental-mounter-path="" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990334 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990338 2577 flags.go:64] FLAG: --fail-swap-on="true" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990343 2577 flags.go:64] FLAG: --feature-gates="" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990349 2577 flags.go:64] FLAG: --file-check-frequency="20s" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990353 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990359 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990364 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990369 2577 flags.go:64] FLAG: --healthz-port="10248" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990376 2577 flags.go:64] FLAG: --help="false" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990380 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-138-83.ec2.internal" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990386 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990391 2577 flags.go:64] FLAG: --http-check-frequency="20s" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990396 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990401 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990407 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990412 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990417 2577 flags.go:64] FLAG: --image-service-endpoint="" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990422 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990426 2577 flags.go:64] FLAG: --kube-api-burst="100" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990431 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990436 2577 flags.go:64] FLAG: --kube-api-qps="50" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990441 2577 flags.go:64] FLAG: --kube-reserved="" Sep 17 01:22:14.998211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990445 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990450 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990455 2577 flags.go:64] FLAG: --kubelet-cgroups="" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990459 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990464 2577 flags.go:64] FLAG: --lock-file="" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990469 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990474 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990479 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990488 2577 flags.go:64] FLAG: --log-json-split-stream="false" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990496 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990501 2577 flags.go:64] FLAG: --log-text-split-stream="false" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990505 2577 flags.go:64] FLAG: --logging-format="text" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990510 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990515 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990520 2577 flags.go:64] FLAG: --manifest-url="" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990525 2577 flags.go:64] FLAG: --manifest-url-header="" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990531 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990536 2577 flags.go:64] FLAG: --max-open-files="1000000" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990543 2577 flags.go:64] FLAG: --max-pods="110" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990548 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990553 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990557 2577 flags.go:64] FLAG: --memory-manager-policy="None" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990562 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990567 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990571 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 17 01:22:14.998867 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990576 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990587 2577 flags.go:64] FLAG: --node-status-max-images="50" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990592 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990597 2577 flags.go:64] FLAG: --oom-score-adj="-999" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990602 2577 flags.go:64] FLAG: --pod-cidr="" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990607 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990615 2577 flags.go:64] FLAG: --pod-manifest-path="" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990620 2577 flags.go:64] FLAG: --pod-max-pids="-1" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990624 2577 flags.go:64] FLAG: --pods-per-core="0" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990629 2577 flags.go:64] FLAG: --port="10250" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990634 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990639 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0890aa6b658e764df" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990644 2577 flags.go:64] FLAG: --qos-reserved="" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990649 2577 flags.go:64] FLAG: --read-only-port="10255" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990655 2577 flags.go:64] FLAG: --register-node="true" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990660 2577 flags.go:64] FLAG: --register-with-taints="" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990665 2577 flags.go:64] FLAG: --registry-burst="10" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990672 2577 flags.go:64] FLAG: --registry-qps="5" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990676 2577 flags.go:64] FLAG: --reserved-cpus="" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990681 2577 flags.go:64] FLAG: --reserved-memory="" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990687 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990692 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990697 2577 flags.go:64] FLAG: --rotate-certificates="false" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990702 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990706 2577 flags.go:64] FLAG: --runonce="false" Sep 17 01:22:14.999522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990711 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990716 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990722 2577 flags.go:64] FLAG: --seccomp-default="false" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990726 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990731 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990737 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990742 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990746 2577 flags.go:64] FLAG: --storage-driver-password="root" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990751 2577 flags.go:64] FLAG: --storage-driver-secure="false" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990756 2577 flags.go:64] FLAG: --storage-driver-table="stats" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990760 2577 flags.go:64] FLAG: --storage-driver-user="root" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990765 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990771 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990776 2577 flags.go:64] FLAG: --system-cgroups="" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990780 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990789 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990793 2577 flags.go:64] FLAG: --tls-cert-file="" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990798 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990804 2577 flags.go:64] FLAG: --tls-min-version="" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990808 2577 flags.go:64] FLAG: --tls-private-key-file="" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990813 2577 flags.go:64] FLAG: --topology-manager-policy="none" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990818 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990824 2577 flags.go:64] FLAG: --topology-manager-scope="container" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990829 2577 flags.go:64] FLAG: --v="2" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990836 2577 flags.go:64] FLAG: --version="false" Sep 17 01:22:15.000147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990842 2577 flags.go:64] FLAG: --vmodule="" Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990849 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.990854 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991078 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991086 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991091 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991096 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991101 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991105 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991109 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991114 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991119 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991124 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991129 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991133 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991138 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991142 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991146 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991150 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991155 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 01:22:15.000754 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991159 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991163 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991167 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991173 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991178 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991183 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991187 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991191 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991195 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991200 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991211 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991218 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991222 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991226 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991231 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991235 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991239 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991246 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991252 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991256 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 01:22:15.001324 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991261 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991266 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991271 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991275 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991280 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991284 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991288 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991293 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991297 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991301 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991306 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991310 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991314 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991318 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991323 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991326 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991331 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991335 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991338 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991343 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 01:22:15.001815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991346 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 01:22:15.002317 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991350 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 01:22:15.002317 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991354 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 01:22:15.002317 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991363 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 01:22:15.002317 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991368 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 01:22:15.002317 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991372 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 01:22:15.002317 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991376 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 01:22:15.002317 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991380 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 01:22:15.002317 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991384 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 01:22:15.002317 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991389 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 01:22:15.002317 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991393 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 01:22:15.002317 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991397 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 01:22:15.002317 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991402 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 01:22:15.002317 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991406 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 01:22:15.002317 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991410 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 01:22:15.002317 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991415 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 01:22:15.002317 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991419 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 01:22:15.002317 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991426 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 01:22:15.002317 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991431 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 01:22:15.002317 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991436 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991440 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991444 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991449 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991453 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991458 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991462 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991466 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991470 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991474 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991478 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991482 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991486 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991490 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991494 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991499 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991506 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991512 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991516 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991520 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 01:22:15.002778 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991524 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 01:22:15.003298 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991528 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 01:22:15.003298 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:14.991533 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 01:22:15.003298 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:14.991541 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 01:22:15.003298 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.000195 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 17 01:22:15.003298 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.000213 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 17 01:22:15.003298 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.000233 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 17 01:22:15.003298 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.000239 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 17 01:22:15.003298 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000314 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 01:22:15.003298 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000319 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 01:22:15.003298 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000322 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 01:22:15.003298 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000325 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 01:22:15.003298 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000328 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 01:22:15.003298 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000331 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 01:22:15.003298 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000334 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 01:22:15.003298 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000337 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000340 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000343 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000345 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000348 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000350 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000353 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000356 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000358 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000362 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000366 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000369 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000372 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000374 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000377 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000380 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000383 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000386 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000388 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000391 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 01:22:15.003675 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000394 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000396 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000405 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000408 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000411 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000414 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000417 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000419 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000422 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000425 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000427 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000430 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000433 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000435 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000438 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000441 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000443 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000446 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000448 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000451 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 01:22:15.004175 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000453 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000456 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000458 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000461 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000464 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000466 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000469 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000471 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000474 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000476 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000479 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000482 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000484 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000488 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000491 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000499 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000502 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000506 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000510 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000513 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 01:22:15.004694 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000516 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 01:22:15.005222 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000519 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 01:22:15.005222 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000522 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 01:22:15.005222 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000524 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 01:22:15.005222 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000527 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 01:22:15.005222 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000530 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 01:22:15.005222 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000532 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 01:22:15.005222 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000535 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 01:22:15.005222 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000537 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 01:22:15.005222 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000540 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 01:22:15.005222 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000542 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 01:22:15.005222 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000545 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 01:22:15.005222 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000548 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 01:22:15.005222 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000550 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 01:22:15.005222 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000553 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 01:22:15.005222 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000555 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 01:22:15.005222 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000558 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 01:22:15.005222 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000560 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 01:22:15.005222 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000563 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 01:22:15.005222 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000565 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 01:22:15.005685 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000568 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 01:22:15.005685 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000571 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 01:22:15.005685 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000573 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 01:22:15.005685 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000576 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 01:22:15.005685 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000578 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 01:22:15.005685 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000582 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 01:22:15.005685 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000585 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 01:22:15.005685 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000589 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 01:22:15.005685 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000598 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 01:22:15.005685 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000601 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 01:22:15.005685 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000604 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 01:22:15.005685 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000607 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 01:22:15.005685 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.000612 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 01:22:15.005685 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000753 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000764 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000768 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000771 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000774 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000777 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000779 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000782 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000785 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000788 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000790 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000793 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000795 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000798 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000800 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000803 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000805 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000808 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000810 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000813 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 01:22:15.006067 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000816 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 01:22:15.006543 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000819 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 01:22:15.006543 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000822 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 01:22:15.006543 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000825 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 01:22:15.006543 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000827 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 01:22:15.006543 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000830 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 01:22:15.006543 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000833 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 01:22:15.006543 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000835 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 01:22:15.006543 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000845 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 01:22:15.006543 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000847 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 01:22:15.006543 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000850 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 01:22:15.006543 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000853 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 01:22:15.006543 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000855 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 01:22:15.006543 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000858 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 01:22:15.006543 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000860 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 01:22:15.006543 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000863 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 01:22:15.006543 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000865 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 01:22:15.006543 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000868 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 01:22:15.006543 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000870 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 01:22:15.006543 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000872 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000875 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000877 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000880 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000883 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000885 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000888 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000890 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000892 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000895 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000897 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000900 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000903 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000905 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000908 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000910 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000913 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000915 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000918 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000921 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 01:22:15.007018 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000923 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 01:22:15.007510 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000926 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 01:22:15.007510 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000954 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 01:22:15.007510 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000958 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 01:22:15.007510 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000961 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 01:22:15.007510 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000963 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 01:22:15.007510 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000966 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 01:22:15.007510 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000969 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 01:22:15.007510 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000973 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 01:22:15.007510 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000976 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 01:22:15.007510 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000979 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 01:22:15.007510 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000982 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 01:22:15.007510 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000985 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 01:22:15.007510 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000988 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 01:22:15.007510 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000991 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 01:22:15.007510 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000993 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 01:22:15.007510 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000996 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 01:22:15.007510 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.000999 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 01:22:15.007510 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001002 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 01:22:15.007510 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001005 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001009 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001012 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001015 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001018 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001021 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001024 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001026 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001029 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001031 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001034 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001037 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001040 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001042 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001045 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001048 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001056 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001059 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001061 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001064 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 01:22:15.008032 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:15.001066 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 01:22:15.008812 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.001071 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 01:22:15.008812 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.001187 2577 server.go:962] "Client rotation is on, will bootstrap in background" Sep 17 01:22:15.008812 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.003156 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 17 01:22:15.008812 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.003918 2577 server.go:1019] "Starting client certificate rotation" Sep 17 01:22:15.008812 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.004028 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 01:22:15.008812 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.004058 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 01:22:15.008812 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.006250 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 01:22:15.011061 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.011042 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 01:22:15.021681 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.021661 2577 log.go:25] "Validated CRI v1 runtime API" Sep 17 01:22:15.024690 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.024676 2577 log.go:25] "Validated CRI v1 image API" Sep 17 01:22:15.025603 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.025584 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 17 01:22:15.029435 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.029416 2577 fs.go:135] Filesystem UUIDs: map[1f377750-2707-4ad8-af2b-1177a02a6c1f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e9996b3c-12a4-40d6-8a03-c227b9648ec3:/dev/nvme0n1p4] Sep 17 01:22:15.029505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.029436 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 17 01:22:15.033228 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.033212 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 01:22:15.034464 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.034364 2577 manager.go:216] Machine: {Timestamp:2026-09-17 01:22:15.032985285 +0000 UTC m=+0.268132463 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3118852 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec243bab172dcc7c1c6b91672414479d SystemUUID:ec243bab-172d-cc7c-1c6b-91672414479d BootID:d1c79865-6c2e-4ddf-82b5-3f0ef5ff81f9 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e1:99:c7:67 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e1:99:c7:67 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:1d:1b:7f:10:ab Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 17 01:22:15.034464 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.034460 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 17 01:22:15.034569 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.034537 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 17 01:22:15.037787 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.037763 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 17 01:22:15.037939 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.037789 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-83.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 17 01:22:15.038559 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.038549 2577 topology_manager.go:138] "Creating topology manager with none policy" Sep 17 01:22:15.038593 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.038562 2577 container_manager_linux.go:309] "Creating device plugin manager" Sep 17 01:22:15.038593 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.038570 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 01:22:15.039327 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.039316 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 01:22:15.040471 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.040461 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 17 01:22:15.044796 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.044783 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 17 01:22:15.044910 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.044900 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 17 01:22:15.044961 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.044941 2577 kubelet.go:486] "Attempting to sync node with API server" Sep 17 01:22:15.044961 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.044951 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 17 01:22:15.044961 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.044961 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 17 01:22:15.045076 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.044972 2577 kubelet.go:398] "Adding apiserver pod source" Sep 17 01:22:15.045076 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.044981 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 17 01:22:15.045314 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.045301 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 01:22:15.045347 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.045326 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 01:22:15.048526 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.048509 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 17 01:22:15.052965 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.052948 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 17 01:22:15.053026 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.052980 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 17 01:22:15.054855 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.054837 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 17 01:22:15.054855 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.054854 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 17 01:22:15.054985 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.054860 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 17 01:22:15.054985 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.054866 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 17 01:22:15.054985 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.054874 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 17 01:22:15.054985 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.054879 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 17 01:22:15.054985 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.054892 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 17 01:22:15.054985 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.054899 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 17 01:22:15.054985 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.054908 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 17 01:22:15.054985 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.054914 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 17 01:22:15.054985 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.054922 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 17 01:22:15.054985 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.054940 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 17 01:22:15.055467 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.055444 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 17 01:22:15.055521 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.055474 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-83.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 17 01:22:15.055869 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.055860 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 17 01:22:15.055898 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.055869 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 17 01:22:15.058128 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.058112 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jltg4" Sep 17 01:22:15.059569 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.059557 2577 server.go:1268] "Started kubelet" Sep 17 01:22:15.059680 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.059636 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 17 01:22:15.059737 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.059649 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 17 01:22:15.059737 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.059695 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 17 01:22:15.060576 ip-10-0-138-83 systemd[1]: Started Kubernetes Kubelet. Sep 17 01:22:15.060790 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.060661 2577 server.go:310] "Adding debug handlers to kubelet server" Sep 17 01:22:15.061353 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.061331 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 17 01:22:15.065113 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.065096 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jltg4" Sep 17 01:22:15.066711 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.066693 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-83.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 17 01:22:15.066889 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.066874 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 17 01:22:15.067823 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.066714 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-83.ec2.internal.18d5f71df95f6620 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-83.ec2.internal,UID:ip-10-0-138-83.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-83.ec2.internal,},FirstTimestamp:2026-09-17 01:22:15.059531296 +0000 UTC m=+0.294678476,LastTimestamp:2026-09-17 01:22:15.059531296 +0000 UTC m=+0.294678476,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-83.ec2.internal,}" Sep 17 01:22:15.069882 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.069866 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 17 01:22:15.070530 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.070512 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 17 01:22:15.070686 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.070664 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-83.ec2.internal\" not found" Sep 17 01:22:15.071348 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.071332 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 17 01:22:15.071509 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.071491 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 17 01:22:15.071622 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.071332 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 17 01:22:15.071622 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.071593 2577 reconstruct.go:97] "Volume reconstruction finished" Sep 17 01:22:15.071622 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.071614 2577 factory.go:153] Registering CRI-O factory Sep 17 01:22:15.071622 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.071620 2577 reconciler.go:29] "Reconciler: start to sync state" Sep 17 01:22:15.071779 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.071627 2577 factory.go:223] Registration of the crio container factory successfully Sep 17 01:22:15.071779 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.071672 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 17 01:22:15.071779 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.071680 2577 factory.go:55] Registering systemd factory Sep 17 01:22:15.071779 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.071684 2577 factory.go:223] Registration of the systemd container factory successfully Sep 17 01:22:15.071779 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.071701 2577 factory.go:103] Registering Raw factory Sep 17 01:22:15.071779 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.071715 2577 manager.go:1196] Started watching for new ooms in manager Sep 17 01:22:15.072083 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.072061 2577 manager.go:349] Starting recovery of all containers Sep 17 01:22:15.073241 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.073209 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 01:22:15.075609 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.075588 2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-83.ec2.internal\" not found" node="ip-10-0-138-83.ec2.internal" Sep 17 01:22:15.084597 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.084440 2577 manager.go:354] Recovery completed Sep 17 01:22:15.089227 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.089211 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:22:15.092078 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.092058 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:22:15.092161 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.092097 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:22:15.092161 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.092113 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:22:15.092715 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.092701 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 17 01:22:15.092715 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.092713 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 17 01:22:15.092831 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.092730 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 17 01:22:15.092831 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.092781 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:22:15.094894 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.094882 2577 policy_none.go:49] "None policy: Start" Sep 17 01:22:15.094950 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.094899 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 17 01:22:15.094950 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.094909 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 17 01:22:15.095558 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.095540 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:22:15.095624 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.095570 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:22:15.095624 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.095583 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:22:15.096965 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.096940 2577 policy_none.go:47] "Start" Sep 17 01:22:15.143919 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.134194 2577 manager.go:337] "Starting Device Plugin manager" Sep 17 01:22:15.143919 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.134233 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 17 01:22:15.143919 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.134245 2577 server.go:87] "Starting device plugin registration server" Sep 17 01:22:15.143919 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.134571 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 17 01:22:15.143919 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.134586 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 17 01:22:15.143919 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.134688 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 17 01:22:15.143919 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.134826 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 17 01:22:15.143919 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.134834 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 17 01:22:15.143919 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.135304 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 17 01:22:15.143919 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.135352 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-83.ec2.internal\" not found" Sep 17 01:22:15.203672 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.203630 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 17 01:22:15.204975 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.204946 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 17 01:22:15.205099 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.205007 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 17 01:22:15.205099 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.205036 2577 kubelet.go:2448] "Starting kubelet main sync loop" Sep 17 01:22:15.205208 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.205138 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 17 01:22:15.206991 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.206971 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 01:22:15.235018 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.234942 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:22:15.235818 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.235803 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:22:15.235900 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.235834 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:22:15.235900 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.235848 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:22:15.235900 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.235877 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-83.ec2.internal" Sep 17 01:22:15.243183 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.243155 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-83.ec2.internal" Sep 17 01:22:15.243291 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.243184 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-83.ec2.internal\": node \"ip-10-0-138-83.ec2.internal\" not found" Sep 17 01:22:15.255674 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.255650 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-83.ec2.internal\" not found" Sep 17 01:22:15.305496 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.305472 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-83.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal"] Sep 17 01:22:15.305582 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.305545 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:22:15.307692 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.307675 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:22:15.307772 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.307707 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:22:15.307772 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.307721 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:22:15.310211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.310198 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:22:15.310320 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.310306 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-83.ec2.internal" Sep 17 01:22:15.310361 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.310331 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:22:15.311209 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.311185 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:22:15.311288 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.311220 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:22:15.311288 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.311230 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:22:15.311288 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.311187 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:22:15.311288 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.311256 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:22:15.311288 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.311272 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:22:15.313552 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.313537 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal" Sep 17 01:22:15.313619 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.313563 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 01:22:15.323213 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.323189 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasSufficientMemory" Sep 17 01:22:15.323298 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.323219 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 01:22:15.323298 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.323230 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeHasSufficientPID" Sep 17 01:22:15.327468 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.327453 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-83.ec2.internal\" not found" node="ip-10-0-138-83.ec2.internal" Sep 17 01:22:15.333048 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.333033 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-83.ec2.internal\" not found" node="ip-10-0-138-83.ec2.internal" Sep 17 01:22:15.355940 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.355915 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-83.ec2.internal\" not found" Sep 17 01:22:15.373831 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.373811 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a1ee08f24cd676f4521086b970d2efd-config\") pod \"kube-apiserver-proxy-ip-10-0-138-83.ec2.internal\" (UID: \"5a1ee08f24cd676f4521086b970d2efd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-83.ec2.internal" Sep 17 01:22:15.373887 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.373834 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f31a61ef741e6dc3a21ef6dd2f12681f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal\" (UID: \"f31a61ef741e6dc3a21ef6dd2f12681f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal" Sep 17 01:22:15.373887 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.373852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31a61ef741e6dc3a21ef6dd2f12681f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal\" (UID: \"f31a61ef741e6dc3a21ef6dd2f12681f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal" Sep 17 01:22:15.456390 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.456356 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-83.ec2.internal\" not found" Sep 17 01:22:15.474766 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.474746 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a1ee08f24cd676f4521086b970d2efd-config\") pod \"kube-apiserver-proxy-ip-10-0-138-83.ec2.internal\" (UID: \"5a1ee08f24cd676f4521086b970d2efd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-83.ec2.internal" Sep 17 01:22:15.474830 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.474768 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f31a61ef741e6dc3a21ef6dd2f12681f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal\" (UID: \"f31a61ef741e6dc3a21ef6dd2f12681f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal" Sep 17 01:22:15.474830 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.474787 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31a61ef741e6dc3a21ef6dd2f12681f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal\" (UID: \"f31a61ef741e6dc3a21ef6dd2f12681f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal" Sep 17 01:22:15.474830 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.474813 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31a61ef741e6dc3a21ef6dd2f12681f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal\" (UID: \"f31a61ef741e6dc3a21ef6dd2f12681f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal" Sep 17 01:22:15.474947 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.474840 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a1ee08f24cd676f4521086b970d2efd-config\") pod \"kube-apiserver-proxy-ip-10-0-138-83.ec2.internal\" (UID: \"5a1ee08f24cd676f4521086b970d2efd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-83.ec2.internal" Sep 17 01:22:15.474947 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.474850 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f31a61ef741e6dc3a21ef6dd2f12681f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal\" (UID: \"f31a61ef741e6dc3a21ef6dd2f12681f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal" Sep 17 01:22:15.557201 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.557134 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-83.ec2.internal\" not found" Sep 17 01:22:15.630674 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.630644 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-83.ec2.internal" Sep 17 01:22:15.636025 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:15.636001 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal" Sep 17 01:22:15.658126 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.658105 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-83.ec2.internal\" not found" Sep 17 01:22:15.758684 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.758645 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-83.ec2.internal\" not found" Sep 17 01:22:15.859214 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.859154 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-83.ec2.internal\" not found" Sep 17 01:22:15.959711 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:15.959682 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-83.ec2.internal\" not found" Sep 17 01:22:16.003731 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:16.003704 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 17 01:22:16.004187 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:16.003898 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 01:22:16.004187 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:16.003900 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 01:22:16.060500 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:16.060475 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-83.ec2.internal\" not found" Sep 17 01:22:16.071327 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:16.071299 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 01:17:15 +0000 UTC" deadline="2028-02-23 20:05:45.021771966 +0000 UTC" Sep 17 01:22:16.071409 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:16.071327 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12594h43m28.950448189s" Sep 17 01:22:16.112236 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:16.112174 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 01:22:16.161062 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:16.161036 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-83.ec2.internal\" not found" Sep 17 01:22:16.244541 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:16.244511 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1ee08f24cd676f4521086b970d2efd.slice/crio-27a82b1cc47c62cb087ebdaf3c2b8374a8e6603ed0ecfb3bcb401a7fe42789d4 WatchSource:0}: Error finding container 27a82b1cc47c62cb087ebdaf3c2b8374a8e6603ed0ecfb3bcb401a7fe42789d4: Status 404 returned error can't find the container with id 27a82b1cc47c62cb087ebdaf3c2b8374a8e6603ed0ecfb3bcb401a7fe42789d4 Sep 17 01:22:16.249320 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:16.249304 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 01:22:16.262123 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:16.262106 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-83.ec2.internal\" not found" Sep 17 01:22:16.355577 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:16.355554 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 01:22:16.364974 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:16.364929 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31a61ef741e6dc3a21ef6dd2f12681f.slice/crio-061539746b32ffb1bb50f9c78be2603fa6e4a336d9c0cb7f193dfff01d5339a3 WatchSource:0}: Error finding container 061539746b32ffb1bb50f9c78be2603fa6e4a336d9c0cb7f193dfff01d5339a3: Status 404 returned error can't find the container with id 061539746b32ffb1bb50f9c78be2603fa6e4a336d9c0cb7f193dfff01d5339a3 Sep 17 01:22:16.371139 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:16.371119 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-83.ec2.internal" Sep 17 01:22:16.380393 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:16.380378 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 01:22:16.381346 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:16.381323 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal" Sep 17 01:22:16.392409 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:16.392393 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 01:22:17.009262 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.009232 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 01:22:17.046108 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.046061 2577 apiserver.go:52] "Watching apiserver" Sep 17 01:22:17.055126 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.055095 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 17 01:22:17.056203 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.056174 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-m9ptd","openshift-multus/network-metrics-daemon-rdhqv","openshift-network-diagnostics/network-check-target-lwr7v","kube-system/kube-apiserver-proxy-ip-10-0-138-83.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk","openshift-cluster-node-tuning-operator/tuned-9qjz8","openshift-multus/multus-cbqj4","openshift-network-operator/iptables-alerter-tkg5m","openshift-ovn-kubernetes/ovnkube-node-ttfl4","kube-system/global-pull-secret-syncer-lkbfd","kube-system/konnectivity-agent-p9475","openshift-dns/node-resolver-ttq94","openshift-image-registry/node-ca-km877","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal"] Sep 17 01:22:17.058899 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.058870 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:17.059026 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:17.058978 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkbfd" podUID="8a366399-229b-40ae-b916-4323338cb805" Sep 17 01:22:17.061380 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.061359 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-km877" Sep 17 01:22:17.063736 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.063668 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gmv4r\"" Sep 17 01:22:17.063987 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.063844 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 17 01:22:17.063987 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.063868 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 17 01:22:17.063987 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.063959 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 17 01:22:17.063987 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.063982 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:17.064218 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:17.064061 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwr7v" podUID="3d7e510c-fe28-46fc-b4b8-bef103adca93" Sep 17 01:22:17.066074 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.066036 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tkg5m" Sep 17 01:22:17.068335 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.068313 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 17 01:22:17.068427 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.068345 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 17 01:22:17.068604 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.068582 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 17 01:22:17.068837 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.068818 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nwrkj\"" Sep 17 01:22:17.069000 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.068588 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.070698 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.070650 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 17 01:22:17.071226 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.071207 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.072249 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.071911 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 17 01:22:17.072249 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.072188 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-27bw4\"" Sep 17 01:22:17.072820 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.072466 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 17 01:22:17.073995 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.073103 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 17 01:22:17.073995 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.073745 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 17 01:22:17.073995 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.073760 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 17 01:22:17.073995 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.074012 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.074280 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.074027 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jrl7k\"" Sep 17 01:22:17.076032 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.075919 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 17 01:22:17.076451 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.076431 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 17 01:22:17.076680 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.076663 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 17 01:22:17.076952 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.076931 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 17 01:22:17.077194 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.077158 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6bw4x\"" Sep 17 01:22:17.077525 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.077508 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 17 01:22:17.077525 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.077510 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 17 01:22:17.077702 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.077571 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.079278 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.079254 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:17.079393 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:17.079369 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdhqv" podUID="32c97ab8-07c6-4f64-9065-2441c8604a5a" Sep 17 01:22:17.080235 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.080215 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ctm22\"" Sep 17 01:22:17.080382 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.080363 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 17 01:22:17.080544 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.080457 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 17 01:22:17.080733 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.080707 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 17 01:22:17.080733 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.080722 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 17 01:22:17.082287 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.082266 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.083275 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.083251 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-run-openvswitch\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.083387 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.083350 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/18244e83-55ca-4e07-8d44-b5d6d06b32a4-ovnkube-config\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.083387 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.083377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/18244e83-55ca-4e07-8d44-b5d6d06b32a4-ovn-node-metrics-cert\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.083502 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.083445 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-run-systemd\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.083559 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.083472 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-var-lib-openvswitch\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.083559 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.083540 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-run-ovn-kubernetes\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.083667 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.083578 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82cjz\" (UniqueName: \"kubernetes.io/projected/18244e83-55ca-4e07-8d44-b5d6d06b32a4-kube-api-access-82cjz\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.083667 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.083620 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b6b4978-53de-4fa4-83cc-21d3de40ea24-host\") pod \"node-ca-km877\" (UID: \"7b6b4978-53de-4fa4-83cc-21d3de40ea24\") " pod="openshift-image-registry/node-ca-km877" Sep 17 01:22:17.083667 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.083654 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-socket-dir\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.083824 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.083689 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-sysconfig\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.083882 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.083831 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 01:22:17.084042 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.084024 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-kubernetes\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.084133 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.084062 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-sysctl-d\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.084196 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.084146 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-systemd\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.084196 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.084179 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-var-lib-kubelet\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.084306 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.084203 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-node-log\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.084306 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.084230 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f41bd42-f471-4885-a023-d2266738fdc3-host-slash\") pod \"iptables-alerter-tkg5m\" (UID: \"4f41bd42-f471-4885-a023-d2266738fdc3\") " pod="openshift-network-operator/iptables-alerter-tkg5m" Sep 17 01:22:17.084306 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.084260 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ckkr\" (UniqueName: \"kubernetes.io/projected/4f41bd42-f471-4885-a023-d2266738fdc3-kube-api-access-8ckkr\") pod \"iptables-alerter-tkg5m\" (UID: \"4f41bd42-f471-4885-a023-d2266738fdc3\") " pod="openshift-network-operator/iptables-alerter-tkg5m" Sep 17 01:22:17.084306 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.084299 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.084509 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.084325 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-registration-dir\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.084509 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.084396 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 17 01:22:17.084509 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.084500 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 17 01:22:17.084664 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.084576 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zx65r\"" Sep 17 01:22:17.084881 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.084856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-run\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.084969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.084893 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-systemd-units\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.084969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.084920 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.084969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.084947 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-modprobe-d\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.085134 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.084977 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/18244e83-55ca-4e07-8d44-b5d6d06b32a4-env-overrides\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.085189 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085170 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a366399-229b-40ae-b916-4323338cb805-kubelet-config\") pod \"global-pull-secret-syncer-lkbfd\" (UID: \"8a366399-229b-40ae-b916-4323338cb805\") " pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:17.085246 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085212 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret\") pod \"global-pull-secret-syncer-lkbfd\" (UID: \"8a366399-229b-40ae-b916-4323338cb805\") " pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:17.085246 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085239 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls682\" (UniqueName: \"kubernetes.io/projected/7b6b4978-53de-4fa4-83cc-21d3de40ea24-kube-api-access-ls682\") pod \"node-ca-km877\" (UID: \"7b6b4978-53de-4fa4-83cc-21d3de40ea24\") " pod="openshift-image-registry/node-ca-km877" Sep 17 01:22:17.085356 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-sysctl-conf\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.085356 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085309 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-cni-netd\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.085459 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085375 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/18244e83-55ca-4e07-8d44-b5d6d06b32a4-ovnkube-script-lib\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.085459 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085407 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b6b4978-53de-4fa4-83cc-21d3de40ea24-serviceca\") pod \"node-ca-km877\" (UID: \"7b6b4978-53de-4fa4-83cc-21d3de40ea24\") " pod="openshift-image-registry/node-ca-km877" Sep 17 01:22:17.085575 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f41bd42-f471-4885-a023-d2266738fdc3-iptables-alerter-script\") pod \"iptables-alerter-tkg5m\" (UID: \"4f41bd42-f471-4885-a023-d2266738fdc3\") " pod="openshift-network-operator/iptables-alerter-tkg5m" Sep 17 01:22:17.085631 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085576 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-sys-fs\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.085631 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085602 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-lib-modules\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.085733 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085638 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-tuned\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.085733 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085662 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-kubelet\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.085733 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085706 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-slash\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.085886 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085730 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-etc-openvswitch\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.085886 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085759 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a366399-229b-40ae-b916-4323338cb805-dbus\") pod \"global-pull-secret-syncer-lkbfd\" (UID: \"8a366399-229b-40ae-b916-4323338cb805\") " pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:17.085886 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085783 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-device-dir\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.085886 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085809 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vtp2\" (UniqueName: \"kubernetes.io/projected/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-kube-api-access-6vtp2\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.085886 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085832 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-host\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.085886 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085853 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef559f28-d4fd-4c60-b17a-626959a1bed9-tmp\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.085886 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085878 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-run-netns\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.086249 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085918 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-run-ovn\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.086249 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.085984 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-log-socket\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.086249 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.086017 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-cni-bin\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.086249 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.086051 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jks58\" (UniqueName: \"kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58\") pod \"network-check-target-lwr7v\" (UID: \"3d7e510c-fe28-46fc-b4b8-bef103adca93\") " pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:17.086249 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.086088 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-etc-selinux\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.086249 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.086117 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-sys\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.086249 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.086162 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxm9g\" (UniqueName: \"kubernetes.io/projected/ef559f28-d4fd-4c60-b17a-626959a1bed9-kube-api-access-lxm9g\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.087598 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.087550 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p9475" Sep 17 01:22:17.090220 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.090192 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 17 01:22:17.090325 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.090262 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 17 01:22:17.090713 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.090689 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gjp8b\"" Sep 17 01:22:17.092215 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.091229 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ttq94" Sep 17 01:22:17.094148 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.094123 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 17 01:22:17.094240 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.094148 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4mqqq\"" Sep 17 01:22:17.094332 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.094315 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 17 01:22:17.111064 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.111044 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6qc8j" Sep 17 01:22:17.122672 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.122652 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6qc8j" Sep 17 01:22:17.173301 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.173273 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 17 01:22:17.186338 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186311 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-cni-netd\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.186477 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186345 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/766b09df-bf87-4ace-8430-628323cd59c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.186477 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186376 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/766b09df-bf87-4ace-8430-628323cd59c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.186477 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186402 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/766b09df-bf87-4ace-8430-628323cd59c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.186477 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186422 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a32cfdee-7fb5-4b6f-bd3a-ce1441f806d4-konnectivity-ca\") pod \"konnectivity-agent-p9475\" (UID: \"a32cfdee-7fb5-4b6f-bd3a-ce1441f806d4\") " pod="kube-system/konnectivity-agent-p9475" Sep 17 01:22:17.186477 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-sys-fs\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.186477 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186449 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-cni-netd\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.186477 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-kubelet\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.186865 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186489 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-kubelet\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.186865 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186519 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a366399-229b-40ae-b916-4323338cb805-dbus\") pod \"global-pull-secret-syncer-lkbfd\" (UID: \"8a366399-229b-40ae-b916-4323338cb805\") " pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:17.186865 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186551 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-sys-fs\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.186865 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186558 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vtp2\" (UniqueName: \"kubernetes.io/projected/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-kube-api-access-6vtp2\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.186865 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186603 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-run-netns\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.186865 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-run-ovn\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.186865 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186652 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-run-netns\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.186865 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186683 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-run-ovn\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.186865 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186693 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-log-socket\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.186865 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186687 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a366399-229b-40ae-b916-4323338cb805-dbus\") pod \"global-pull-secret-syncer-lkbfd\" (UID: \"8a366399-229b-40ae-b916-4323338cb805\") " pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:17.186865 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186725 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-hostroot\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.186865 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186731 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-log-socket\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.186865 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jks58\" (UniqueName: \"kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58\") pod \"network-check-target-lwr7v\" (UID: \"3d7e510c-fe28-46fc-b4b8-bef103adca93\") " pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:17.186865 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186790 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-etc-selinux\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.186865 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186821 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxm9g\" (UniqueName: \"kubernetes.io/projected/ef559f28-d4fd-4c60-b17a-626959a1bed9-kube-api-access-lxm9g\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.187675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186875 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-etc-selinux\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.187675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186907 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-multus-conf-dir\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.187675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186933 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-etc-kubernetes\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.187675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186973 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs\") pod \"network-metrics-daemon-rdhqv\" (UID: \"32c97ab8-07c6-4f64-9065-2441c8604a5a\") " pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:17.187675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.186997 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a32cfdee-7fb5-4b6f-bd3a-ce1441f806d4-agent-certs\") pod \"konnectivity-agent-p9475\" (UID: \"a32cfdee-7fb5-4b6f-bd3a-ce1441f806d4\") " pod="kube-system/konnectivity-agent-p9475" Sep 17 01:22:17.187675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-run-systemd\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.187675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187053 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-var-lib-openvswitch\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.187675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187083 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82cjz\" (UniqueName: \"kubernetes.io/projected/18244e83-55ca-4e07-8d44-b5d6d06b32a4-kube-api-access-82cjz\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.187675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187108 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-sysconfig\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.187675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187114 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-var-lib-openvswitch\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.187675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187082 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-run-systemd\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.187675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187133 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-kubernetes\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.187675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187157 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-sysctl-d\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.187675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187172 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-sysconfig\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.187675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187178 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-node-log\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.187675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187208 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-node-log\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.187675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.188505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187242 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-kubernetes\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.188505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/18244e83-55ca-4e07-8d44-b5d6d06b32a4-ovnkube-script-lib\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.188505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187271 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.188505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-multus-daemon-config\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.188505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187303 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-sysctl-d\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.188505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187309 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78186fc7-1ae0-4bf9-89ec-38c47218f909-hosts-file\") pod \"node-resolver-ttq94\" (UID: \"78186fc7-1ae0-4bf9-89ec-38c47218f909\") " pod="openshift-dns/node-resolver-ttq94" Sep 17 01:22:17.188505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187366 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f41bd42-f471-4885-a023-d2266738fdc3-host-slash\") pod \"iptables-alerter-tkg5m\" (UID: \"4f41bd42-f471-4885-a023-d2266738fdc3\") " pod="openshift-network-operator/iptables-alerter-tkg5m" Sep 17 01:22:17.188505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187400 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ckkr\" (UniqueName: \"kubernetes.io/projected/4f41bd42-f471-4885-a023-d2266738fdc3-kube-api-access-8ckkr\") pod \"iptables-alerter-tkg5m\" (UID: \"4f41bd42-f471-4885-a023-d2266738fdc3\") " pod="openshift-network-operator/iptables-alerter-tkg5m" Sep 17 01:22:17.188505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187473 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f41bd42-f471-4885-a023-d2266738fdc3-host-slash\") pod \"iptables-alerter-tkg5m\" (UID: \"4f41bd42-f471-4885-a023-d2266738fdc3\") " pod="openshift-network-operator/iptables-alerter-tkg5m" Sep 17 01:22:17.188505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187494 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.188505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187518 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-run\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.188505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187549 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-systemd-units\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.188505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187576 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-host-var-lib-cni-multus\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.188505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187609 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/766b09df-bf87-4ace-8430-628323cd59c9-cnibin\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.188505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187620 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-run\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.188505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187628 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.188505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-modprobe-d\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.189304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187648 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-systemd-units\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.189304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187681 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-sysctl-conf\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.189304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b6b4978-53de-4fa4-83cc-21d3de40ea24-serviceca\") pod \"node-ca-km877\" (UID: \"7b6b4978-53de-4fa4-83cc-21d3de40ea24\") " pod="openshift-image-registry/node-ca-km877" Sep 17 01:22:17.189304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187761 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-cnibin\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.189304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187730 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-modprobe-d\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.189304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187789 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqgch\" (UniqueName: \"kubernetes.io/projected/32c97ab8-07c6-4f64-9065-2441c8604a5a-kube-api-access-jqgch\") pod \"network-metrics-daemon-rdhqv\" (UID: \"32c97ab8-07c6-4f64-9065-2441c8604a5a\") " pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:17.189304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187812 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-sysctl-conf\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.189304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187834 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f41bd42-f471-4885-a023-d2266738fdc3-iptables-alerter-script\") pod \"iptables-alerter-tkg5m\" (UID: \"4f41bd42-f471-4885-a023-d2266738fdc3\") " pod="openshift-network-operator/iptables-alerter-tkg5m" Sep 17 01:22:17.189304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187845 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/18244e83-55ca-4e07-8d44-b5d6d06b32a4-ovnkube-script-lib\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.189304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187856 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-lib-modules\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.189304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187876 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-tuned\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.189304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187900 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-slash\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.189304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-etc-openvswitch\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.189304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187964 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b6b4978-53de-4fa4-83cc-21d3de40ea24-host\") pod \"node-ca-km877\" (UID: \"7b6b4978-53de-4fa4-83cc-21d3de40ea24\") " pod="openshift-image-registry/node-ca-km877" Sep 17 01:22:17.189304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.187993 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-host-run-multus-certs\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.189304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188000 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-lib-modules\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.189304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188020 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/766b09df-bf87-4ace-8430-628323cd59c9-system-cni-dir\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.190094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-device-dir\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.190094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188054 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-etc-openvswitch\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.190094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188075 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-host\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.190094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188094 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-slash\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.190094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef559f28-d4fd-4c60-b17a-626959a1bed9-tmp\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.190094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-cni-bin\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.190094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188141 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b6b4978-53de-4fa4-83cc-21d3de40ea24-host\") pod \"node-ca-km877\" (UID: \"7b6b4978-53de-4fa4-83cc-21d3de40ea24\") " pod="openshift-image-registry/node-ca-km877" Sep 17 01:22:17.190094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188153 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/18244e83-55ca-4e07-8d44-b5d6d06b32a4-ovnkube-config\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.190094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/18244e83-55ca-4e07-8d44-b5d6d06b32a4-ovn-node-metrics-cert\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.190094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188194 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-host\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.190094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188209 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j6qx\" (UniqueName: \"kubernetes.io/projected/766b09df-bf87-4ace-8430-628323cd59c9-kube-api-access-6j6qx\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.190094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188311 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-device-dir\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.190094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188182 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b6b4978-53de-4fa4-83cc-21d3de40ea24-serviceca\") pod \"node-ca-km877\" (UID: \"7b6b4978-53de-4fa4-83cc-21d3de40ea24\") " pod="openshift-image-registry/node-ca-km877" Sep 17 01:22:17.190094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188382 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f41bd42-f471-4885-a023-d2266738fdc3-iptables-alerter-script\") pod \"iptables-alerter-tkg5m\" (UID: \"4f41bd42-f471-4885-a023-d2266738fdc3\") " pod="openshift-network-operator/iptables-alerter-tkg5m" Sep 17 01:22:17.190094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188441 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-cni-bin\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.190094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188482 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frjdt\" (UniqueName: \"kubernetes.io/projected/78186fc7-1ae0-4bf9-89ec-38c47218f909-kube-api-access-frjdt\") pod \"node-resolver-ttq94\" (UID: \"78186fc7-1ae0-4bf9-89ec-38c47218f909\") " pod="openshift-dns/node-resolver-ttq94" Sep 17 01:22:17.190094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188516 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-sys\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.190094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188559 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 17 01:22:17.190969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188570 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-sys\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.190969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188561 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-run-openvswitch\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.190969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188594 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-run-openvswitch\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.190969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188614 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-system-cni-dir\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.190969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188643 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-multus-socket-dir-parent\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.190969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188665 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/18244e83-55ca-4e07-8d44-b5d6d06b32a4-ovnkube-config\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.190969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c24hb\" (UniqueName: \"kubernetes.io/projected/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-kube-api-access-c24hb\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.190969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188707 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-run-ovn-kubernetes\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.190969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188734 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-host-run-k8s-cni-cncf-io\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.190969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188763 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-socket-dir\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.190969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188781 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18244e83-55ca-4e07-8d44-b5d6d06b32a4-host-run-ovn-kubernetes\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.190969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188789 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-systemd\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.190969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188851 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-var-lib-kubelet\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.190969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188910 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-host-run-netns\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.190969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188908 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-var-lib-kubelet\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.190969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188954 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-host-var-lib-cni-bin\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.190969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188966 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-systemd\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.191915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188980 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-socket-dir\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.191915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.188984 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-registration-dir\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.191915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.189031 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/18244e83-55ca-4e07-8d44-b5d6d06b32a4-env-overrides\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.191915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.189058 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-multus-cni-dir\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.191915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.189083 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78186fc7-1ae0-4bf9-89ec-38c47218f909-tmp-dir\") pod \"node-resolver-ttq94\" (UID: \"78186fc7-1ae0-4bf9-89ec-38c47218f909\") " pod="openshift-dns/node-resolver-ttq94" Sep 17 01:22:17.191915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.189109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a366399-229b-40ae-b916-4323338cb805-kubelet-config\") pod \"global-pull-secret-syncer-lkbfd\" (UID: \"8a366399-229b-40ae-b916-4323338cb805\") " pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:17.191915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.189111 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-registration-dir\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.191915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.189148 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret\") pod \"global-pull-secret-syncer-lkbfd\" (UID: \"8a366399-229b-40ae-b916-4323338cb805\") " pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:17.191915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.189220 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls682\" (UniqueName: \"kubernetes.io/projected/7b6b4978-53de-4fa4-83cc-21d3de40ea24-kube-api-access-ls682\") pod \"node-ca-km877\" (UID: \"7b6b4978-53de-4fa4-83cc-21d3de40ea24\") " pod="openshift-image-registry/node-ca-km877" Sep 17 01:22:17.191915 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:17.189257 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:22:17.191915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.189260 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-cni-binary-copy\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.191915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.189307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-host-var-lib-kubelet\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.191915 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:17.189360 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret podName:8a366399-229b-40ae-b916-4323338cb805 nodeName:}" failed. No retries permitted until 2026-09-17 01:22:17.689323325 +0000 UTC m=+2.924470497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret") pod "global-pull-secret-syncer-lkbfd" (UID: "8a366399-229b-40ae-b916-4323338cb805") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:22:17.191915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.189381 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a366399-229b-40ae-b916-4323338cb805-kubelet-config\") pod \"global-pull-secret-syncer-lkbfd\" (UID: \"8a366399-229b-40ae-b916-4323338cb805\") " pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:17.191915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.189392 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/766b09df-bf87-4ace-8430-628323cd59c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.191915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.189567 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/18244e83-55ca-4e07-8d44-b5d6d06b32a4-env-overrides\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.192801 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.192779 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef559f28-d4fd-4c60-b17a-626959a1bed9-etc-tuned\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.192996 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.192850 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/18244e83-55ca-4e07-8d44-b5d6d06b32a4-ovn-node-metrics-cert\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.193110 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.193032 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef559f28-d4fd-4c60-b17a-626959a1bed9-tmp\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.193522 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:17.193363 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:22:17.193522 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:17.193405 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:22:17.193522 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:17.193420 2577 projected.go:196] Error preparing data for projected volume kube-api-access-jks58 for pod openshift-network-diagnostics/network-check-target-lwr7v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:22:17.193522 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:17.193480 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58 podName:3d7e510c-fe28-46fc-b4b8-bef103adca93 nodeName:}" failed. No retries permitted until 2026-09-17 01:22:17.693461314 +0000 UTC m=+2.928608481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jks58" (UniqueName: "kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58") pod "network-check-target-lwr7v" (UID: "3d7e510c-fe28-46fc-b4b8-bef103adca93") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:22:17.195414 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.195086 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vtp2\" (UniqueName: \"kubernetes.io/projected/087a1d49-8d5d-4925-8d56-9c5e3383f0a9-kube-api-access-6vtp2\") pod \"aws-ebs-csi-driver-node-srvsk\" (UID: \"087a1d49-8d5d-4925-8d56-9c5e3383f0a9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.196164 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.196129 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxm9g\" (UniqueName: \"kubernetes.io/projected/ef559f28-d4fd-4c60-b17a-626959a1bed9-kube-api-access-lxm9g\") pod \"tuned-9qjz8\" (UID: \"ef559f28-d4fd-4c60-b17a-626959a1bed9\") " pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.196454 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.196432 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ckkr\" (UniqueName: \"kubernetes.io/projected/4f41bd42-f471-4885-a023-d2266738fdc3-kube-api-access-8ckkr\") pod \"iptables-alerter-tkg5m\" (UID: \"4f41bd42-f471-4885-a023-d2266738fdc3\") " pod="openshift-network-operator/iptables-alerter-tkg5m" Sep 17 01:22:17.197598 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.197529 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82cjz\" (UniqueName: \"kubernetes.io/projected/18244e83-55ca-4e07-8d44-b5d6d06b32a4-kube-api-access-82cjz\") pod \"ovnkube-node-ttfl4\" (UID: \"18244e83-55ca-4e07-8d44-b5d6d06b32a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.199904 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.199858 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls682\" (UniqueName: \"kubernetes.io/projected/7b6b4978-53de-4fa4-83cc-21d3de40ea24-kube-api-access-ls682\") pod \"node-ca-km877\" (UID: \"7b6b4978-53de-4fa4-83cc-21d3de40ea24\") " pod="openshift-image-registry/node-ca-km877" Sep 17 01:22:17.210548 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.210494 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal" event={"ID":"f31a61ef741e6dc3a21ef6dd2f12681f","Type":"ContainerStarted","Data":"061539746b32ffb1bb50f9c78be2603fa6e4a336d9c0cb7f193dfff01d5339a3"} Sep 17 01:22:17.211740 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.211715 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-83.ec2.internal" event={"ID":"5a1ee08f24cd676f4521086b970d2efd","Type":"ContainerStarted","Data":"27a82b1cc47c62cb087ebdaf3c2b8374a8e6603ed0ecfb3bcb401a7fe42789d4"} Sep 17 01:22:17.290331 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290255 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j6qx\" (UniqueName: \"kubernetes.io/projected/766b09df-bf87-4ace-8430-628323cd59c9-kube-api-access-6j6qx\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.290331 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290302 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frjdt\" (UniqueName: \"kubernetes.io/projected/78186fc7-1ae0-4bf9-89ec-38c47218f909-kube-api-access-frjdt\") pod \"node-resolver-ttq94\" (UID: \"78186fc7-1ae0-4bf9-89ec-38c47218f909\") " pod="openshift-dns/node-resolver-ttq94" Sep 17 01:22:17.290331 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290333 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-system-cni-dir\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.290606 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290407 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-system-cni-dir\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.290606 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290442 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-multus-socket-dir-parent\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.290606 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290494 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c24hb\" (UniqueName: \"kubernetes.io/projected/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-kube-api-access-c24hb\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.290606 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290523 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-host-run-k8s-cni-cncf-io\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.290606 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290555 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-host-run-netns\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.290606 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-host-var-lib-cni-bin\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.290606 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290540 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-multus-socket-dir-parent\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.291053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290605 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-host-run-netns\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.291053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290610 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-multus-cni-dir\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.291053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290621 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-host-run-k8s-cni-cncf-io\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.291053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290654 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78186fc7-1ae0-4bf9-89ec-38c47218f909-tmp-dir\") pod \"node-resolver-ttq94\" (UID: \"78186fc7-1ae0-4bf9-89ec-38c47218f909\") " pod="openshift-dns/node-resolver-ttq94" Sep 17 01:22:17.291053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290682 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-host-var-lib-cni-bin\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.291053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290692 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-multus-cni-dir\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.291053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-cni-binary-copy\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.291053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290764 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-host-var-lib-kubelet\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.291053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290799 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/766b09df-bf87-4ace-8430-628323cd59c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.291053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290828 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/766b09df-bf87-4ace-8430-628323cd59c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.291053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290860 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/766b09df-bf87-4ace-8430-628323cd59c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.291053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290865 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-host-var-lib-kubelet\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.291053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290887 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/766b09df-bf87-4ace-8430-628323cd59c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.291053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.290961 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a32cfdee-7fb5-4b6f-bd3a-ce1441f806d4-konnectivity-ca\") pod \"konnectivity-agent-p9475\" (UID: \"a32cfdee-7fb5-4b6f-bd3a-ce1441f806d4\") " pod="kube-system/konnectivity-agent-p9475" Sep 17 01:22:17.291053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-hostroot\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.291053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291035 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78186fc7-1ae0-4bf9-89ec-38c47218f909-tmp-dir\") pod \"node-resolver-ttq94\" (UID: \"78186fc7-1ae0-4bf9-89ec-38c47218f909\") " pod="openshift-dns/node-resolver-ttq94" Sep 17 01:22:17.291053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291066 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-multus-conf-dir\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.291905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291089 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-etc-kubernetes\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.291905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs\") pod \"network-metrics-daemon-rdhqv\" (UID: \"32c97ab8-07c6-4f64-9065-2441c8604a5a\") " pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:17.291905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a32cfdee-7fb5-4b6f-bd3a-ce1441f806d4-agent-certs\") pod \"konnectivity-agent-p9475\" (UID: \"a32cfdee-7fb5-4b6f-bd3a-ce1441f806d4\") " pod="kube-system/konnectivity-agent-p9475" Sep 17 01:22:17.291905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291178 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-multus-daemon-config\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.291905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78186fc7-1ae0-4bf9-89ec-38c47218f909-hosts-file\") pod \"node-resolver-ttq94\" (UID: \"78186fc7-1ae0-4bf9-89ec-38c47218f909\") " pod="openshift-dns/node-resolver-ttq94" Sep 17 01:22:17.291905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291241 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-host-var-lib-cni-multus\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.291905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291277 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/766b09df-bf87-4ace-8430-628323cd59c9-cnibin\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.291905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291307 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-cnibin\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.291905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqgch\" (UniqueName: \"kubernetes.io/projected/32c97ab8-07c6-4f64-9065-2441c8604a5a-kube-api-access-jqgch\") pod \"network-metrics-daemon-rdhqv\" (UID: \"32c97ab8-07c6-4f64-9065-2441c8604a5a\") " pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:17.291905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291366 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-host-run-multus-certs\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.291905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291392 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/766b09df-bf87-4ace-8430-628323cd59c9-system-cni-dir\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.291905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291418 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/766b09df-bf87-4ace-8430-628323cd59c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.291905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291422 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-cni-binary-copy\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.291905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291449 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/766b09df-bf87-4ace-8430-628323cd59c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.291905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291503 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-multus-conf-dir\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.291905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291511 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78186fc7-1ae0-4bf9-89ec-38c47218f909-hosts-file\") pod \"node-resolver-ttq94\" (UID: \"78186fc7-1ae0-4bf9-89ec-38c47218f909\") " pod="openshift-dns/node-resolver-ttq94" Sep 17 01:22:17.291905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291553 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/766b09df-bf87-4ace-8430-628323cd59c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.291905 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:17.291574 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:22:17.292811 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291576 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/766b09df-bf87-4ace-8430-628323cd59c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.292811 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291609 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/766b09df-bf87-4ace-8430-628323cd59c9-cnibin\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.292811 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291616 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-cnibin\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.292811 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291560 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-hostroot\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.292811 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291628 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/766b09df-bf87-4ace-8430-628323cd59c9-system-cni-dir\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.292811 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:17.291626 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs podName:32c97ab8-07c6-4f64-9065-2441c8604a5a nodeName:}" failed. No retries permitted until 2026-09-17 01:22:17.791612706 +0000 UTC m=+3.026759874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs") pod "network-metrics-daemon-rdhqv" (UID: "32c97ab8-07c6-4f64-9065-2441c8604a5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:22:17.292811 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291669 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-host-var-lib-cni-multus\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.292811 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291674 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-host-run-multus-certs\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.292811 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291707 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-etc-kubernetes\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.292811 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.291912 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a32cfdee-7fb5-4b6f-bd3a-ce1441f806d4-konnectivity-ca\") pod \"konnectivity-agent-p9475\" (UID: \"a32cfdee-7fb5-4b6f-bd3a-ce1441f806d4\") " pod="kube-system/konnectivity-agent-p9475" Sep 17 01:22:17.292811 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.292204 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-multus-daemon-config\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.294530 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.294507 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a32cfdee-7fb5-4b6f-bd3a-ce1441f806d4-agent-certs\") pod \"konnectivity-agent-p9475\" (UID: \"a32cfdee-7fb5-4b6f-bd3a-ce1441f806d4\") " pod="kube-system/konnectivity-agent-p9475" Sep 17 01:22:17.298742 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.298718 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frjdt\" (UniqueName: \"kubernetes.io/projected/78186fc7-1ae0-4bf9-89ec-38c47218f909-kube-api-access-frjdt\") pod \"node-resolver-ttq94\" (UID: \"78186fc7-1ae0-4bf9-89ec-38c47218f909\") " pod="openshift-dns/node-resolver-ttq94" Sep 17 01:22:17.298742 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.298735 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c24hb\" (UniqueName: \"kubernetes.io/projected/282d394d-c1cd-4368-bbb5-f8a67ab9c07b-kube-api-access-c24hb\") pod \"multus-cbqj4\" (UID: \"282d394d-c1cd-4368-bbb5-f8a67ab9c07b\") " pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.300241 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.300195 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j6qx\" (UniqueName: \"kubernetes.io/projected/766b09df-bf87-4ace-8430-628323cd59c9-kube-api-access-6j6qx\") pod \"multus-additional-cni-plugins-m9ptd\" (UID: \"766b09df-bf87-4ace-8430-628323cd59c9\") " pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.302145 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.302113 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqgch\" (UniqueName: \"kubernetes.io/projected/32c97ab8-07c6-4f64-9065-2441c8604a5a-kube-api-access-jqgch\") pod \"network-metrics-daemon-rdhqv\" (UID: \"32c97ab8-07c6-4f64-9065-2441c8604a5a\") " pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:17.345310 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.345282 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 01:22:17.378295 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.378263 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-km877" Sep 17 01:22:17.387049 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.387024 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tkg5m" Sep 17 01:22:17.399143 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.399124 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" Sep 17 01:22:17.406112 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.406090 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p9475" Sep 17 01:22:17.413984 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.413960 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" Sep 17 01:22:17.422053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.422025 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:17.429025 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.429005 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cbqj4" Sep 17 01:22:17.437069 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.437044 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m9ptd" Sep 17 01:22:17.442139 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.442122 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ttq94" Sep 17 01:22:17.695638 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.695557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret\") pod \"global-pull-secret-syncer-lkbfd\" (UID: \"8a366399-229b-40ae-b916-4323338cb805\") " pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:17.695638 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.695609 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jks58\" (UniqueName: \"kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58\") pod \"network-check-target-lwr7v\" (UID: \"3d7e510c-fe28-46fc-b4b8-bef103adca93\") " pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:17.695868 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:17.695733 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:22:17.695868 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:17.695750 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:22:17.695868 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:17.695768 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:22:17.695868 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:17.695781 2577 projected.go:196] Error preparing data for projected volume kube-api-access-jks58 for pod openshift-network-diagnostics/network-check-target-lwr7v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:22:17.695868 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:17.695810 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret podName:8a366399-229b-40ae-b916-4323338cb805 nodeName:}" failed. No retries permitted until 2026-09-17 01:22:18.695788879 +0000 UTC m=+3.930936066 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret") pod "global-pull-secret-syncer-lkbfd" (UID: "8a366399-229b-40ae-b916-4323338cb805") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:22:17.695868 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:17.695834 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58 podName:3d7e510c-fe28-46fc-b4b8-bef103adca93 nodeName:}" failed. No retries permitted until 2026-09-17 01:22:18.695818111 +0000 UTC m=+3.930965298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jks58" (UniqueName: "kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58") pod "network-check-target-lwr7v" (UID: "3d7e510c-fe28-46fc-b4b8-bef103adca93") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:22:17.796522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:17.796490 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs\") pod \"network-metrics-daemon-rdhqv\" (UID: \"32c97ab8-07c6-4f64-9065-2441c8604a5a\") " pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:17.796694 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:17.796663 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:22:17.796754 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:17.796739 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs podName:32c97ab8-07c6-4f64-9065-2441c8604a5a nodeName:}" failed. No retries permitted until 2026-09-17 01:22:18.796723772 +0000 UTC m=+4.031870940 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs") pod "network-metrics-daemon-rdhqv" (UID: "32c97ab8-07c6-4f64-9065-2441c8604a5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:22:18.126355 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:18.126240 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 01:17:17 +0000 UTC" deadline="2028-06-28 12:46:10.63374314 +0000 UTC" Sep 17 01:22:18.126355 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:18.126298 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15611h23m52.507449621s" Sep 17 01:22:18.205458 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:18.205431 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:18.205616 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:18.205533 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwr7v" podUID="3d7e510c-fe28-46fc-b4b8-bef103adca93" Sep 17 01:22:18.265047 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:18.264983 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6b4978_53de_4fa4_83cc_21d3de40ea24.slice/crio-1960e80b4853c25703211b848a1f45657639193f48b65a5f4d8915e036983112 WatchSource:0}: Error finding container 1960e80b4853c25703211b848a1f45657639193f48b65a5f4d8915e036983112: Status 404 returned error can't find the container with id 1960e80b4853c25703211b848a1f45657639193f48b65a5f4d8915e036983112 Sep 17 01:22:18.267012 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:18.266954 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f41bd42_f471_4885_a023_d2266738fdc3.slice/crio-153cd9fe869814b101f794b8ccb6a26adec4dd4f990f95f3c6fde7561ecf4801 WatchSource:0}: Error finding container 153cd9fe869814b101f794b8ccb6a26adec4dd4f990f95f3c6fde7561ecf4801: Status 404 returned error can't find the container with id 153cd9fe869814b101f794b8ccb6a26adec4dd4f990f95f3c6fde7561ecf4801 Sep 17 01:22:18.270664 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:18.270625 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087a1d49_8d5d_4925_8d56_9c5e3383f0a9.slice/crio-16d1fd6c53643d482152833b0aeb9c42caedf75ce28f4f6953c1f9adcf84be1a WatchSource:0}: Error finding container 16d1fd6c53643d482152833b0aeb9c42caedf75ce28f4f6953c1f9adcf84be1a: Status 404 returned error can't find the container with id 16d1fd6c53643d482152833b0aeb9c42caedf75ce28f4f6953c1f9adcf84be1a Sep 17 01:22:18.277135 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:18.277108 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32cfdee_7fb5_4b6f_bd3a_ce1441f806d4.slice/crio-861cb5b03c1263ce04d56397b9c12c740bf8982b82758e6688c171af17e3c303 WatchSource:0}: Error finding container 861cb5b03c1263ce04d56397b9c12c740bf8982b82758e6688c171af17e3c303: Status 404 returned error can't find the container with id 861cb5b03c1263ce04d56397b9c12c740bf8982b82758e6688c171af17e3c303 Sep 17 01:22:18.277923 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:18.277906 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78186fc7_1ae0_4bf9_89ec_38c47218f909.slice/crio-13abf13ffebd0306cc690f8cbaf9e6c1a22c71f4ab99001e6e1e8e8d8e0c26d0 WatchSource:0}: Error finding container 13abf13ffebd0306cc690f8cbaf9e6c1a22c71f4ab99001e6e1e8e8d8e0c26d0: Status 404 returned error can't find the container with id 13abf13ffebd0306cc690f8cbaf9e6c1a22c71f4ab99001e6e1e8e8d8e0c26d0 Sep 17 01:22:18.279819 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:18.279477 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766b09df_bf87_4ace_8430_628323cd59c9.slice/crio-e53ca541860e9e4c23cea2aeb81add3714e8992915dcda1d33677d56d9a05d92 WatchSource:0}: Error finding container e53ca541860e9e4c23cea2aeb81add3714e8992915dcda1d33677d56d9a05d92: Status 404 returned error can't find the container with id e53ca541860e9e4c23cea2aeb81add3714e8992915dcda1d33677d56d9a05d92 Sep 17 01:22:18.280953 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:18.280667 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282d394d_c1cd_4368_bbb5_f8a67ab9c07b.slice/crio-280c307782b3617837c0969d50326334420145ec9c6971224c2204b8ef81109d WatchSource:0}: Error finding container 280c307782b3617837c0969d50326334420145ec9c6971224c2204b8ef81109d: Status 404 returned error can't find the container with id 280c307782b3617837c0969d50326334420145ec9c6971224c2204b8ef81109d Sep 17 01:22:18.282412 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:18.282318 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef559f28_d4fd_4c60_b17a_626959a1bed9.slice/crio-6c96c9e77d01fe07a10714107155db3b240dcae6eb71ca2ee120b9c52e91aa20 WatchSource:0}: Error finding container 6c96c9e77d01fe07a10714107155db3b240dcae6eb71ca2ee120b9c52e91aa20: Status 404 returned error can't find the container with id 6c96c9e77d01fe07a10714107155db3b240dcae6eb71ca2ee120b9c52e91aa20 Sep 17 01:22:18.283668 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:18.283646 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18244e83_55ca_4e07_8d44_b5d6d06b32a4.slice/crio-28542d047d6d4a9ad814790bf98fccd29136a3b58a44fb2c38390f92523fc879 WatchSource:0}: Error finding container 28542d047d6d4a9ad814790bf98fccd29136a3b58a44fb2c38390f92523fc879: Status 404 returned error can't find the container with id 28542d047d6d4a9ad814790bf98fccd29136a3b58a44fb2c38390f92523fc879 Sep 17 01:22:18.703266 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:18.703225 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret\") pod \"global-pull-secret-syncer-lkbfd\" (UID: \"8a366399-229b-40ae-b916-4323338cb805\") " pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:18.703458 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:18.703291 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jks58\" (UniqueName: \"kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58\") pod \"network-check-target-lwr7v\" (UID: \"3d7e510c-fe28-46fc-b4b8-bef103adca93\") " pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:18.703535 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:18.703490 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:22:18.703535 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:18.703513 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:22:18.703535 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:18.703524 2577 projected.go:196] Error preparing data for projected volume kube-api-access-jks58 for pod openshift-network-diagnostics/network-check-target-lwr7v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:22:18.703676 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:18.703568 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:22:18.703676 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:18.703583 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58 podName:3d7e510c-fe28-46fc-b4b8-bef103adca93 nodeName:}" failed. No retries permitted until 2026-09-17 01:22:20.703567994 +0000 UTC m=+5.938715183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jks58" (UniqueName: "kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58") pod "network-check-target-lwr7v" (UID: "3d7e510c-fe28-46fc-b4b8-bef103adca93") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:22:18.703676 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:18.703638 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret podName:8a366399-229b-40ae-b916-4323338cb805 nodeName:}" failed. No retries permitted until 2026-09-17 01:22:20.703619349 +0000 UTC m=+5.938766523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret") pod "global-pull-secret-syncer-lkbfd" (UID: "8a366399-229b-40ae-b916-4323338cb805") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:22:18.804492 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:18.804412 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs\") pod \"network-metrics-daemon-rdhqv\" (UID: \"32c97ab8-07c6-4f64-9065-2441c8604a5a\") " pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:18.804662 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:18.804609 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:22:18.804727 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:18.804673 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs podName:32c97ab8-07c6-4f64-9065-2441c8604a5a nodeName:}" failed. No retries permitted until 2026-09-17 01:22:20.804654713 +0000 UTC m=+6.039801887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs") pod "network-metrics-daemon-rdhqv" (UID: "32c97ab8-07c6-4f64-9065-2441c8604a5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:22:19.127112 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:19.126966 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 01:17:17 +0000 UTC" deadline="2028-04-11 18:05:37.833404882 +0000 UTC" Sep 17 01:22:19.127112 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:19.127010 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13744h43m18.706399876s" Sep 17 01:22:19.216768 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:19.216148 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:19.216768 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:19.216331 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdhqv" podUID="32c97ab8-07c6-4f64-9065-2441c8604a5a" Sep 17 01:22:19.216768 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:19.216506 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:19.216768 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:19.216631 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkbfd" podUID="8a366399-229b-40ae-b916-4323338cb805" Sep 17 01:22:19.249879 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:19.249839 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" event={"ID":"ef559f28-d4fd-4c60-b17a-626959a1bed9","Type":"ContainerStarted","Data":"6c96c9e77d01fe07a10714107155db3b240dcae6eb71ca2ee120b9c52e91aa20"} Sep 17 01:22:19.259947 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:19.259842 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" event={"ID":"087a1d49-8d5d-4925-8d56-9c5e3383f0a9","Type":"ContainerStarted","Data":"16d1fd6c53643d482152833b0aeb9c42caedf75ce28f4f6953c1f9adcf84be1a"} Sep 17 01:22:19.265582 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:19.265524 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tkg5m" event={"ID":"4f41bd42-f471-4885-a023-d2266738fdc3","Type":"ContainerStarted","Data":"153cd9fe869814b101f794b8ccb6a26adec4dd4f990f95f3c6fde7561ecf4801"} Sep 17 01:22:19.272415 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:19.272322 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-km877" event={"ID":"7b6b4978-53de-4fa4-83cc-21d3de40ea24","Type":"ContainerStarted","Data":"1960e80b4853c25703211b848a1f45657639193f48b65a5f4d8915e036983112"} Sep 17 01:22:19.280077 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:19.280048 2577 generic.go:356] "Generic (PLEG): container finished" podID="f31a61ef741e6dc3a21ef6dd2f12681f" containerID="4dc78e395578629f9cc4fa9f752a03f15a51146b5f1531698bea141e280790c8" exitCode=0 Sep 17 01:22:19.280254 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:19.280142 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal" event={"ID":"f31a61ef741e6dc3a21ef6dd2f12681f","Type":"ContainerDied","Data":"4dc78e395578629f9cc4fa9f752a03f15a51146b5f1531698bea141e280790c8"} Sep 17 01:22:19.287928 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:19.287893 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" event={"ID":"18244e83-55ca-4e07-8d44-b5d6d06b32a4","Type":"ContainerStarted","Data":"28542d047d6d4a9ad814790bf98fccd29136a3b58a44fb2c38390f92523fc879"} Sep 17 01:22:19.303808 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:19.303768 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cbqj4" event={"ID":"282d394d-c1cd-4368-bbb5-f8a67ab9c07b","Type":"ContainerStarted","Data":"280c307782b3617837c0969d50326334420145ec9c6971224c2204b8ef81109d"} Sep 17 01:22:19.307147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:19.307121 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9ptd" event={"ID":"766b09df-bf87-4ace-8430-628323cd59c9","Type":"ContainerStarted","Data":"e53ca541860e9e4c23cea2aeb81add3714e8992915dcda1d33677d56d9a05d92"} Sep 17 01:22:19.310928 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:19.310904 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ttq94" event={"ID":"78186fc7-1ae0-4bf9-89ec-38c47218f909","Type":"ContainerStarted","Data":"13abf13ffebd0306cc690f8cbaf9e6c1a22c71f4ab99001e6e1e8e8d8e0c26d0"} Sep 17 01:22:19.314100 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:19.314078 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p9475" event={"ID":"a32cfdee-7fb5-4b6f-bd3a-ce1441f806d4","Type":"ContainerStarted","Data":"861cb5b03c1263ce04d56397b9c12c740bf8982b82758e6688c171af17e3c303"} Sep 17 01:22:19.321301 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:19.321273 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-83.ec2.internal" event={"ID":"5a1ee08f24cd676f4521086b970d2efd","Type":"ContainerStarted","Data":"e09f8988aebe9904a20cfc7e80501e2f03fd602521564feaab1739919974175e"} Sep 17 01:22:20.206212 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:20.206173 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:20.206969 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:20.206307 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwr7v" podUID="3d7e510c-fe28-46fc-b4b8-bef103adca93" Sep 17 01:22:20.340219 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:20.340184 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal" event={"ID":"f31a61ef741e6dc3a21ef6dd2f12681f","Type":"ContainerStarted","Data":"d74642c5edce8959012327d7ec255e44e98f7f1a3843f849ca3c77514b1c0e40"} Sep 17 01:22:20.354276 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:20.354220 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-83.ec2.internal" podStartSLOduration=4.354199161 podStartE2EDuration="4.354199161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:20.353436535 +0000 UTC m=+5.588583726" watchObservedRunningTime="2026-09-17 01:22:20.354199161 +0000 UTC m=+5.589346352" Sep 17 01:22:20.354544 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:20.354513 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-83.ec2.internal" podStartSLOduration=4.354503133 podStartE2EDuration="4.354503133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:19.334646341 +0000 UTC m=+4.569793535" watchObservedRunningTime="2026-09-17 01:22:20.354503133 +0000 UTC m=+5.589650323" Sep 17 01:22:20.732944 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:20.732054 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret\") pod \"global-pull-secret-syncer-lkbfd\" (UID: \"8a366399-229b-40ae-b916-4323338cb805\") " pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:20.732944 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:20.732144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jks58\" (UniqueName: \"kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58\") pod \"network-check-target-lwr7v\" (UID: \"3d7e510c-fe28-46fc-b4b8-bef103adca93\") " pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:20.732944 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:20.732346 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:22:20.732944 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:20.732369 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:22:20.732944 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:20.732383 2577 projected.go:196] Error preparing data for projected volume kube-api-access-jks58 for pod openshift-network-diagnostics/network-check-target-lwr7v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:22:20.732944 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:20.732441 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58 podName:3d7e510c-fe28-46fc-b4b8-bef103adca93 nodeName:}" failed. No retries permitted until 2026-09-17 01:22:24.732422834 +0000 UTC m=+9.967570007 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jks58" (UniqueName: "kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58") pod "network-check-target-lwr7v" (UID: "3d7e510c-fe28-46fc-b4b8-bef103adca93") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:22:20.732944 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:20.732848 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:22:20.732944 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:20.732905 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret podName:8a366399-229b-40ae-b916-4323338cb805 nodeName:}" failed. No retries permitted until 2026-09-17 01:22:24.732886664 +0000 UTC m=+9.968033835 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret") pod "global-pull-secret-syncer-lkbfd" (UID: "8a366399-229b-40ae-b916-4323338cb805") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:22:20.832837 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:20.832804 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs\") pod \"network-metrics-daemon-rdhqv\" (UID: \"32c97ab8-07c6-4f64-9065-2441c8604a5a\") " pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:20.833015 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:20.832976 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:22:20.833088 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:20.833039 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs podName:32c97ab8-07c6-4f64-9065-2441c8604a5a nodeName:}" failed. No retries permitted until 2026-09-17 01:22:24.83302024 +0000 UTC m=+10.068167420 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs") pod "network-metrics-daemon-rdhqv" (UID: "32c97ab8-07c6-4f64-9065-2441c8604a5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:22:21.207147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:21.207025 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:21.207147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:21.207064 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:21.207638 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:21.207151 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkbfd" podUID="8a366399-229b-40ae-b916-4323338cb805" Sep 17 01:22:21.207638 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:21.207560 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdhqv" podUID="32c97ab8-07c6-4f64-9065-2441c8604a5a" Sep 17 01:22:22.205965 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:22.205471 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:22.205965 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:22.205597 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwr7v" podUID="3d7e510c-fe28-46fc-b4b8-bef103adca93" Sep 17 01:22:23.207097 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:23.206214 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:23.207097 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:23.207093 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkbfd" podUID="8a366399-229b-40ae-b916-4323338cb805" Sep 17 01:22:23.207625 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:23.207472 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:23.207951 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:23.207797 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdhqv" podUID="32c97ab8-07c6-4f64-9065-2441c8604a5a" Sep 17 01:22:24.205883 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:24.205804 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:24.206048 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:24.206012 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwr7v" podUID="3d7e510c-fe28-46fc-b4b8-bef103adca93" Sep 17 01:22:24.765957 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:24.765911 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret\") pod \"global-pull-secret-syncer-lkbfd\" (UID: \"8a366399-229b-40ae-b916-4323338cb805\") " pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:24.766379 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:24.765984 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jks58\" (UniqueName: \"kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58\") pod \"network-check-target-lwr7v\" (UID: \"3d7e510c-fe28-46fc-b4b8-bef103adca93\") " pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:24.766379 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:24.766049 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:22:24.766379 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:24.766122 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret podName:8a366399-229b-40ae-b916-4323338cb805 nodeName:}" failed. No retries permitted until 2026-09-17 01:22:32.766100505 +0000 UTC m=+18.001247688 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret") pod "global-pull-secret-syncer-lkbfd" (UID: "8a366399-229b-40ae-b916-4323338cb805") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:22:24.766379 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:24.766161 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:22:24.766379 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:24.766186 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:22:24.766379 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:24.766200 2577 projected.go:196] Error preparing data for projected volume kube-api-access-jks58 for pod openshift-network-diagnostics/network-check-target-lwr7v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:22:24.766379 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:24.766260 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58 podName:3d7e510c-fe28-46fc-b4b8-bef103adca93 nodeName:}" failed. No retries permitted until 2026-09-17 01:22:32.766243498 +0000 UTC m=+18.001390672 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jks58" (UniqueName: "kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58") pod "network-check-target-lwr7v" (UID: "3d7e510c-fe28-46fc-b4b8-bef103adca93") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:22:24.866983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:24.866944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs\") pod \"network-metrics-daemon-rdhqv\" (UID: \"32c97ab8-07c6-4f64-9065-2441c8604a5a\") " pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:24.867144 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:24.867123 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:22:24.867205 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:24.867198 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs podName:32c97ab8-07c6-4f64-9065-2441c8604a5a nodeName:}" failed. No retries permitted until 2026-09-17 01:22:32.867177843 +0000 UTC m=+18.102325026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs") pod "network-metrics-daemon-rdhqv" (UID: "32c97ab8-07c6-4f64-9065-2441c8604a5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:22:25.207549 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:25.207242 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:25.207549 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:25.207413 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdhqv" podUID="32c97ab8-07c6-4f64-9065-2441c8604a5a" Sep 17 01:22:25.207816 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:25.207792 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:25.207959 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:25.207908 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkbfd" podUID="8a366399-229b-40ae-b916-4323338cb805" Sep 17 01:22:26.205548 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:26.205516 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:26.206005 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:26.205652 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwr7v" podUID="3d7e510c-fe28-46fc-b4b8-bef103adca93" Sep 17 01:22:27.206270 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:27.206238 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:27.206681 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:27.206376 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkbfd" podUID="8a366399-229b-40ae-b916-4323338cb805" Sep 17 01:22:27.206681 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:27.206431 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:27.206681 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:27.206532 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdhqv" podUID="32c97ab8-07c6-4f64-9065-2441c8604a5a" Sep 17 01:22:28.206234 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:28.206203 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:28.206410 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:28.206321 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwr7v" podUID="3d7e510c-fe28-46fc-b4b8-bef103adca93" Sep 17 01:22:29.205969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:29.205876 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:29.206123 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:29.206021 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkbfd" podUID="8a366399-229b-40ae-b916-4323338cb805" Sep 17 01:22:29.206123 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:29.206080 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:29.206220 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:29.206198 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdhqv" podUID="32c97ab8-07c6-4f64-9065-2441c8604a5a" Sep 17 01:22:30.206084 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:30.206050 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:30.206479 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:30.206155 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwr7v" podUID="3d7e510c-fe28-46fc-b4b8-bef103adca93" Sep 17 01:22:31.205510 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:31.205475 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:31.205510 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:31.205499 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:31.205761 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:31.205618 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdhqv" podUID="32c97ab8-07c6-4f64-9065-2441c8604a5a" Sep 17 01:22:31.205761 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:31.205721 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkbfd" podUID="8a366399-229b-40ae-b916-4323338cb805" Sep 17 01:22:32.205966 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:32.205916 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:32.206503 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:32.206072 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwr7v" podUID="3d7e510c-fe28-46fc-b4b8-bef103adca93" Sep 17 01:22:32.827877 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:32.827798 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret\") pod \"global-pull-secret-syncer-lkbfd\" (UID: \"8a366399-229b-40ae-b916-4323338cb805\") " pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:32.827877 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:32.827849 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jks58\" (UniqueName: \"kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58\") pod \"network-check-target-lwr7v\" (UID: \"3d7e510c-fe28-46fc-b4b8-bef103adca93\") " pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:32.828103 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:32.827987 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:22:32.828103 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:32.828070 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret podName:8a366399-229b-40ae-b916-4323338cb805 nodeName:}" failed. No retries permitted until 2026-09-17 01:22:48.828049819 +0000 UTC m=+34.063196988 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret") pod "global-pull-secret-syncer-lkbfd" (UID: "8a366399-229b-40ae-b916-4323338cb805") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:22:32.828103 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:32.828095 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:22:32.828263 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:32.828119 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:22:32.828263 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:32.828131 2577 projected.go:196] Error preparing data for projected volume kube-api-access-jks58 for pod openshift-network-diagnostics/network-check-target-lwr7v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:22:32.828263 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:32.828182 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58 podName:3d7e510c-fe28-46fc-b4b8-bef103adca93 nodeName:}" failed. No retries permitted until 2026-09-17 01:22:48.828167258 +0000 UTC m=+34.063314448 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jks58" (UniqueName: "kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58") pod "network-check-target-lwr7v" (UID: "3d7e510c-fe28-46fc-b4b8-bef103adca93") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:22:32.928997 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:32.928942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs\") pod \"network-metrics-daemon-rdhqv\" (UID: \"32c97ab8-07c6-4f64-9065-2441c8604a5a\") " pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:32.929152 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:32.929066 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:22:32.929152 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:32.929143 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs podName:32c97ab8-07c6-4f64-9065-2441c8604a5a nodeName:}" failed. No retries permitted until 2026-09-17 01:22:48.929126696 +0000 UTC m=+34.164273861 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs") pod "network-metrics-daemon-rdhqv" (UID: "32c97ab8-07c6-4f64-9065-2441c8604a5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:22:33.205675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:33.205574 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:33.205826 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:33.205574 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:33.205826 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:33.205715 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdhqv" podUID="32c97ab8-07c6-4f64-9065-2441c8604a5a" Sep 17 01:22:33.205826 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:33.205762 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkbfd" podUID="8a366399-229b-40ae-b916-4323338cb805" Sep 17 01:22:34.205762 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:34.205580 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:34.206206 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:34.205834 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwr7v" podUID="3d7e510c-fe28-46fc-b4b8-bef103adca93" Sep 17 01:22:35.206448 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:35.206414 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:35.206855 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:35.206479 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:35.206855 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:35.206575 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdhqv" podUID="32c97ab8-07c6-4f64-9065-2441c8604a5a" Sep 17 01:22:35.206855 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:35.206691 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkbfd" podUID="8a366399-229b-40ae-b916-4323338cb805" Sep 17 01:22:36.205794 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:36.205764 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:36.205969 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:36.205855 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwr7v" podUID="3d7e510c-fe28-46fc-b4b8-bef103adca93" Sep 17 01:22:37.206033 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.205772 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:37.206799 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.205772 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:37.206799 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:37.206154 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdhqv" podUID="32c97ab8-07c6-4f64-9065-2441c8604a5a" Sep 17 01:22:37.206799 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:37.206234 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkbfd" podUID="8a366399-229b-40ae-b916-4323338cb805" Sep 17 01:22:37.375386 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.375353 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p9475" event={"ID":"a32cfdee-7fb5-4b6f-bd3a-ce1441f806d4","Type":"ContainerStarted","Data":"51a867e481c0cfde30288519e0e20e30ef09a4244bf608c323e1ddc8d30762e5"} Sep 17 01:22:37.376768 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.376737 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" event={"ID":"ef559f28-d4fd-4c60-b17a-626959a1bed9","Type":"ContainerStarted","Data":"a8879d20e972bf05608ae138c0281dde8d8c65f16310c1e57435a8dae37370de"} Sep 17 01:22:37.378051 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.378031 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" event={"ID":"087a1d49-8d5d-4925-8d56-9c5e3383f0a9","Type":"ContainerStarted","Data":"aaf6d2dd3dab85f2e0c1dca2ddfd5d944d9def9ef55d2294cc1f18fca2c56dc7"} Sep 17 01:22:37.379368 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.379346 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-km877" event={"ID":"7b6b4978-53de-4fa4-83cc-21d3de40ea24","Type":"ContainerStarted","Data":"0fbd854d4743dc48ce61802851ba99f1bc43074e8139fa0dbc69bbb284b1bc32"} Sep 17 01:22:37.382175 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.382156 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" event={"ID":"18244e83-55ca-4e07-8d44-b5d6d06b32a4","Type":"ContainerStarted","Data":"85a8ea3ffb919e1c8378e3040de4a1b566c7f54cdd6159da1fbc492c65de7c4c"} Sep 17 01:22:37.382267 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.382215 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" event={"ID":"18244e83-55ca-4e07-8d44-b5d6d06b32a4","Type":"ContainerStarted","Data":"106c6de6ec3788f269f5a8e144b5a4f4f292bc987e1a8849a77219f228fe6351"} Sep 17 01:22:37.382267 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.382246 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" event={"ID":"18244e83-55ca-4e07-8d44-b5d6d06b32a4","Type":"ContainerStarted","Data":"5ab264920a9078020e332b9b378970a6417f7c645c0fd0633688e2ad7d54fd13"} Sep 17 01:22:37.382349 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.382280 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" event={"ID":"18244e83-55ca-4e07-8d44-b5d6d06b32a4","Type":"ContainerStarted","Data":"fd90970f09fd6204cb5d0bd6dfb28702902194baeb17402ad1cdadc6b51562e8"} Sep 17 01:22:37.382349 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.382315 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" event={"ID":"18244e83-55ca-4e07-8d44-b5d6d06b32a4","Type":"ContainerStarted","Data":"9b1424f5b706fa0271ad86c9deb73fcd17ea7ee9daf4e6fd2881d836f85f8464"} Sep 17 01:22:37.382349 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.382342 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" event={"ID":"18244e83-55ca-4e07-8d44-b5d6d06b32a4","Type":"ContainerStarted","Data":"f6c298a325529c7ec574c6059fed95e3af08fcf6f0cb1097388d592edc989552"} Sep 17 01:22:37.383604 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.383583 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cbqj4" event={"ID":"282d394d-c1cd-4368-bbb5-f8a67ab9c07b","Type":"ContainerStarted","Data":"e83685194747935fd513445f75e3154ae5bdf709334fc8314b280d95eb386812"} Sep 17 01:22:37.385078 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.385056 2577 generic.go:356] "Generic (PLEG): container finished" podID="766b09df-bf87-4ace-8430-628323cd59c9" containerID="2495a4fba791841c87354c85f8808eafc71a46dfd6702231fc2f5d4f40fb1e24" exitCode=0 Sep 17 01:22:37.385173 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.385116 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9ptd" event={"ID":"766b09df-bf87-4ace-8430-628323cd59c9","Type":"ContainerDied","Data":"2495a4fba791841c87354c85f8808eafc71a46dfd6702231fc2f5d4f40fb1e24"} Sep 17 01:22:37.386523 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.386483 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ttq94" event={"ID":"78186fc7-1ae0-4bf9-89ec-38c47218f909","Type":"ContainerStarted","Data":"9b84ef9213bbe6bca4e0cb53835f59c4cf67c7b224d6accf8076e1435c97b85c"} Sep 17 01:22:37.388355 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.388319 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-p9475" podStartSLOduration=8.198015224 podStartE2EDuration="22.38830606s" totalImagesPullingTime="14.190290836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:37.388047081 +0000 UTC m=+22.623194266" watchObservedRunningTime="2026-09-17 01:22:37.38830606 +0000 UTC m=+22.623453321" Sep 17 01:22:37.416320 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.416271 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ttq94" podStartSLOduration=4.3559164169999995 podStartE2EDuration="22.416256111s" totalImagesPullingTime="18.060339694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:37.416071144 +0000 UTC m=+22.651218332" watchObservedRunningTime="2026-09-17 01:22:37.416256111 +0000 UTC m=+22.651403298" Sep 17 01:22:37.428117 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.428069 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-km877" podStartSLOduration=4.354314425 podStartE2EDuration="22.428054689s" totalImagesPullingTime="18.073740264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:37.427988439 +0000 UTC m=+22.663135627" watchObservedRunningTime="2026-09-17 01:22:37.428054689 +0000 UTC m=+22.663201871" Sep 17 01:22:37.441754 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.441713 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9qjz8" podStartSLOduration=4.372441747 podStartE2EDuration="22.44170261s" totalImagesPullingTime="18.069260863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:37.441567709 +0000 UTC m=+22.676714897" watchObservedRunningTime="2026-09-17 01:22:37.44170261 +0000 UTC m=+22.676849798" Sep 17 01:22:37.460358 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.460315 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cbqj4" podStartSLOduration=4.337484267 podStartE2EDuration="22.460293997s" totalImagesPullingTime="18.12280973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:37.459777005 +0000 UTC m=+22.694924195" watchObservedRunningTime="2026-09-17 01:22:37.460293997 +0000 UTC m=+22.695441187" Sep 17 01:22:37.998686 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:37.998658 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 17 01:22:38.150206 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:38.150109 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T01:22:37.998677236Z","UUID":"c6007c22-f202-4282-9715-cbb87544df90","Handler":null,"Name":"","Endpoint":""} Sep 17 01:22:38.151765 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:38.151744 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 17 01:22:38.151882 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:38.151775 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 17 01:22:38.205274 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:38.205240 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:38.205419 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:38.205372 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwr7v" podUID="3d7e510c-fe28-46fc-b4b8-bef103adca93" Sep 17 01:22:38.390254 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:38.390215 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" event={"ID":"087a1d49-8d5d-4925-8d56-9c5e3383f0a9","Type":"ContainerStarted","Data":"95f8c18cb0056d4487dd706586911c814e3a2e678f13fd0696edcd174aac80b6"} Sep 17 01:22:38.391678 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:38.391647 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tkg5m" event={"ID":"4f41bd42-f471-4885-a023-d2266738fdc3","Type":"ContainerStarted","Data":"35c2ce0874540a13a2f0d901a42e2e988571b67cb8b1255fc3035fabb6729213"} Sep 17 01:22:38.406459 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:38.406418 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tkg5m" podStartSLOduration=5.321944962 podStartE2EDuration="23.406405428s" totalImagesPullingTime="18.084460466s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:38.406014396 +0000 UTC m=+23.641161584" watchObservedRunningTime="2026-09-17 01:22:38.406405428 +0000 UTC m=+23.641552616" Sep 17 01:22:39.206278 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:39.206197 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:39.206438 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:39.206364 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkbfd" podUID="8a366399-229b-40ae-b916-4323338cb805" Sep 17 01:22:39.206438 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:39.206400 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:39.206559 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:39.206483 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdhqv" podUID="32c97ab8-07c6-4f64-9065-2441c8604a5a" Sep 17 01:22:39.398568 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:39.398532 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" event={"ID":"18244e83-55ca-4e07-8d44-b5d6d06b32a4","Type":"ContainerStarted","Data":"487d38a9186605dffc570ef8637105ab11624527778387a92f9fa82e7c8fa74e"} Sep 17 01:22:39.400988 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:39.400958 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" event={"ID":"087a1d49-8d5d-4925-8d56-9c5e3383f0a9","Type":"ContainerStarted","Data":"9eb1873e196dfd5b29d25fa66a0790e798298805303c624a5b24ae1ae8770afb"} Sep 17 01:22:39.416618 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:39.416579 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-srvsk" podStartSLOduration=3.92029996 podStartE2EDuration="24.416564929s" totalImagesPullingTime="20.496264969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:39.416375388 +0000 UTC m=+24.651522575" watchObservedRunningTime="2026-09-17 01:22:39.416564929 +0000 UTC m=+24.651712166" Sep 17 01:22:40.205855 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:40.205823 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:40.206102 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:40.205977 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwr7v" podUID="3d7e510c-fe28-46fc-b4b8-bef103adca93" Sep 17 01:22:40.528904 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:40.528818 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-p9475" Sep 17 01:22:40.529394 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:40.529378 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-p9475" Sep 17 01:22:41.206183 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:41.206144 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:41.206376 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:41.206144 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:41.206376 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:41.206280 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkbfd" podUID="8a366399-229b-40ae-b916-4323338cb805" Sep 17 01:22:41.206376 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:41.206337 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdhqv" podUID="32c97ab8-07c6-4f64-9065-2441c8604a5a" Sep 17 01:22:41.818990 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:41.818950 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-p9475" Sep 17 01:22:41.819850 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:41.819830 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-p9475" Sep 17 01:22:42.206032 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:42.206003 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:42.206183 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:42.206110 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwr7v" podUID="3d7e510c-fe28-46fc-b4b8-bef103adca93" Sep 17 01:22:43.205664 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:43.205465 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:43.206071 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:43.205520 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:43.206071 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:43.205745 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkbfd" podUID="8a366399-229b-40ae-b916-4323338cb805" Sep 17 01:22:43.206071 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:43.205814 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdhqv" podUID="32c97ab8-07c6-4f64-9065-2441c8604a5a" Sep 17 01:22:43.411431 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:43.411401 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" event={"ID":"18244e83-55ca-4e07-8d44-b5d6d06b32a4","Type":"ContainerStarted","Data":"ef833e40d518a3235a8d7f30a752a28e7cb1e1cebd0779d25497dc500041c411"} Sep 17 01:22:43.411743 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:43.411630 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:43.413402 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:43.413379 2577 generic.go:356] "Generic (PLEG): container finished" podID="766b09df-bf87-4ace-8430-628323cd59c9" containerID="f36085dc4fd8990722ca6021b632352a76431f0a2c96aff4a8e457e9887c938b" exitCode=0 Sep 17 01:22:43.413513 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:43.413414 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9ptd" event={"ID":"766b09df-bf87-4ace-8430-628323cd59c9","Type":"ContainerDied","Data":"f36085dc4fd8990722ca6021b632352a76431f0a2c96aff4a8e457e9887c938b"} Sep 17 01:22:43.427454 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:43.427430 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:43.435982 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:43.435925 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" podStartSLOduration=10.321873412 podStartE2EDuration="28.435911679s" totalImagesPullingTime="18.114038267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:43.434471138 +0000 UTC m=+28.669618326" watchObservedRunningTime="2026-09-17 01:22:43.435911679 +0000 UTC m=+28.671058867" Sep 17 01:22:44.205289 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:44.205260 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:44.205466 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:44.205388 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwr7v" podUID="3d7e510c-fe28-46fc-b4b8-bef103adca93" Sep 17 01:22:44.380107 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:44.380071 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lkbfd"] Sep 17 01:22:44.380900 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:44.380200 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:44.380900 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:44.380300 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkbfd" podUID="8a366399-229b-40ae-b916-4323338cb805" Sep 17 01:22:44.383037 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:44.382995 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lwr7v"] Sep 17 01:22:44.383595 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:44.383573 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rdhqv"] Sep 17 01:22:44.383702 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:44.383676 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:44.383779 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:44.383763 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdhqv" podUID="32c97ab8-07c6-4f64-9065-2441c8604a5a" Sep 17 01:22:44.415804 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:44.415774 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:44.415981 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:44.415899 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwr7v" podUID="3d7e510c-fe28-46fc-b4b8-bef103adca93" Sep 17 01:22:44.416071 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:44.415980 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:44.416128 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:44.416088 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:44.431133 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:44.431106 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:22:45.419843 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:45.419806 2577 generic.go:356] "Generic (PLEG): container finished" podID="766b09df-bf87-4ace-8430-628323cd59c9" containerID="d1d9e3921845590dd7d4ede655e6663d39092ded8d729c68d8e79a79cc42b2b9" exitCode=0 Sep 17 01:22:45.420286 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:45.419890 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9ptd" event={"ID":"766b09df-bf87-4ace-8430-628323cd59c9","Type":"ContainerDied","Data":"d1d9e3921845590dd7d4ede655e6663d39092ded8d729c68d8e79a79cc42b2b9"} Sep 17 01:22:46.205566 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:46.205534 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:46.205566 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:46.205557 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:46.205780 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:46.205634 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:46.205780 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:46.205652 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkbfd" podUID="8a366399-229b-40ae-b916-4323338cb805" Sep 17 01:22:46.205780 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:46.205702 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwr7v" podUID="3d7e510c-fe28-46fc-b4b8-bef103adca93" Sep 17 01:22:46.205885 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:46.205811 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdhqv" podUID="32c97ab8-07c6-4f64-9065-2441c8604a5a" Sep 17 01:22:47.426902 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:47.426862 2577 generic.go:356] "Generic (PLEG): container finished" podID="766b09df-bf87-4ace-8430-628323cd59c9" containerID="2d8028ed19c18d9c5af6e9cd1bcd4248ed90faaeecdd2e6ffddcef2052ac08e7" exitCode=0 Sep 17 01:22:47.427265 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:47.426909 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9ptd" event={"ID":"766b09df-bf87-4ace-8430-628323cd59c9","Type":"ContainerDied","Data":"2d8028ed19c18d9c5af6e9cd1bcd4248ed90faaeecdd2e6ffddcef2052ac08e7"} Sep 17 01:22:48.205432 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:48.205399 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:48.205432 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:48.205432 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:48.206395 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:48.205401 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:48.206395 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:48.205516 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lwr7v" podUID="3d7e510c-fe28-46fc-b4b8-bef103adca93" Sep 17 01:22:48.206395 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:48.205623 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkbfd" podUID="8a366399-229b-40ae-b916-4323338cb805" Sep 17 01:22:48.206395 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:48.205730 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdhqv" podUID="32c97ab8-07c6-4f64-9065-2441c8604a5a" Sep 17 01:22:48.841322 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:48.841284 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret\") pod \"global-pull-secret-syncer-lkbfd\" (UID: \"8a366399-229b-40ae-b916-4323338cb805\") " pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:48.841847 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:48.841336 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jks58\" (UniqueName: \"kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58\") pod \"network-check-target-lwr7v\" (UID: \"3d7e510c-fe28-46fc-b4b8-bef103adca93\") " pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:48.841847 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:48.841484 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 01:22:48.841847 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:48.841578 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret podName:8a366399-229b-40ae-b916-4323338cb805 nodeName:}" failed. No retries permitted until 2026-09-17 01:23:20.841556976 +0000 UTC m=+66.076704159 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret") pod "global-pull-secret-syncer-lkbfd" (UID: "8a366399-229b-40ae-b916-4323338cb805") : object "kube-system"/"original-pull-secret" not registered Sep 17 01:22:48.841847 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:48.841494 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 01:22:48.841847 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:48.841615 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 01:22:48.841847 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:48.841629 2577 projected.go:196] Error preparing data for projected volume kube-api-access-jks58 for pod openshift-network-diagnostics/network-check-target-lwr7v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:22:48.841847 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:48.841685 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58 podName:3d7e510c-fe28-46fc-b4b8-bef103adca93 nodeName:}" failed. No retries permitted until 2026-09-17 01:23:20.841670492 +0000 UTC m=+66.076817662 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jks58" (UniqueName: "kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58") pod "network-check-target-lwr7v" (UID: "3d7e510c-fe28-46fc-b4b8-bef103adca93") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 01:22:48.942357 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:48.942319 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs\") pod \"network-metrics-daemon-rdhqv\" (UID: \"32c97ab8-07c6-4f64-9065-2441c8604a5a\") " pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:48.942516 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:48.942461 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:22:48.942572 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:48.942533 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs podName:32c97ab8-07c6-4f64-9065-2441c8604a5a nodeName:}" failed. No retries permitted until 2026-09-17 01:23:20.942514798 +0000 UTC m=+66.177661978 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs") pod "network-metrics-daemon-rdhqv" (UID: "32c97ab8-07c6-4f64-9065-2441c8604a5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 01:22:49.104593 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.104525 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-83.ec2.internal" event="NodeReady" Sep 17 01:22:49.104738 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.104684 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 17 01:22:49.144832 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.144798 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-m4hzm"] Sep 17 01:22:49.162809 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.162780 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6p5q7"] Sep 17 01:22:49.162961 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.162903 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m4hzm" Sep 17 01:22:49.165119 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.165100 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 17 01:22:49.165229 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.165105 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sdvq5\"" Sep 17 01:22:49.165374 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.165358 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 17 01:22:49.184860 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.184836 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m4hzm"] Sep 17 01:22:49.185015 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.184876 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6p5q7"] Sep 17 01:22:49.185015 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.184981 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6p5q7" Sep 17 01:22:49.187578 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.187523 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 17 01:22:49.187695 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.187672 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mtkb2\"" Sep 17 01:22:49.187762 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.187529 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 17 01:22:49.187844 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.187529 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 17 01:22:49.244545 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.244510 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/942e7bc8-54bc-4e0f-939b-814164a4ed3d-config-volume\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:22:49.244691 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.244567 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/942e7bc8-54bc-4e0f-939b-814164a4ed3d-tmp-dir\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:22:49.244691 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.244677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:22:49.244759 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.244729 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmpcv\" (UniqueName: \"kubernetes.io/projected/942e7bc8-54bc-4e0f-939b-814164a4ed3d-kube-api-access-pmpcv\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:22:49.345849 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.345817 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/942e7bc8-54bc-4e0f-939b-814164a4ed3d-tmp-dir\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:22:49.345849 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.345855 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a717347-6484-44b8-bdb8-19ca455b2905-cert\") pod \"ingress-canary-6p5q7\" (UID: \"2a717347-6484-44b8-bdb8-19ca455b2905\") " pod="openshift-ingress-canary/ingress-canary-6p5q7" Sep 17 01:22:49.346073 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.345874 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9889x\" (UniqueName: \"kubernetes.io/projected/2a717347-6484-44b8-bdb8-19ca455b2905-kube-api-access-9889x\") pod \"ingress-canary-6p5q7\" (UID: \"2a717347-6484-44b8-bdb8-19ca455b2905\") " pod="openshift-ingress-canary/ingress-canary-6p5q7" Sep 17 01:22:49.346073 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.345920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:22:49.346073 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.345981 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmpcv\" (UniqueName: \"kubernetes.io/projected/942e7bc8-54bc-4e0f-939b-814164a4ed3d-kube-api-access-pmpcv\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:22:49.346073 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.346008 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/942e7bc8-54bc-4e0f-939b-814164a4ed3d-config-volume\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:22:49.346218 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:49.346075 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:22:49.346218 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:49.346137 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls podName:942e7bc8-54bc-4e0f-939b-814164a4ed3d nodeName:}" failed. No retries permitted until 2026-09-17 01:22:49.846118171 +0000 UTC m=+35.081265341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls") pod "dns-default-m4hzm" (UID: "942e7bc8-54bc-4e0f-939b-814164a4ed3d") : secret "dns-default-metrics-tls" not found Sep 17 01:22:49.346218 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.346168 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/942e7bc8-54bc-4e0f-939b-814164a4ed3d-tmp-dir\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:22:49.346533 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.346517 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/942e7bc8-54bc-4e0f-939b-814164a4ed3d-config-volume\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:22:49.355140 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.355087 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmpcv\" (UniqueName: \"kubernetes.io/projected/942e7bc8-54bc-4e0f-939b-814164a4ed3d-kube-api-access-pmpcv\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:22:49.446652 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.446624 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a717347-6484-44b8-bdb8-19ca455b2905-cert\") pod \"ingress-canary-6p5q7\" (UID: \"2a717347-6484-44b8-bdb8-19ca455b2905\") " pod="openshift-ingress-canary/ingress-canary-6p5q7" Sep 17 01:22:49.446652 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.446655 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9889x\" (UniqueName: \"kubernetes.io/projected/2a717347-6484-44b8-bdb8-19ca455b2905-kube-api-access-9889x\") pod \"ingress-canary-6p5q7\" (UID: \"2a717347-6484-44b8-bdb8-19ca455b2905\") " pod="openshift-ingress-canary/ingress-canary-6p5q7" Sep 17 01:22:49.449230 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.449205 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a717347-6484-44b8-bdb8-19ca455b2905-cert\") pod \"ingress-canary-6p5q7\" (UID: \"2a717347-6484-44b8-bdb8-19ca455b2905\") " pod="openshift-ingress-canary/ingress-canary-6p5q7" Sep 17 01:22:49.453647 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.453628 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9889x\" (UniqueName: \"kubernetes.io/projected/2a717347-6484-44b8-bdb8-19ca455b2905-kube-api-access-9889x\") pod \"ingress-canary-6p5q7\" (UID: \"2a717347-6484-44b8-bdb8-19ca455b2905\") " pod="openshift-ingress-canary/ingress-canary-6p5q7" Sep 17 01:22:49.496634 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.496605 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6p5q7" Sep 17 01:22:49.799991 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.799684 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6p5q7"] Sep 17 01:22:49.804036 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:22:49.804000 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a717347_6484_44b8_bdb8_19ca455b2905.slice/crio-ad07186c95f78e89b013190313067abccd0e20472832542bf373fb0ab45b7205 WatchSource:0}: Error finding container ad07186c95f78e89b013190313067abccd0e20472832542bf373fb0ab45b7205: Status 404 returned error can't find the container with id ad07186c95f78e89b013190313067abccd0e20472832542bf373fb0ab45b7205 Sep 17 01:22:49.849350 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:49.849323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:22:49.849694 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:49.849420 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:22:49.849694 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:49.849466 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls podName:942e7bc8-54bc-4e0f-939b-814164a4ed3d nodeName:}" failed. No retries permitted until 2026-09-17 01:22:50.849452962 +0000 UTC m=+36.084600127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls") pod "dns-default-m4hzm" (UID: "942e7bc8-54bc-4e0f-939b-814164a4ed3d") : secret "dns-default-metrics-tls" not found Sep 17 01:22:50.205881 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:50.205842 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:22:50.206169 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:50.205842 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:22:50.206169 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:50.205842 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:22:50.209600 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:50.209400 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k6hcd\"" Sep 17 01:22:50.209600 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:50.209420 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 01:22:50.209600 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:50.209439 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 01:22:50.209600 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:50.209454 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 01:22:50.209600 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:50.209400 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 01:22:50.209600 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:50.209400 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wvztc\"" Sep 17 01:22:50.436383 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:50.436348 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6p5q7" event={"ID":"2a717347-6484-44b8-bdb8-19ca455b2905","Type":"ContainerStarted","Data":"ad07186c95f78e89b013190313067abccd0e20472832542bf373fb0ab45b7205"} Sep 17 01:22:50.857602 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:50.857534 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:22:50.858103 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:50.857665 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:22:50.858103 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:50.857744 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls podName:942e7bc8-54bc-4e0f-939b-814164a4ed3d nodeName:}" failed. No retries permitted until 2026-09-17 01:22:52.857724349 +0000 UTC m=+38.092871516 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls") pod "dns-default-m4hzm" (UID: "942e7bc8-54bc-4e0f-939b-814164a4ed3d") : secret "dns-default-metrics-tls" not found Sep 17 01:22:52.874419 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:52.874377 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:22:52.875042 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:52.874519 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:22:52.875042 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:52.874580 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls podName:942e7bc8-54bc-4e0f-939b-814164a4ed3d nodeName:}" failed. No retries permitted until 2026-09-17 01:22:56.874566962 +0000 UTC m=+42.109714128 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls") pod "dns-default-m4hzm" (UID: "942e7bc8-54bc-4e0f-939b-814164a4ed3d") : secret "dns-default-metrics-tls" not found Sep 17 01:22:54.448844 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:54.448812 2577 generic.go:356] "Generic (PLEG): container finished" podID="766b09df-bf87-4ace-8430-628323cd59c9" containerID="86f6a5f3b4061fcc88a4fe224144589e5a39a50f9765b60b898d3aedd7b019e7" exitCode=0 Sep 17 01:22:54.449509 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:54.448896 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9ptd" event={"ID":"766b09df-bf87-4ace-8430-628323cd59c9","Type":"ContainerDied","Data":"86f6a5f3b4061fcc88a4fe224144589e5a39a50f9765b60b898d3aedd7b019e7"} Sep 17 01:22:54.450440 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:54.450419 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6p5q7" event={"ID":"2a717347-6484-44b8-bdb8-19ca455b2905","Type":"ContainerStarted","Data":"921393555cac31b5ab8b2c5d9b7a60cd3d8a2591fb4e28291955f46bf9717073"} Sep 17 01:22:54.478968 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:54.478899 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6p5q7" podStartSLOduration=1.709798047 podStartE2EDuration="5.47887708s" totalImagesPullingTime="3.769079033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:54.478671733 +0000 UTC m=+39.713818920" watchObservedRunningTime="2026-09-17 01:22:54.47887708 +0000 UTC m=+39.714024269" Sep 17 01:22:55.458773 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:55.458734 2577 generic.go:356] "Generic (PLEG): container finished" podID="766b09df-bf87-4ace-8430-628323cd59c9" containerID="44b59e0fef376321e51c411387b03e205a9f49d1ead9c65c6ad8b81a71ce33a3" exitCode=0 Sep 17 01:22:55.459212 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:55.458810 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9ptd" event={"ID":"766b09df-bf87-4ace-8430-628323cd59c9","Type":"ContainerDied","Data":"44b59e0fef376321e51c411387b03e205a9f49d1ead9c65c6ad8b81a71ce33a3"} Sep 17 01:22:56.464031 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:56.463992 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m9ptd" event={"ID":"766b09df-bf87-4ace-8430-628323cd59c9","Type":"ContainerStarted","Data":"00530fedfd6d4b79ca9d06729ad7993fabe27ff8181d1835415a28b903f166ef"} Sep 17 01:22:56.484413 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:56.484369 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m9ptd" podStartSLOduration=9.584477112 podStartE2EDuration="41.484353146s" totalImagesPullingTime="31.899876034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:22:56.48303509 +0000 UTC m=+41.718182278" watchObservedRunningTime="2026-09-17 01:22:56.484353146 +0000 UTC m=+41.719500334" Sep 17 01:22:56.902284 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:22:56.902246 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:22:56.902450 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:56.902389 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:22:56.902450 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:22:56.902447 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls podName:942e7bc8-54bc-4e0f-939b-814164a4ed3d nodeName:}" failed. No retries permitted until 2026-09-17 01:23:04.90243396 +0000 UTC m=+50.137581131 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls") pod "dns-default-m4hzm" (UID: "942e7bc8-54bc-4e0f-939b-814164a4ed3d") : secret "dns-default-metrics-tls" not found Sep 17 01:23:04.959780 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:04.959724 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:23:04.960215 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:23:04.959892 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:23:04.960215 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:23:04.959978 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls podName:942e7bc8-54bc-4e0f-939b-814164a4ed3d nodeName:}" failed. No retries permitted until 2026-09-17 01:23:20.95996011 +0000 UTC m=+66.195107294 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls") pod "dns-default-m4hzm" (UID: "942e7bc8-54bc-4e0f-939b-814164a4ed3d") : secret "dns-default-metrics-tls" not found Sep 17 01:23:16.433841 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:16.433813 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ttfl4" Sep 17 01:23:20.866371 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:20.866329 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret\") pod \"global-pull-secret-syncer-lkbfd\" (UID: \"8a366399-229b-40ae-b916-4323338cb805\") " pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:23:20.866371 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:20.866375 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jks58\" (UniqueName: \"kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58\") pod \"network-check-target-lwr7v\" (UID: \"3d7e510c-fe28-46fc-b4b8-bef103adca93\") " pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:23:20.868745 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:20.868720 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 01:23:20.868864 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:20.868727 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 01:23:20.878315 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:20.878296 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 01:23:20.879427 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:20.879410 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a366399-229b-40ae-b916-4323338cb805-original-pull-secret\") pod \"global-pull-secret-syncer-lkbfd\" (UID: \"8a366399-229b-40ae-b916-4323338cb805\") " pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:23:20.888916 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:20.888893 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jks58\" (UniqueName: \"kubernetes.io/projected/3d7e510c-fe28-46fc-b4b8-bef103adca93-kube-api-access-jks58\") pod \"network-check-target-lwr7v\" (UID: \"3d7e510c-fe28-46fc-b4b8-bef103adca93\") " pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:23:20.967115 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:20.967064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs\") pod \"network-metrics-daemon-rdhqv\" (UID: \"32c97ab8-07c6-4f64-9065-2441c8604a5a\") " pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:23:20.967115 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:20.967120 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:23:20.967290 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:23:20.967216 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:23:20.967290 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:23:20.967268 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls podName:942e7bc8-54bc-4e0f-939b-814164a4ed3d nodeName:}" failed. No retries permitted until 2026-09-17 01:23:52.967255348 +0000 UTC m=+98.202402514 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls") pod "dns-default-m4hzm" (UID: "942e7bc8-54bc-4e0f-939b-814164a4ed3d") : secret "dns-default-metrics-tls" not found Sep 17 01:23:20.969821 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:20.969800 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 01:23:20.977580 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:23:20.977562 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 01:23:20.977627 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:23:20.977611 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs podName:32c97ab8-07c6-4f64-9065-2441c8604a5a nodeName:}" failed. No retries permitted until 2026-09-17 01:24:24.97759828 +0000 UTC m=+130.212745446 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs") pod "network-metrics-daemon-rdhqv" (UID: "32c97ab8-07c6-4f64-9065-2441c8604a5a") : secret "metrics-daemon-secret" not found Sep 17 01:23:21.120293 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:21.120223 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wvztc\"" Sep 17 01:23:21.129093 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:21.129070 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:23:21.130315 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:21.130300 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkbfd" Sep 17 01:23:21.403524 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:21.403443 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lkbfd"] Sep 17 01:23:21.406804 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:21.406778 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lwr7v"] Sep 17 01:23:21.408654 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:23:21.408625 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a366399_229b_40ae_b916_4323338cb805.slice/crio-a4efb4be4f792d412afeb98115b0ce62106dcca9c6a4c5cceb619500490c4542 WatchSource:0}: Error finding container a4efb4be4f792d412afeb98115b0ce62106dcca9c6a4c5cceb619500490c4542: Status 404 returned error can't find the container with id a4efb4be4f792d412afeb98115b0ce62106dcca9c6a4c5cceb619500490c4542 Sep 17 01:23:21.410075 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:23:21.410056 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7e510c_fe28_46fc_b4b8_bef103adca93.slice/crio-f121ff56fe66b154dd780abb385cfe267a46d83184769cc4faefba076513604c WatchSource:0}: Error finding container f121ff56fe66b154dd780abb385cfe267a46d83184769cc4faefba076513604c: Status 404 returned error can't find the container with id f121ff56fe66b154dd780abb385cfe267a46d83184769cc4faefba076513604c Sep 17 01:23:21.522114 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:21.522076 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lkbfd" event={"ID":"8a366399-229b-40ae-b916-4323338cb805","Type":"ContainerStarted","Data":"a4efb4be4f792d412afeb98115b0ce62106dcca9c6a4c5cceb619500490c4542"} Sep 17 01:23:21.523174 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:21.523153 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lwr7v" event={"ID":"3d7e510c-fe28-46fc-b4b8-bef103adca93","Type":"ContainerStarted","Data":"f121ff56fe66b154dd780abb385cfe267a46d83184769cc4faefba076513604c"} Sep 17 01:23:26.539843 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:26.539805 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lkbfd" event={"ID":"8a366399-229b-40ae-b916-4323338cb805","Type":"ContainerStarted","Data":"96f3a57b0c02e3d6f132f32f96c631bb577df883a580e433e8801aea393c1bcb"} Sep 17 01:23:26.541409 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:26.541386 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lwr7v" event={"ID":"3d7e510c-fe28-46fc-b4b8-bef103adca93","Type":"ContainerStarted","Data":"dd14c71e3450de8780840aba5419dda30aa28e312554f94a063ac111d9927a0e"} Sep 17 01:23:26.541527 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:26.541517 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:23:26.552881 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:26.552837 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lkbfd" podStartSLOduration=66.651156288 podStartE2EDuration="1m11.552823006s" totalImagesPullingTime="4.901666718s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:23:26.552282442 +0000 UTC m=+71.787429630" watchObservedRunningTime="2026-09-17 01:23:26.552823006 +0000 UTC m=+71.787970193" Sep 17 01:23:26.564625 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:26.564550 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lwr7v" podStartSLOduration=66.664673397 podStartE2EDuration="1m11.564535737s" totalImagesPullingTime="4.89986234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:23:26.564205149 +0000 UTC m=+71.799352336" watchObservedRunningTime="2026-09-17 01:23:26.564535737 +0000 UTC m=+71.799682925" Sep 17 01:23:52.981230 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:52.981180 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:23:52.981719 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:23:52.981337 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:23:52.981719 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:23:52.981415 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls podName:942e7bc8-54bc-4e0f-939b-814164a4ed3d nodeName:}" failed. No retries permitted until 2026-09-17 01:24:56.981399236 +0000 UTC m=+162.216546402 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls") pod "dns-default-m4hzm" (UID: "942e7bc8-54bc-4e0f-939b-814164a4ed3d") : secret "dns-default-metrics-tls" not found Sep 17 01:23:57.545721 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:23:57.545692 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lwr7v" Sep 17 01:24:24.994232 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:24:24.994184 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs\") pod \"network-metrics-daemon-rdhqv\" (UID: \"32c97ab8-07c6-4f64-9065-2441c8604a5a\") " pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:24:24.994720 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:24:24.994309 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 01:24:24.994720 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:24:24.994375 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs podName:32c97ab8-07c6-4f64-9065-2441c8604a5a nodeName:}" failed. No retries permitted until 2026-09-17 01:26:26.994361307 +0000 UTC m=+252.229508473 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs") pod "network-metrics-daemon-rdhqv" (UID: "32c97ab8-07c6-4f64-9065-2441c8604a5a") : secret "metrics-daemon-secret" not found Sep 17 01:24:52.172488 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:24:52.172439 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-m4hzm" podUID="942e7bc8-54bc-4e0f-939b-814164a4ed3d" Sep 17 01:24:52.746374 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:24:52.746344 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m4hzm" Sep 17 01:24:53.225173 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:24:53.225142 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-rdhqv" podUID="32c97ab8-07c6-4f64-9065-2441c8604a5a" Sep 17 01:24:57.003754 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:24:57.003706 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:24:57.004176 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:24:57.003886 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 01:24:57.004176 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:24:57.003982 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls podName:942e7bc8-54bc-4e0f-939b-814164a4ed3d nodeName:}" failed. No retries permitted until 2026-09-17 01:26:59.003964607 +0000 UTC m=+284.239111772 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls") pod "dns-default-m4hzm" (UID: "942e7bc8-54bc-4e0f-939b-814164a4ed3d") : secret "dns-default-metrics-tls" not found Sep 17 01:25:04.205499 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:25:04.205461 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:26:04.769510 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.769479 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-b87fj"] Sep 17 01:26:04.772190 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.772165 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-b87fj" Sep 17 01:26:04.773016 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.772991 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-585fc78676-kqsfm"] Sep 17 01:26:04.774485 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.774466 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wf9tj\"" Sep 17 01:26:04.774874 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.774858 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 17 01:26:04.774963 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.774907 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 17 01:26:04.775582 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.775566 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:04.777763 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.777714 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-q4nln\"" Sep 17 01:26:04.777884 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.777864 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 17 01:26:04.778946 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.778357 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 17 01:26:04.778946 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.778403 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 17 01:26:04.778946 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.778604 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 17 01:26:04.778946 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.778880 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 17 01:26:04.779223 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.779090 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 17 01:26:04.798083 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.798056 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-b87fj"] Sep 17 01:26:04.798992 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.798973 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-585fc78676-kqsfm"] Sep 17 01:26:04.848288 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.848259 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvwzd\" (UniqueName: \"kubernetes.io/projected/535cb079-9f6c-4cab-ace5-1664b0dd0523-kube-api-access-mvwzd\") pod \"volume-data-source-validator-86695b7cc7-b87fj\" (UID: \"535cb079-9f6c-4cab-ace5-1664b0dd0523\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-b87fj" Sep 17 01:26:04.848429 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.848293 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-metrics-certs\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:04.848429 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.848316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-default-certificate\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:04.848429 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.848351 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d1b91db-e40c-48f8-8008-f01fd5801ec0-service-ca-bundle\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:04.848429 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.848425 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-stats-auth\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:04.848568 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.848443 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lss9s\" (UniqueName: \"kubernetes.io/projected/4d1b91db-e40c-48f8-8008-f01fd5801ec0-kube-api-access-lss9s\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:04.869199 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.869172 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gtwjd"] Sep 17 01:26:04.871993 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.871970 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-9zzb8"] Sep 17 01:26:04.872114 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.872095 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gtwjd" Sep 17 01:26:04.874407 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.874387 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-thk7k\"" Sep 17 01:26:04.874407 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.874397 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 17 01:26:04.874550 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.874456 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 17 01:26:04.874651 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.874633 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-ptlrg"] Sep 17 01:26:04.874718 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.874650 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 17 01:26:04.874771 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.874745 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" Sep 17 01:26:04.874825 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.874768 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 17 01:26:04.876926 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.876909 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 17 01:26:04.877033 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.877013 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 17 01:26:04.877093 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.877082 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-spb8g\"" Sep 17 01:26:04.877148 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.877102 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 17 01:26:04.877148 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.877128 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 17 01:26:04.877332 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.877311 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ptlrg" Sep 17 01:26:04.879081 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.879058 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mpbds\"" Sep 17 01:26:04.881058 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.881037 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 17 01:26:04.882431 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.882412 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gtwjd"] Sep 17 01:26:04.883457 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.883436 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-ptlrg"] Sep 17 01:26:04.895134 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.895112 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-9zzb8"] Sep 17 01:26:04.949639 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.949604 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4kc8\" (UniqueName: \"kubernetes.io/projected/24dd28d0-d328-41a2-83bc-d392d123272b-kube-api-access-x4kc8\") pod \"kube-storage-version-migrator-operator-6c8f84f897-gtwjd\" (UID: \"24dd28d0-d328-41a2-83bc-d392d123272b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gtwjd" Sep 17 01:26:04.949824 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.949644 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvwzd\" (UniqueName: \"kubernetes.io/projected/535cb079-9f6c-4cab-ace5-1664b0dd0523-kube-api-access-mvwzd\") pod \"volume-data-source-validator-86695b7cc7-b87fj\" (UID: \"535cb079-9f6c-4cab-ace5-1664b0dd0523\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-b87fj" Sep 17 01:26:04.949824 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.949671 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdcp4\" (UniqueName: \"kubernetes.io/projected/086884dc-5574-41f1-952f-f57a355d5294-kube-api-access-vdcp4\") pod \"console-operator-696884fb9c-9zzb8\" (UID: \"086884dc-5574-41f1-952f-f57a355d5294\") " pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" Sep 17 01:26:04.949824 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.949700 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-metrics-certs\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:04.949824 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.949728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-default-certificate\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:04.949824 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.949750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d1b91db-e40c-48f8-8008-f01fd5801ec0-service-ca-bundle\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:04.950096 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:04.949820 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 01:26:04.950096 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:04.949866 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d1b91db-e40c-48f8-8008-f01fd5801ec0-service-ca-bundle podName:4d1b91db-e40c-48f8-8008-f01fd5801ec0 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:05.449846805 +0000 UTC m=+230.684993971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4d1b91db-e40c-48f8-8008-f01fd5801ec0-service-ca-bundle") pod "router-default-585fc78676-kqsfm" (UID: "4d1b91db-e40c-48f8-8008-f01fd5801ec0") : configmap references non-existent config key: service-ca.crt Sep 17 01:26:04.950096 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:04.949887 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-metrics-certs podName:4d1b91db-e40c-48f8-8008-f01fd5801ec0 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:05.449877017 +0000 UTC m=+230.685024189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-metrics-certs") pod "router-default-585fc78676-kqsfm" (UID: "4d1b91db-e40c-48f8-8008-f01fd5801ec0") : secret "router-metrics-certs-default" not found Sep 17 01:26:04.950096 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.949862 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24dd28d0-d328-41a2-83bc-d392d123272b-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-gtwjd\" (UID: \"24dd28d0-d328-41a2-83bc-d392d123272b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gtwjd" Sep 17 01:26:04.950096 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.949952 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/086884dc-5574-41f1-952f-f57a355d5294-serving-cert\") pod \"console-operator-696884fb9c-9zzb8\" (UID: \"086884dc-5574-41f1-952f-f57a355d5294\") " pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" Sep 17 01:26:04.950096 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.949986 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24dd28d0-d328-41a2-83bc-d392d123272b-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-gtwjd\" (UID: \"24dd28d0-d328-41a2-83bc-d392d123272b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gtwjd" Sep 17 01:26:04.950096 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.950011 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/086884dc-5574-41f1-952f-f57a355d5294-config\") pod \"console-operator-696884fb9c-9zzb8\" (UID: \"086884dc-5574-41f1-952f-f57a355d5294\") " pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" Sep 17 01:26:04.950096 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.950035 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5hmc\" (UniqueName: \"kubernetes.io/projected/48427bec-aba8-484c-824f-f3391875b2e4-kube-api-access-k5hmc\") pod \"network-check-source-6f8b768455-ptlrg\" (UID: \"48427bec-aba8-484c-824f-f3391875b2e4\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-ptlrg" Sep 17 01:26:04.950448 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.950130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-stats-auth\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:04.950448 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.950152 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lss9s\" (UniqueName: \"kubernetes.io/projected/4d1b91db-e40c-48f8-8008-f01fd5801ec0-kube-api-access-lss9s\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:04.950448 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.950175 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/086884dc-5574-41f1-952f-f57a355d5294-trusted-ca\") pod \"console-operator-696884fb9c-9zzb8\" (UID: \"086884dc-5574-41f1-952f-f57a355d5294\") " pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" Sep 17 01:26:04.952323 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.952302 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-default-certificate\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:04.952379 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.952365 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-stats-auth\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:04.959519 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.959497 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lss9s\" (UniqueName: \"kubernetes.io/projected/4d1b91db-e40c-48f8-8008-f01fd5801ec0-kube-api-access-lss9s\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:04.959794 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:04.959770 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvwzd\" (UniqueName: \"kubernetes.io/projected/535cb079-9f6c-4cab-ace5-1664b0dd0523-kube-api-access-mvwzd\") pod \"volume-data-source-validator-86695b7cc7-b87fj\" (UID: \"535cb079-9f6c-4cab-ace5-1664b0dd0523\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-b87fj" Sep 17 01:26:05.050606 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.050525 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24dd28d0-d328-41a2-83bc-d392d123272b-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-gtwjd\" (UID: \"24dd28d0-d328-41a2-83bc-d392d123272b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gtwjd" Sep 17 01:26:05.050606 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.050560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/086884dc-5574-41f1-952f-f57a355d5294-config\") pod \"console-operator-696884fb9c-9zzb8\" (UID: \"086884dc-5574-41f1-952f-f57a355d5294\") " pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" Sep 17 01:26:05.050606 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.050578 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5hmc\" (UniqueName: \"kubernetes.io/projected/48427bec-aba8-484c-824f-f3391875b2e4-kube-api-access-k5hmc\") pod \"network-check-source-6f8b768455-ptlrg\" (UID: \"48427bec-aba8-484c-824f-f3391875b2e4\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-ptlrg" Sep 17 01:26:05.050863 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.050630 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/086884dc-5574-41f1-952f-f57a355d5294-trusted-ca\") pod \"console-operator-696884fb9c-9zzb8\" (UID: \"086884dc-5574-41f1-952f-f57a355d5294\") " pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" Sep 17 01:26:05.050863 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.050654 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4kc8\" (UniqueName: \"kubernetes.io/projected/24dd28d0-d328-41a2-83bc-d392d123272b-kube-api-access-x4kc8\") pod \"kube-storage-version-migrator-operator-6c8f84f897-gtwjd\" (UID: \"24dd28d0-d328-41a2-83bc-d392d123272b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gtwjd" Sep 17 01:26:05.050863 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.050671 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdcp4\" (UniqueName: \"kubernetes.io/projected/086884dc-5574-41f1-952f-f57a355d5294-kube-api-access-vdcp4\") pod \"console-operator-696884fb9c-9zzb8\" (UID: \"086884dc-5574-41f1-952f-f57a355d5294\") " pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" Sep 17 01:26:05.050863 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.050728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24dd28d0-d328-41a2-83bc-d392d123272b-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-gtwjd\" (UID: \"24dd28d0-d328-41a2-83bc-d392d123272b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gtwjd" Sep 17 01:26:05.050863 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.050752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/086884dc-5574-41f1-952f-f57a355d5294-serving-cert\") pod \"console-operator-696884fb9c-9zzb8\" (UID: \"086884dc-5574-41f1-952f-f57a355d5294\") " pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" Sep 17 01:26:05.051244 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.051212 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24dd28d0-d328-41a2-83bc-d392d123272b-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-gtwjd\" (UID: \"24dd28d0-d328-41a2-83bc-d392d123272b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gtwjd" Sep 17 01:26:05.051362 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.051344 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/086884dc-5574-41f1-952f-f57a355d5294-config\") pod \"console-operator-696884fb9c-9zzb8\" (UID: \"086884dc-5574-41f1-952f-f57a355d5294\") " pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" Sep 17 01:26:05.051724 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.051701 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/086884dc-5574-41f1-952f-f57a355d5294-trusted-ca\") pod \"console-operator-696884fb9c-9zzb8\" (UID: \"086884dc-5574-41f1-952f-f57a355d5294\") " pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" Sep 17 01:26:05.053106 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.053086 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24dd28d0-d328-41a2-83bc-d392d123272b-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-gtwjd\" (UID: \"24dd28d0-d328-41a2-83bc-d392d123272b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gtwjd" Sep 17 01:26:05.053202 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.053129 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/086884dc-5574-41f1-952f-f57a355d5294-serving-cert\") pod \"console-operator-696884fb9c-9zzb8\" (UID: \"086884dc-5574-41f1-952f-f57a355d5294\") " pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" Sep 17 01:26:05.057808 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.057785 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4kc8\" (UniqueName: \"kubernetes.io/projected/24dd28d0-d328-41a2-83bc-d392d123272b-kube-api-access-x4kc8\") pod \"kube-storage-version-migrator-operator-6c8f84f897-gtwjd\" (UID: \"24dd28d0-d328-41a2-83bc-d392d123272b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gtwjd" Sep 17 01:26:05.058149 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.058132 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5hmc\" (UniqueName: \"kubernetes.io/projected/48427bec-aba8-484c-824f-f3391875b2e4-kube-api-access-k5hmc\") pod \"network-check-source-6f8b768455-ptlrg\" (UID: \"48427bec-aba8-484c-824f-f3391875b2e4\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-ptlrg" Sep 17 01:26:05.058242 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.058181 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdcp4\" (UniqueName: \"kubernetes.io/projected/086884dc-5574-41f1-952f-f57a355d5294-kube-api-access-vdcp4\") pod \"console-operator-696884fb9c-9zzb8\" (UID: \"086884dc-5574-41f1-952f-f57a355d5294\") " pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" Sep 17 01:26:05.083802 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.083780 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-b87fj" Sep 17 01:26:05.183259 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.183231 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gtwjd" Sep 17 01:26:05.191051 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.191032 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" Sep 17 01:26:05.194643 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.194627 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ptlrg" Sep 17 01:26:05.351326 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.351298 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-b87fj"] Sep 17 01:26:05.354313 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:26:05.354281 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535cb079_9f6c_4cab_ace5_1664b0dd0523.slice/crio-c96ebe9c653855625d933ffafccd0ffd57c1ee0f2f7be6b056d5f4042ac17e2d WatchSource:0}: Error finding container c96ebe9c653855625d933ffafccd0ffd57c1ee0f2f7be6b056d5f4042ac17e2d: Status 404 returned error can't find the container with id c96ebe9c653855625d933ffafccd0ffd57c1ee0f2f7be6b056d5f4042ac17e2d Sep 17 01:26:05.448762 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.448733 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gtwjd"] Sep 17 01:26:05.452477 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:26:05.452450 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24dd28d0_d328_41a2_83bc_d392d123272b.slice/crio-6a6ebeeda6d5c2a6bdc0e4fb29e6c9a7fb183b20a42b2f2cb3593db165c4ecf1 WatchSource:0}: Error finding container 6a6ebeeda6d5c2a6bdc0e4fb29e6c9a7fb183b20a42b2f2cb3593db165c4ecf1: Status 404 returned error can't find the container with id 6a6ebeeda6d5c2a6bdc0e4fb29e6c9a7fb183b20a42b2f2cb3593db165c4ecf1 Sep 17 01:26:05.453625 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.453601 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d1b91db-e40c-48f8-8008-f01fd5801ec0-service-ca-bundle\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:05.453730 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.453703 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-metrics-certs\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:05.453790 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:05.453778 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d1b91db-e40c-48f8-8008-f01fd5801ec0-service-ca-bundle podName:4d1b91db-e40c-48f8-8008-f01fd5801ec0 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:06.453755965 +0000 UTC m=+231.688903147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4d1b91db-e40c-48f8-8008-f01fd5801ec0-service-ca-bundle") pod "router-default-585fc78676-kqsfm" (UID: "4d1b91db-e40c-48f8-8008-f01fd5801ec0") : configmap references non-existent config key: service-ca.crt Sep 17 01:26:05.453849 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:05.453788 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 01:26:05.453849 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:05.453831 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-metrics-certs podName:4d1b91db-e40c-48f8-8008-f01fd5801ec0 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:06.45381926 +0000 UTC m=+231.688966425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-metrics-certs") pod "router-default-585fc78676-kqsfm" (UID: "4d1b91db-e40c-48f8-8008-f01fd5801ec0") : secret "router-metrics-certs-default" not found Sep 17 01:26:05.470431 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.470402 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-ptlrg"] Sep 17 01:26:05.473275 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.473249 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-9zzb8"] Sep 17 01:26:05.473945 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:26:05.473908 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48427bec_aba8_484c_824f_f3391875b2e4.slice/crio-fc5b5303a40de6275f7c1d9bd7290b62df8bebaa784968fa694d0031d56736e6 WatchSource:0}: Error finding container fc5b5303a40de6275f7c1d9bd7290b62df8bebaa784968fa694d0031d56736e6: Status 404 returned error can't find the container with id fc5b5303a40de6275f7c1d9bd7290b62df8bebaa784968fa694d0031d56736e6 Sep 17 01:26:05.476443 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:26:05.476356 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086884dc_5574_41f1_952f_f57a355d5294.slice/crio-a118ef0ddd7cf898ffa576641032cccca73ec2f4cbe8899474687f0cf8af944c WatchSource:0}: Error finding container a118ef0ddd7cf898ffa576641032cccca73ec2f4cbe8899474687f0cf8af944c: Status 404 returned error can't find the container with id a118ef0ddd7cf898ffa576641032cccca73ec2f4cbe8899474687f0cf8af944c Sep 17 01:26:05.913974 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.913914 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gtwjd" event={"ID":"24dd28d0-d328-41a2-83bc-d392d123272b","Type":"ContainerStarted","Data":"6a6ebeeda6d5c2a6bdc0e4fb29e6c9a7fb183b20a42b2f2cb3593db165c4ecf1"} Sep 17 01:26:05.915380 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.915351 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ptlrg" event={"ID":"48427bec-aba8-484c-824f-f3391875b2e4","Type":"ContainerStarted","Data":"64a1f2c070a648b2406e552e0de50027c6a77dc611ca9d193fc5a0355cb37b67"} Sep 17 01:26:05.915510 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.915395 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ptlrg" event={"ID":"48427bec-aba8-484c-824f-f3391875b2e4","Type":"ContainerStarted","Data":"fc5b5303a40de6275f7c1d9bd7290b62df8bebaa784968fa694d0031d56736e6"} Sep 17 01:26:05.916437 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.916416 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" event={"ID":"086884dc-5574-41f1-952f-f57a355d5294","Type":"ContainerStarted","Data":"a118ef0ddd7cf898ffa576641032cccca73ec2f4cbe8899474687f0cf8af944c"} Sep 17 01:26:05.917418 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.917400 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-b87fj" event={"ID":"535cb079-9f6c-4cab-ace5-1664b0dd0523","Type":"ContainerStarted","Data":"c96ebe9c653855625d933ffafccd0ffd57c1ee0f2f7be6b056d5f4042ac17e2d"} Sep 17 01:26:05.928905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:05.928866 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-ptlrg" podStartSLOduration=1.928854153 podStartE2EDuration="1.928854153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:05.928108482 +0000 UTC m=+231.163255670" watchObservedRunningTime="2026-09-17 01:26:05.928854153 +0000 UTC m=+231.164001338" Sep 17 01:26:06.463803 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:06.463755 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-metrics-certs\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:06.464028 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:06.463818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d1b91db-e40c-48f8-8008-f01fd5801ec0-service-ca-bundle\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:06.464028 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:06.463946 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 01:26:06.464167 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:06.464026 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d1b91db-e40c-48f8-8008-f01fd5801ec0-service-ca-bundle podName:4d1b91db-e40c-48f8-8008-f01fd5801ec0 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:08.464003048 +0000 UTC m=+233.699150228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4d1b91db-e40c-48f8-8008-f01fd5801ec0-service-ca-bundle") pod "router-default-585fc78676-kqsfm" (UID: "4d1b91db-e40c-48f8-8008-f01fd5801ec0") : configmap references non-existent config key: service-ca.crt Sep 17 01:26:06.464167 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:06.464054 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-metrics-certs podName:4d1b91db-e40c-48f8-8008-f01fd5801ec0 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:08.464043647 +0000 UTC m=+233.699190819 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-metrics-certs") pod "router-default-585fc78676-kqsfm" (UID: "4d1b91db-e40c-48f8-8008-f01fd5801ec0") : secret "router-metrics-certs-default" not found Sep 17 01:26:08.480445 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.480357 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-metrics-certs\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:08.480445 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.480400 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d1b91db-e40c-48f8-8008-f01fd5801ec0-service-ca-bundle\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:08.480904 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:08.480512 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 01:26:08.480904 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:08.480577 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d1b91db-e40c-48f8-8008-f01fd5801ec0-service-ca-bundle podName:4d1b91db-e40c-48f8-8008-f01fd5801ec0 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:12.48056414 +0000 UTC m=+237.715711305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4d1b91db-e40c-48f8-8008-f01fd5801ec0-service-ca-bundle") pod "router-default-585fc78676-kqsfm" (UID: "4d1b91db-e40c-48f8-8008-f01fd5801ec0") : configmap references non-existent config key: service-ca.crt Sep 17 01:26:08.480904 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:08.480593 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-metrics-certs podName:4d1b91db-e40c-48f8-8008-f01fd5801ec0 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:12.480587392 +0000 UTC m=+237.715734558 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-metrics-certs") pod "router-default-585fc78676-kqsfm" (UID: "4d1b91db-e40c-48f8-8008-f01fd5801ec0") : secret "router-metrics-certs-default" not found Sep 17 01:26:08.792663 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.792580 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6drdk"] Sep 17 01:26:08.795702 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.795684 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:08.797808 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.797784 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 17 01:26:08.798604 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.798584 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qkzkb\"" Sep 17 01:26:08.798604 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.798591 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 17 01:26:08.798604 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.798585 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 17 01:26:08.798827 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.798580 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 17 01:26:08.804550 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.804528 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6drdk"] Sep 17 01:26:08.883003 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.882975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f7e40a74-41cf-4f58-a203-0450fc55391e-crio-socket\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:08.883168 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.883034 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7e40a74-41cf-4f58-a203-0450fc55391e-data-volume\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:08.883168 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.883054 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:08.883296 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.883159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7e40a74-41cf-4f58-a203-0450fc55391e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:08.883296 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.883229 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rdj9\" (UniqueName: \"kubernetes.io/projected/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-api-access-2rdj9\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:08.928108 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.928074 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gtwjd" event={"ID":"24dd28d0-d328-41a2-83bc-d392d123272b","Type":"ContainerStarted","Data":"9eea35f8742d2bc6cc1fcd18f07d33cab59e3b9ca0bd8a441b51eb5b4d403aea"} Sep 17 01:26:08.929666 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.929647 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/0.log" Sep 17 01:26:08.929759 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.929679 2577 generic.go:356] "Generic (PLEG): container finished" podID="086884dc-5574-41f1-952f-f57a355d5294" containerID="275955a1304b4833d45f7113108819de2e4a5562c4113b678dd5e9de40830613" exitCode=255 Sep 17 01:26:08.929759 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.929739 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" event={"ID":"086884dc-5574-41f1-952f-f57a355d5294","Type":"ContainerDied","Data":"275955a1304b4833d45f7113108819de2e4a5562c4113b678dd5e9de40830613"} Sep 17 01:26:08.929919 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.929905 2577 scope.go:117] "RemoveContainer" containerID="275955a1304b4833d45f7113108819de2e4a5562c4113b678dd5e9de40830613" Sep 17 01:26:08.931377 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.931355 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-b87fj" event={"ID":"535cb079-9f6c-4cab-ace5-1664b0dd0523","Type":"ContainerStarted","Data":"186f737ae15a9e4421567def06f6e89666698f090bb0ac8373834194c8da21f4"} Sep 17 01:26:08.944392 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.944300 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gtwjd" podStartSLOduration=2.166281937 podStartE2EDuration="4.944282902s" totalImagesPullingTime="2.778000965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:08.942775679 +0000 UTC m=+234.177922868" watchObservedRunningTime="2026-09-17 01:26:08.944282902 +0000 UTC m=+234.179430090" Sep 17 01:26:08.975740 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.975684 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-b87fj" podStartSLOduration=2.1048301719999998 podStartE2EDuration="4.975663252s" totalImagesPullingTime="2.87083308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:08.974649861 +0000 UTC m=+234.209797048" watchObservedRunningTime="2026-09-17 01:26:08.975663252 +0000 UTC m=+234.210810442" Sep 17 01:26:08.984017 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.983990 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f7e40a74-41cf-4f58-a203-0450fc55391e-crio-socket\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:08.984156 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.984079 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f7e40a74-41cf-4f58-a203-0450fc55391e-crio-socket\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:08.984156 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.984101 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7e40a74-41cf-4f58-a203-0450fc55391e-data-volume\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:08.984156 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.984127 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:08.984423 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.984197 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7e40a74-41cf-4f58-a203-0450fc55391e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:08.984423 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.984270 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rdj9\" (UniqueName: \"kubernetes.io/projected/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-api-access-2rdj9\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:08.984900 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:08.984878 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 01:26:08.985014 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:08.984958 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e40a74-41cf-4f58-a203-0450fc55391e-insights-runtime-extractor-tls podName:f7e40a74-41cf-4f58-a203-0450fc55391e nodeName:}" failed. No retries permitted until 2026-09-17 01:26:09.484920897 +0000 UTC m=+234.720068080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7e40a74-41cf-4f58-a203-0450fc55391e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6drdk" (UID: "f7e40a74-41cf-4f58-a203-0450fc55391e") : secret "insights-runtime-extractor-tls" not found Sep 17 01:26:08.985320 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:08.985300 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 01:26:08.985392 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:08.985365 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-rbac-proxy-cm podName:f7e40a74-41cf-4f58-a203-0450fc55391e nodeName:}" failed. No retries permitted until 2026-09-17 01:26:09.485348398 +0000 UTC m=+234.720495587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6drdk" (UID: "f7e40a74-41cf-4f58-a203-0450fc55391e") : configmap "kube-rbac-proxy" not found Sep 17 01:26:08.985454 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.985418 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7e40a74-41cf-4f58-a203-0450fc55391e-data-volume\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:08.992901 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:08.992874 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rdj9\" (UniqueName: \"kubernetes.io/projected/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-api-access-2rdj9\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:09.489552 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:09.489517 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:09.490046 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:09.489560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7e40a74-41cf-4f58-a203-0450fc55391e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:09.490046 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:09.489626 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 01:26:09.490046 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:09.489659 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 01:26:09.490046 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:09.489681 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-rbac-proxy-cm podName:f7e40a74-41cf-4f58-a203-0450fc55391e nodeName:}" failed. No retries permitted until 2026-09-17 01:26:10.489668396 +0000 UTC m=+235.724815562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6drdk" (UID: "f7e40a74-41cf-4f58-a203-0450fc55391e") : configmap "kube-rbac-proxy" not found Sep 17 01:26:09.490046 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:09.489694 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e40a74-41cf-4f58-a203-0450fc55391e-insights-runtime-extractor-tls podName:f7e40a74-41cf-4f58-a203-0450fc55391e nodeName:}" failed. No retries permitted until 2026-09-17 01:26:10.489688631 +0000 UTC m=+235.724835796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7e40a74-41cf-4f58-a203-0450fc55391e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6drdk" (UID: "f7e40a74-41cf-4f58-a203-0450fc55391e") : secret "insights-runtime-extractor-tls" not found Sep 17 01:26:09.936015 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:09.935984 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 01:26:09.936471 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:09.936449 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/0.log" Sep 17 01:26:09.936590 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:09.936495 2577 generic.go:356] "Generic (PLEG): container finished" podID="086884dc-5574-41f1-952f-f57a355d5294" containerID="4c35a44e815320bbf10522a1b2e56f73232efce8bc8db67997e7f18deeebe57c" exitCode=255 Sep 17 01:26:09.936590 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:09.936567 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" event={"ID":"086884dc-5574-41f1-952f-f57a355d5294","Type":"ContainerDied","Data":"4c35a44e815320bbf10522a1b2e56f73232efce8bc8db67997e7f18deeebe57c"} Sep 17 01:26:09.936698 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:09.936632 2577 scope.go:117] "RemoveContainer" containerID="275955a1304b4833d45f7113108819de2e4a5562c4113b678dd5e9de40830613" Sep 17 01:26:09.936842 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:09.936826 2577 scope.go:117] "RemoveContainer" containerID="4c35a44e815320bbf10522a1b2e56f73232efce8bc8db67997e7f18deeebe57c" Sep 17 01:26:09.937039 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:09.937016 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-9zzb8_openshift-console-operator(086884dc-5574-41f1-952f-f57a355d5294)\"" pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" podUID="086884dc-5574-41f1-952f-f57a355d5294" Sep 17 01:26:10.184328 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:10.184295 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-g227w"] Sep 17 01:26:10.188086 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:10.188022 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-g227w" Sep 17 01:26:10.190112 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:10.190089 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 17 01:26:10.190278 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:10.190206 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 17 01:26:10.190278 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:10.190264 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nb6gw\"" Sep 17 01:26:10.195033 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:10.194652 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-g227w"] Sep 17 01:26:10.296751 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:10.296716 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm9fb\" (UniqueName: \"kubernetes.io/projected/69f841a5-fa85-4f3d-bbe6-0f86fec7af12-kube-api-access-zm9fb\") pod \"migrator-8b4d697c5-g227w\" (UID: \"69f841a5-fa85-4f3d-bbe6-0f86fec7af12\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-g227w" Sep 17 01:26:10.397153 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:10.397115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm9fb\" (UniqueName: \"kubernetes.io/projected/69f841a5-fa85-4f3d-bbe6-0f86fec7af12-kube-api-access-zm9fb\") pod \"migrator-8b4d697c5-g227w\" (UID: \"69f841a5-fa85-4f3d-bbe6-0f86fec7af12\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-g227w" Sep 17 01:26:10.404768 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:10.404743 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm9fb\" (UniqueName: \"kubernetes.io/projected/69f841a5-fa85-4f3d-bbe6-0f86fec7af12-kube-api-access-zm9fb\") pod \"migrator-8b4d697c5-g227w\" (UID: \"69f841a5-fa85-4f3d-bbe6-0f86fec7af12\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-g227w" Sep 17 01:26:10.497607 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:10.497513 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-g227w" Sep 17 01:26:10.498004 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:10.497672 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:10.498004 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:10.497728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7e40a74-41cf-4f58-a203-0450fc55391e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:10.498004 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:10.497797 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 01:26:10.498004 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:10.497886 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-rbac-proxy-cm podName:f7e40a74-41cf-4f58-a203-0450fc55391e nodeName:}" failed. No retries permitted until 2026-09-17 01:26:12.497866319 +0000 UTC m=+237.733013504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6drdk" (UID: "f7e40a74-41cf-4f58-a203-0450fc55391e") : configmap "kube-rbac-proxy" not found Sep 17 01:26:10.498004 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:10.497888 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 01:26:10.498004 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:10.497946 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e40a74-41cf-4f58-a203-0450fc55391e-insights-runtime-extractor-tls podName:f7e40a74-41cf-4f58-a203-0450fc55391e nodeName:}" failed. No retries permitted until 2026-09-17 01:26:12.497916928 +0000 UTC m=+237.733064102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7e40a74-41cf-4f58-a203-0450fc55391e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6drdk" (UID: "f7e40a74-41cf-4f58-a203-0450fc55391e") : secret "insights-runtime-extractor-tls" not found Sep 17 01:26:10.760228 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:10.760150 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-g227w"] Sep 17 01:26:10.763120 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:26:10.763089 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f841a5_fa85_4f3d_bbe6_0f86fec7af12.slice/crio-ac70a6da2e0c78a997b5d9311c7357e46a3c46c79c60ff830d11919f3c70cba3 WatchSource:0}: Error finding container ac70a6da2e0c78a997b5d9311c7357e46a3c46c79c60ff830d11919f3c70cba3: Status 404 returned error can't find the container with id ac70a6da2e0c78a997b5d9311c7357e46a3c46c79c60ff830d11919f3c70cba3 Sep 17 01:26:10.940807 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:10.940771 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-g227w" event={"ID":"69f841a5-fa85-4f3d-bbe6-0f86fec7af12","Type":"ContainerStarted","Data":"ac70a6da2e0c78a997b5d9311c7357e46a3c46c79c60ff830d11919f3c70cba3"} Sep 17 01:26:10.942305 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:10.942284 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 01:26:10.942594 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:10.942579 2577 scope.go:117] "RemoveContainer" containerID="4c35a44e815320bbf10522a1b2e56f73232efce8bc8db67997e7f18deeebe57c" Sep 17 01:26:10.942751 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:10.942732 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-9zzb8_openshift-console-operator(086884dc-5574-41f1-952f-f57a355d5294)\"" pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" podUID="086884dc-5574-41f1-952f-f57a355d5294" Sep 17 01:26:10.951287 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:10.951272 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ttq94_78186fc7-1ae0-4bf9-89ec-38c47218f909/dns-node-resolver/0.log" Sep 17 01:26:11.748659 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:11.748629 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-km877_7b6b4978-53de-4fa4-83cc-21d3de40ea24/node-ca/0.log" Sep 17 01:26:12.348427 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:12.348394 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6p5q7_2a717347-6484-44b8-bdb8-19ca455b2905/serve-healthcheck-canary/0.log" Sep 17 01:26:12.511595 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:12.511562 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-metrics-certs\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:12.511696 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:12.511601 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:12.511696 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:12.511619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d1b91db-e40c-48f8-8008-f01fd5801ec0-service-ca-bundle\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:12.511696 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:12.511644 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7e40a74-41cf-4f58-a203-0450fc55391e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:12.511696 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:12.511681 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 01:26:12.511844 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:12.511704 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 17 01:26:12.511844 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:12.511734 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-rbac-proxy-cm podName:f7e40a74-41cf-4f58-a203-0450fc55391e nodeName:}" failed. No retries permitted until 2026-09-17 01:26:16.511721298 +0000 UTC m=+241.746868464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6drdk" (UID: "f7e40a74-41cf-4f58-a203-0450fc55391e") : configmap "kube-rbac-proxy" not found Sep 17 01:26:12.511844 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:12.511754 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 01:26:12.511844 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:12.511755 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-metrics-certs podName:4d1b91db-e40c-48f8-8008-f01fd5801ec0 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:20.511741612 +0000 UTC m=+245.746888778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-metrics-certs") pod "router-default-585fc78676-kqsfm" (UID: "4d1b91db-e40c-48f8-8008-f01fd5801ec0") : secret "router-metrics-certs-default" not found Sep 17 01:26:12.511844 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:12.511798 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d1b91db-e40c-48f8-8008-f01fd5801ec0-service-ca-bundle podName:4d1b91db-e40c-48f8-8008-f01fd5801ec0 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:20.511785874 +0000 UTC m=+245.746933046 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4d1b91db-e40c-48f8-8008-f01fd5801ec0-service-ca-bundle") pod "router-default-585fc78676-kqsfm" (UID: "4d1b91db-e40c-48f8-8008-f01fd5801ec0") : configmap references non-existent config key: service-ca.crt Sep 17 01:26:12.511844 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:12.511811 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e40a74-41cf-4f58-a203-0450fc55391e-insights-runtime-extractor-tls podName:f7e40a74-41cf-4f58-a203-0450fc55391e nodeName:}" failed. No retries permitted until 2026-09-17 01:26:16.511804401 +0000 UTC m=+241.746951566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7e40a74-41cf-4f58-a203-0450fc55391e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6drdk" (UID: "f7e40a74-41cf-4f58-a203-0450fc55391e") : secret "insights-runtime-extractor-tls" not found Sep 17 01:26:12.950472 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:12.950438 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-g227w" event={"ID":"69f841a5-fa85-4f3d-bbe6-0f86fec7af12","Type":"ContainerStarted","Data":"8e9d90d939b46465d28a3c35cb2264b67cdb1e88d09c8756dd90fa00da73284d"} Sep 17 01:26:12.950828 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:12.950530 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-g227w" event={"ID":"69f841a5-fa85-4f3d-bbe6-0f86fec7af12","Type":"ContainerStarted","Data":"36f28beda3dd518be7d766178d0871f89cecbdeebc41610163cfc3bd067c2085"} Sep 17 01:26:12.964809 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:12.964765 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-g227w" podStartSLOduration=1.827990417 podStartE2EDuration="2.96475217s" totalImagesPullingTime="1.136761753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:12.964071759 +0000 UTC m=+238.199218948" watchObservedRunningTime="2026-09-17 01:26:12.96475217 +0000 UTC m=+238.199899357" Sep 17 01:26:15.192069 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:15.192045 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" Sep 17 01:26:15.192403 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:15.192085 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" Sep 17 01:26:15.192403 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:15.192340 2577 scope.go:117] "RemoveContainer" containerID="4c35a44e815320bbf10522a1b2e56f73232efce8bc8db67997e7f18deeebe57c" Sep 17 01:26:15.192479 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:15.192461 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-9zzb8_openshift-console-operator(086884dc-5574-41f1-952f-f57a355d5294)\"" pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" podUID="086884dc-5574-41f1-952f-f57a355d5294" Sep 17 01:26:16.541631 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:16.541592 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:16.542042 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:16.541650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7e40a74-41cf-4f58-a203-0450fc55391e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:16.542042 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:16.541713 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 01:26:16.542042 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:16.541741 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 01:26:16.542042 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:16.541780 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-rbac-proxy-cm podName:f7e40a74-41cf-4f58-a203-0450fc55391e nodeName:}" failed. No retries permitted until 2026-09-17 01:26:24.54176495 +0000 UTC m=+249.776912116 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6drdk" (UID: "f7e40a74-41cf-4f58-a203-0450fc55391e") : configmap "kube-rbac-proxy" not found Sep 17 01:26:16.542042 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:16.541795 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e40a74-41cf-4f58-a203-0450fc55391e-insights-runtime-extractor-tls podName:f7e40a74-41cf-4f58-a203-0450fc55391e nodeName:}" failed. No retries permitted until 2026-09-17 01:26:24.541788676 +0000 UTC m=+249.776935842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7e40a74-41cf-4f58-a203-0450fc55391e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6drdk" (UID: "f7e40a74-41cf-4f58-a203-0450fc55391e") : secret "insights-runtime-extractor-tls" not found Sep 17 01:26:20.569001 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:20.568954 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-metrics-certs\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:20.569001 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:20.569011 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d1b91db-e40c-48f8-8008-f01fd5801ec0-service-ca-bundle\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:20.569576 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:20.569557 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d1b91db-e40c-48f8-8008-f01fd5801ec0-service-ca-bundle\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:20.571255 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:20.571232 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1b91db-e40c-48f8-8008-f01fd5801ec0-metrics-certs\") pod \"router-default-585fc78676-kqsfm\" (UID: \"4d1b91db-e40c-48f8-8008-f01fd5801ec0\") " pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:20.690993 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:20.690965 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-q4nln\"" Sep 17 01:26:20.699616 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:20.699596 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:20.967513 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:20.967482 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-585fc78676-kqsfm"] Sep 17 01:26:20.970560 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:26:20.970530 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d1b91db_e40c_48f8_8008_f01fd5801ec0.slice/crio-4bf414f58f33342945c63928efba94db2a3ea161bca744ae4a7e8cb648a0b607 WatchSource:0}: Error finding container 4bf414f58f33342945c63928efba94db2a3ea161bca744ae4a7e8cb648a0b607: Status 404 returned error can't find the container with id 4bf414f58f33342945c63928efba94db2a3ea161bca744ae4a7e8cb648a0b607 Sep 17 01:26:20.975157 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:20.975135 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-585fc78676-kqsfm" event={"ID":"4d1b91db-e40c-48f8-8008-f01fd5801ec0","Type":"ContainerStarted","Data":"4bf414f58f33342945c63928efba94db2a3ea161bca744ae4a7e8cb648a0b607"} Sep 17 01:26:21.982907 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:21.982849 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-585fc78676-kqsfm" event={"ID":"4d1b91db-e40c-48f8-8008-f01fd5801ec0","Type":"ContainerStarted","Data":"b186bf1f9ce22a4d03328cc99e63a301f4835fc2a8291f44ee54ef7381ce8a86"} Sep 17 01:26:21.999486 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:21.999434 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-585fc78676-kqsfm" podStartSLOduration=17.999421237 podStartE2EDuration="17.999421237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:21.998382106 +0000 UTC m=+247.233529295" watchObservedRunningTime="2026-09-17 01:26:21.999421237 +0000 UTC m=+247.234568425" Sep 17 01:26:22.700621 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:22.700579 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:22.703243 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:22.703218 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:22.985988 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:22.985881 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:22.987184 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:22.987161 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-585fc78676-kqsfm" Sep 17 01:26:24.597572 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:24.597541 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:24.597572 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:24.597576 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7e40a74-41cf-4f58-a203-0450fc55391e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:24.598096 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:24.598074 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7e40a74-41cf-4f58-a203-0450fc55391e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:24.599823 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:24.599797 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7e40a74-41cf-4f58-a203-0450fc55391e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6drdk\" (UID: \"f7e40a74-41cf-4f58-a203-0450fc55391e\") " pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:24.708224 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:24.708197 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qkzkb\"" Sep 17 01:26:24.716772 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:24.716754 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6drdk" Sep 17 01:26:24.981818 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:24.981741 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6drdk"] Sep 17 01:26:24.984464 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:26:24.984439 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e40a74_41cf_4f58_a203_0450fc55391e.slice/crio-11b2c59a2624cbe47d44995e6aa251f07587e7a3a6ddf4c73e6ffc6da94c9651 WatchSource:0}: Error finding container 11b2c59a2624cbe47d44995e6aa251f07587e7a3a6ddf4c73e6ffc6da94c9651: Status 404 returned error can't find the container with id 11b2c59a2624cbe47d44995e6aa251f07587e7a3a6ddf4c73e6ffc6da94c9651 Sep 17 01:26:24.992737 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:24.992709 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6drdk" event={"ID":"f7e40a74-41cf-4f58-a203-0450fc55391e","Type":"ContainerStarted","Data":"11b2c59a2624cbe47d44995e6aa251f07587e7a3a6ddf4c73e6ffc6da94c9651"} Sep 17 01:26:26.000091 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:26.000011 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6drdk" event={"ID":"f7e40a74-41cf-4f58-a203-0450fc55391e","Type":"ContainerStarted","Data":"e51da7ec178fcef21de352b39eb2c820195f68069d2bdab88d6e01b7655814ed"} Sep 17 01:26:26.000091 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:26.000066 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6drdk" event={"ID":"f7e40a74-41cf-4f58-a203-0450fc55391e","Type":"ContainerStarted","Data":"44674096b49901193a5bea1c1c75cc76ac97e4ec9d283cc5f7644e766b033dcc"} Sep 17 01:26:26.206378 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:26.206352 2577 scope.go:117] "RemoveContainer" containerID="4c35a44e815320bbf10522a1b2e56f73232efce8bc8db67997e7f18deeebe57c" Sep 17 01:26:27.006283 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:27.006250 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 01:26:27.006723 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:27.006343 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" event={"ID":"086884dc-5574-41f1-952f-f57a355d5294","Type":"ContainerStarted","Data":"7a82c4af70ff95d0a099e4df6e072733806728a97b95a05896fe4980e54b1df6"} Sep 17 01:26:27.006723 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:27.006649 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" Sep 17 01:26:27.012101 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:27.012078 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" Sep 17 01:26:27.016472 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:27.016450 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs\") pod \"network-metrics-daemon-rdhqv\" (UID: \"32c97ab8-07c6-4f64-9065-2441c8604a5a\") " pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:26:27.018982 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:27.018958 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32c97ab8-07c6-4f64-9065-2441c8604a5a-metrics-certs\") pod \"network-metrics-daemon-rdhqv\" (UID: \"32c97ab8-07c6-4f64-9065-2441c8604a5a\") " pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:26:27.022042 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:27.022004 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-9zzb8" podStartSLOduration=20.2668782 podStartE2EDuration="23.021987697s" totalImagesPullingTime="2.755109497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:27.021352353 +0000 UTC m=+252.256499543" watchObservedRunningTime="2026-09-17 01:26:27.021987697 +0000 UTC m=+252.257134905" Sep 17 01:26:27.307733 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:27.307668 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k6hcd\"" Sep 17 01:26:27.316375 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:27.316357 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdhqv" Sep 17 01:26:27.583852 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:27.583765 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rdhqv"] Sep 17 01:26:27.586498 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:26:27.586471 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c97ab8_07c6_4f64_9065_2441c8604a5a.slice/crio-81c53d74f0b73b0a7b03d591ece465aabfe4a053d5109513c14f004fd3fc42b3 WatchSource:0}: Error finding container 81c53d74f0b73b0a7b03d591ece465aabfe4a053d5109513c14f004fd3fc42b3: Status 404 returned error can't find the container with id 81c53d74f0b73b0a7b03d591ece465aabfe4a053d5109513c14f004fd3fc42b3 Sep 17 01:26:28.011299 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:28.011264 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6drdk" event={"ID":"f7e40a74-41cf-4f58-a203-0450fc55391e","Type":"ContainerStarted","Data":"d4c4e4e9b57394af946ec2071f060fba0757699489f313b55c4e783d42a9e58b"} Sep 17 01:26:28.012548 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:28.012522 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rdhqv" event={"ID":"32c97ab8-07c6-4f64-9065-2441c8604a5a","Type":"ContainerStarted","Data":"81c53d74f0b73b0a7b03d591ece465aabfe4a053d5109513c14f004fd3fc42b3"} Sep 17 01:26:28.028766 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:28.028719 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6drdk" podStartSLOduration=17.934969815 podStartE2EDuration="20.028706966s" totalImagesPullingTime="2.093737151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:26:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:28.028120069 +0000 UTC m=+253.263267257" watchObservedRunningTime="2026-09-17 01:26:28.028706966 +0000 UTC m=+253.263854148" Sep 17 01:26:29.018042 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:29.017953 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rdhqv" event={"ID":"32c97ab8-07c6-4f64-9065-2441c8604a5a","Type":"ContainerStarted","Data":"54c3c5a5bc258ce48728cd8cc31db98c03ff3a66bbc58c52170143ac35ef3b4f"} Sep 17 01:26:29.018042 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:29.018014 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rdhqv" event={"ID":"32c97ab8-07c6-4f64-9065-2441c8604a5a","Type":"ContainerStarted","Data":"9957b92151347655b9afa30317f0e5f64a12c306149e457b1d67acda3815f396"} Sep 17 01:26:29.031991 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:29.031925 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rdhqv" podStartSLOduration=252.980461181 podStartE2EDuration="4m14.031911843s" totalImagesPullingTime="1.051450662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:29.031436223 +0000 UTC m=+254.266583410" watchObservedRunningTime="2026-09-17 01:26:29.031911843 +0000 UTC m=+254.267059061" Sep 17 01:26:30.981580 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:30.981547 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9n89"] Sep 17 01:26:30.985034 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:30.985006 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9n89" Sep 17 01:26:30.988108 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:30.988086 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-wgdrp\"" Sep 17 01:26:30.988218 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:30.988108 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 17 01:26:30.994897 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:30.994877 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9n89"] Sep 17 01:26:30.995725 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:30.995704 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-lwhdx"] Sep 17 01:26:30.998749 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:30.998730 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-lwhdx" Sep 17 01:26:31.005151 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:31.005127 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 17 01:26:31.005151 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:31.005127 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 17 01:26:31.005316 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:31.005182 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gkv5r\"" Sep 17 01:26:31.021319 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:31.020754 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-lwhdx"] Sep 17 01:26:31.046532 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:31.046504 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8139b531-69ac-4848-9fd9-92e36ef2a87a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-h9n89\" (UID: \"8139b531-69ac-4848-9fd9-92e36ef2a87a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9n89" Sep 17 01:26:31.147338 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:31.147302 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sw74\" (UniqueName: \"kubernetes.io/projected/51861ecb-5643-4b47-9c98-c083d6abec0d-kube-api-access-4sw74\") pod \"downloads-76f99584f4-lwhdx\" (UID: \"51861ecb-5643-4b47-9c98-c083d6abec0d\") " pod="openshift-console/downloads-76f99584f4-lwhdx" Sep 17 01:26:31.147463 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:31.147371 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8139b531-69ac-4848-9fd9-92e36ef2a87a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-h9n89\" (UID: \"8139b531-69ac-4848-9fd9-92e36ef2a87a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9n89" Sep 17 01:26:31.149583 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:31.149558 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8139b531-69ac-4848-9fd9-92e36ef2a87a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-h9n89\" (UID: \"8139b531-69ac-4848-9fd9-92e36ef2a87a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9n89" Sep 17 01:26:31.248467 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:31.248394 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sw74\" (UniqueName: \"kubernetes.io/projected/51861ecb-5643-4b47-9c98-c083d6abec0d-kube-api-access-4sw74\") pod \"downloads-76f99584f4-lwhdx\" (UID: \"51861ecb-5643-4b47-9c98-c083d6abec0d\") " pod="openshift-console/downloads-76f99584f4-lwhdx" Sep 17 01:26:31.258889 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:31.258862 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sw74\" (UniqueName: \"kubernetes.io/projected/51861ecb-5643-4b47-9c98-c083d6abec0d-kube-api-access-4sw74\") pod \"downloads-76f99584f4-lwhdx\" (UID: \"51861ecb-5643-4b47-9c98-c083d6abec0d\") " pod="openshift-console/downloads-76f99584f4-lwhdx" Sep 17 01:26:31.294915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:31.294890 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9n89" Sep 17 01:26:31.311918 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:31.311888 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-lwhdx" Sep 17 01:26:31.563955 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:31.563778 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9n89"] Sep 17 01:26:31.566338 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:26:31.566311 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8139b531_69ac_4848_9fd9_92e36ef2a87a.slice/crio-42862d50068be593bfc97edd4dabfaaa58342c6d2fbf6aa0a4de7de0aad9fd55 WatchSource:0}: Error finding container 42862d50068be593bfc97edd4dabfaaa58342c6d2fbf6aa0a4de7de0aad9fd55: Status 404 returned error can't find the container with id 42862d50068be593bfc97edd4dabfaaa58342c6d2fbf6aa0a4de7de0aad9fd55 Sep 17 01:26:31.588734 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:31.588706 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-lwhdx"] Sep 17 01:26:31.592656 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:26:31.592630 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51861ecb_5643_4b47_9c98_c083d6abec0d.slice/crio-c741940f8454d001aace6767ed786ff5029b839f9dd00cf342b31c0953c1bcea WatchSource:0}: Error finding container c741940f8454d001aace6767ed786ff5029b839f9dd00cf342b31c0953c1bcea: Status 404 returned error can't find the container with id c741940f8454d001aace6767ed786ff5029b839f9dd00cf342b31c0953c1bcea Sep 17 01:26:32.029699 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.029664 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-lwhdx" event={"ID":"51861ecb-5643-4b47-9c98-c083d6abec0d","Type":"ContainerStarted","Data":"c741940f8454d001aace6767ed786ff5029b839f9dd00cf342b31c0953c1bcea"} Sep 17 01:26:32.030815 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.030793 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9n89" event={"ID":"8139b531-69ac-4848-9fd9-92e36ef2a87a","Type":"ContainerStarted","Data":"42862d50068be593bfc97edd4dabfaaa58342c6d2fbf6aa0a4de7de0aad9fd55"} Sep 17 01:26:32.067331 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.067301 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-644fc4dcdd-kgghn"] Sep 17 01:26:32.070474 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.070454 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.073758 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.073737 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 17 01:26:32.073758 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.073755 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 17 01:26:32.073943 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.073776 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 17 01:26:32.073943 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.073737 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2tnjq\"" Sep 17 01:26:32.073943 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.073833 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 17 01:26:32.073943 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.073836 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 17 01:26:32.079746 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.079725 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-644fc4dcdd-kgghn"] Sep 17 01:26:32.157887 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.157860 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/930470db-7a0f-4d96-b72e-04833c1f12d8-console-serving-cert\") pod \"console-644fc4dcdd-kgghn\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.158029 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.157892 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/930470db-7a0f-4d96-b72e-04833c1f12d8-console-oauth-config\") pod \"console-644fc4dcdd-kgghn\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.158029 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.157909 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/930470db-7a0f-4d96-b72e-04833c1f12d8-console-config\") pod \"console-644fc4dcdd-kgghn\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.158029 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.157991 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/930470db-7a0f-4d96-b72e-04833c1f12d8-service-ca\") pod \"console-644fc4dcdd-kgghn\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.158146 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.158078 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/930470db-7a0f-4d96-b72e-04833c1f12d8-oauth-serving-cert\") pod \"console-644fc4dcdd-kgghn\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.158146 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.158107 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nclsc\" (UniqueName: \"kubernetes.io/projected/930470db-7a0f-4d96-b72e-04833c1f12d8-kube-api-access-nclsc\") pod \"console-644fc4dcdd-kgghn\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.258467 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.258429 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/930470db-7a0f-4d96-b72e-04833c1f12d8-oauth-serving-cert\") pod \"console-644fc4dcdd-kgghn\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.258645 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.258476 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nclsc\" (UniqueName: \"kubernetes.io/projected/930470db-7a0f-4d96-b72e-04833c1f12d8-kube-api-access-nclsc\") pod \"console-644fc4dcdd-kgghn\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.258645 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.258514 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/930470db-7a0f-4d96-b72e-04833c1f12d8-console-serving-cert\") pod \"console-644fc4dcdd-kgghn\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.258645 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.258539 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/930470db-7a0f-4d96-b72e-04833c1f12d8-console-oauth-config\") pod \"console-644fc4dcdd-kgghn\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.258645 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.258556 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/930470db-7a0f-4d96-b72e-04833c1f12d8-console-config\") pod \"console-644fc4dcdd-kgghn\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.258645 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.258588 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/930470db-7a0f-4d96-b72e-04833c1f12d8-service-ca\") pod \"console-644fc4dcdd-kgghn\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.259297 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.259260 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/930470db-7a0f-4d96-b72e-04833c1f12d8-oauth-serving-cert\") pod \"console-644fc4dcdd-kgghn\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.259297 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.259263 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/930470db-7a0f-4d96-b72e-04833c1f12d8-service-ca\") pod \"console-644fc4dcdd-kgghn\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.259556 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.259529 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/930470db-7a0f-4d96-b72e-04833c1f12d8-console-config\") pod \"console-644fc4dcdd-kgghn\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.261254 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.261233 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/930470db-7a0f-4d96-b72e-04833c1f12d8-console-serving-cert\") pod \"console-644fc4dcdd-kgghn\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.261368 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.261262 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/930470db-7a0f-4d96-b72e-04833c1f12d8-console-oauth-config\") pod \"console-644fc4dcdd-kgghn\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.266387 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.266367 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nclsc\" (UniqueName: \"kubernetes.io/projected/930470db-7a0f-4d96-b72e-04833c1f12d8-kube-api-access-nclsc\") pod \"console-644fc4dcdd-kgghn\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.380576 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.380539 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:32.664155 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:32.664126 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-644fc4dcdd-kgghn"] Sep 17 01:26:32.791960 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:26:32.791902 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930470db_7a0f_4d96_b72e_04833c1f12d8.slice/crio-bde56d98c3743b51710d4090ff4aa4c73c45c16f15ce8fb9f5325025e1197ed5 WatchSource:0}: Error finding container bde56d98c3743b51710d4090ff4aa4c73c45c16f15ce8fb9f5325025e1197ed5: Status 404 returned error can't find the container with id bde56d98c3743b51710d4090ff4aa4c73c45c16f15ce8fb9f5325025e1197ed5 Sep 17 01:26:33.035442 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:33.035365 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9n89" event={"ID":"8139b531-69ac-4848-9fd9-92e36ef2a87a","Type":"ContainerStarted","Data":"1c7aa20e1f61a1182122c07f8e59df59f7b328c9699151f0c9ee96e3b351a872"} Sep 17 01:26:33.035834 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:33.035543 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9n89" Sep 17 01:26:33.036728 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:33.036702 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644fc4dcdd-kgghn" event={"ID":"930470db-7a0f-4d96-b72e-04833c1f12d8","Type":"ContainerStarted","Data":"bde56d98c3743b51710d4090ff4aa4c73c45c16f15ce8fb9f5325025e1197ed5"} Sep 17 01:26:33.040076 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:33.040053 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9n89" Sep 17 01:26:33.049361 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:33.049324 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h9n89" podStartSLOduration=1.780004264 podStartE2EDuration="3.049309686s" totalImagesPullingTime="1.269305422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:33.047900319 +0000 UTC m=+258.283047485" watchObservedRunningTime="2026-09-17 01:26:33.049309686 +0000 UTC m=+258.284456874" Sep 17 01:26:37.057143 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:37.057105 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644fc4dcdd-kgghn" event={"ID":"930470db-7a0f-4d96-b72e-04833c1f12d8","Type":"ContainerStarted","Data":"e7967d20cc56f0e4ca539f6a67f38a3af41831fb35be7de91f3e8e75d5b0d94f"} Sep 17 01:26:37.074358 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:37.074303 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-644fc4dcdd-kgghn" podStartSLOduration=1.880650622 podStartE2EDuration="5.074282314s" totalImagesPullingTime="3.193631692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:37.072577718 +0000 UTC m=+262.307724909" watchObservedRunningTime="2026-09-17 01:26:37.074282314 +0000 UTC m=+262.309429498" Sep 17 01:26:38.304909 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.304875 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz"] Sep 17 01:26:38.308378 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.308348 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz" Sep 17 01:26:38.311261 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.311233 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 17 01:26:38.312323 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.312290 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 17 01:26:38.312457 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.312327 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 17 01:26:38.312457 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.312353 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 17 01:26:38.312622 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.312461 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-w857c\"" Sep 17 01:26:38.312924 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.312711 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 17 01:26:38.321206 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.321185 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz"] Sep 17 01:26:38.333523 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.333503 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-stc9t"] Sep 17 01:26:38.336785 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.336766 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.339033 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.339013 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 17 01:26:38.339325 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.339303 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 17 01:26:38.339428 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.339339 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hlk7m\"" Sep 17 01:26:38.339428 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.339343 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 17 01:26:38.408336 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.408302 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p5zk\" (UniqueName: \"kubernetes.io/projected/ba8140a0-85da-41de-a192-93694fa5b116-kube-api-access-7p5zk\") pod \"openshift-state-metrics-67dcf8c979-9jqgz\" (UID: \"ba8140a0-85da-41de-a192-93694fa5b116\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz" Sep 17 01:26:38.408506 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.408352 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba8140a0-85da-41de-a192-93694fa5b116-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-9jqgz\" (UID: \"ba8140a0-85da-41de-a192-93694fa5b116\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz" Sep 17 01:26:38.408506 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.408384 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f809c0db-e228-416b-b96c-bdd10e842c22-root\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.408506 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.408400 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f809c0db-e228-416b-b96c-bdd10e842c22-metrics-client-ca\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.408506 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.408418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f809c0db-e228-416b-b96c-bdd10e842c22-node-exporter-tls\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.408506 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.408448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba8140a0-85da-41de-a192-93694fa5b116-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-9jqgz\" (UID: \"ba8140a0-85da-41de-a192-93694fa5b116\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz" Sep 17 01:26:38.408506 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.408473 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f809c0db-e228-416b-b96c-bdd10e842c22-sys\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.408506 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.408500 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f809c0db-e228-416b-b96c-bdd10e842c22-node-exporter-accelerators-collector-config\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.408763 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.408549 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f809c0db-e228-416b-b96c-bdd10e842c22-node-exporter-textfile\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.408763 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.408576 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f809c0db-e228-416b-b96c-bdd10e842c22-node-exporter-wtmp\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.408763 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.408600 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba8140a0-85da-41de-a192-93694fa5b116-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-9jqgz\" (UID: \"ba8140a0-85da-41de-a192-93694fa5b116\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz" Sep 17 01:26:38.408763 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.408626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f809c0db-e228-416b-b96c-bdd10e842c22-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.408763 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.408651 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29fkg\" (UniqueName: \"kubernetes.io/projected/f809c0db-e228-416b-b96c-bdd10e842c22-kube-api-access-29fkg\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.509587 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.509556 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f809c0db-e228-416b-b96c-bdd10e842c22-node-exporter-textfile\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.509587 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.509596 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f809c0db-e228-416b-b96c-bdd10e842c22-node-exporter-wtmp\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.509819 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.509623 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba8140a0-85da-41de-a192-93694fa5b116-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-9jqgz\" (UID: \"ba8140a0-85da-41de-a192-93694fa5b116\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz" Sep 17 01:26:38.509819 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.509647 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f809c0db-e228-416b-b96c-bdd10e842c22-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.509819 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.509670 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29fkg\" (UniqueName: \"kubernetes.io/projected/f809c0db-e228-416b-b96c-bdd10e842c22-kube-api-access-29fkg\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.509819 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.509716 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p5zk\" (UniqueName: \"kubernetes.io/projected/ba8140a0-85da-41de-a192-93694fa5b116-kube-api-access-7p5zk\") pod \"openshift-state-metrics-67dcf8c979-9jqgz\" (UID: \"ba8140a0-85da-41de-a192-93694fa5b116\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz" Sep 17 01:26:38.509819 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.509744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba8140a0-85da-41de-a192-93694fa5b116-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-9jqgz\" (UID: \"ba8140a0-85da-41de-a192-93694fa5b116\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz" Sep 17 01:26:38.509819 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.509775 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f809c0db-e228-416b-b96c-bdd10e842c22-root\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.509819 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.509776 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f809c0db-e228-416b-b96c-bdd10e842c22-node-exporter-wtmp\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.509819 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.509798 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f809c0db-e228-416b-b96c-bdd10e842c22-metrics-client-ca\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.510242 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.509824 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f809c0db-e228-416b-b96c-bdd10e842c22-node-exporter-tls\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.510242 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.509859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba8140a0-85da-41de-a192-93694fa5b116-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-9jqgz\" (UID: \"ba8140a0-85da-41de-a192-93694fa5b116\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz" Sep 17 01:26:38.510242 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.509880 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f809c0db-e228-416b-b96c-bdd10e842c22-sys\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.510242 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.509905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f809c0db-e228-416b-b96c-bdd10e842c22-node-exporter-accelerators-collector-config\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.510242 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.509980 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f809c0db-e228-416b-b96c-bdd10e842c22-node-exporter-textfile\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.510242 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.510068 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f809c0db-e228-416b-b96c-bdd10e842c22-root\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.510529 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.510488 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f809c0db-e228-416b-b96c-bdd10e842c22-sys\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.510529 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.510496 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f809c0db-e228-416b-b96c-bdd10e842c22-node-exporter-accelerators-collector-config\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.510529 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.510516 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba8140a0-85da-41de-a192-93694fa5b116-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-9jqgz\" (UID: \"ba8140a0-85da-41de-a192-93694fa5b116\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz" Sep 17 01:26:38.511006 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.510948 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f809c0db-e228-416b-b96c-bdd10e842c22-metrics-client-ca\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.512906 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.512476 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f809c0db-e228-416b-b96c-bdd10e842c22-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.513067 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.513033 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba8140a0-85da-41de-a192-93694fa5b116-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-9jqgz\" (UID: \"ba8140a0-85da-41de-a192-93694fa5b116\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz" Sep 17 01:26:38.513233 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.513208 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f809c0db-e228-416b-b96c-bdd10e842c22-node-exporter-tls\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.513440 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.513412 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba8140a0-85da-41de-a192-93694fa5b116-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-9jqgz\" (UID: \"ba8140a0-85da-41de-a192-93694fa5b116\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz" Sep 17 01:26:38.518424 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.518393 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29fkg\" (UniqueName: \"kubernetes.io/projected/f809c0db-e228-416b-b96c-bdd10e842c22-kube-api-access-29fkg\") pod \"node-exporter-stc9t\" (UID: \"f809c0db-e228-416b-b96c-bdd10e842c22\") " pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.518896 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.518872 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p5zk\" (UniqueName: \"kubernetes.io/projected/ba8140a0-85da-41de-a192-93694fa5b116-kube-api-access-7p5zk\") pod \"openshift-state-metrics-67dcf8c979-9jqgz\" (UID: \"ba8140a0-85da-41de-a192-93694fa5b116\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz" Sep 17 01:26:38.620057 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.620018 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz" Sep 17 01:26:38.647422 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.647394 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-stc9t" Sep 17 01:26:38.658906 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:26:38.658875 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf809c0db_e228_416b_b96c_bdd10e842c22.slice/crio-bbc09b53a527005fdad8cc8eb2ba7cf3eb132d7eb04af7d46de4e22dd62653da WatchSource:0}: Error finding container bbc09b53a527005fdad8cc8eb2ba7cf3eb132d7eb04af7d46de4e22dd62653da: Status 404 returned error can't find the container with id bbc09b53a527005fdad8cc8eb2ba7cf3eb132d7eb04af7d46de4e22dd62653da Sep 17 01:26:38.898280 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:38.898196 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz"] Sep 17 01:26:38.901918 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:26:38.901889 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8140a0_85da_41de_a192_93694fa5b116.slice/crio-f4799484f5fa2e40f1c5f9a4fc474e5a8735100e63fa29a80783c540ceacf4eb WatchSource:0}: Error finding container f4799484f5fa2e40f1c5f9a4fc474e5a8735100e63fa29a80783c540ceacf4eb: Status 404 returned error can't find the container with id f4799484f5fa2e40f1c5f9a4fc474e5a8735100e63fa29a80783c540ceacf4eb Sep 17 01:26:39.066722 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.066690 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-stc9t" event={"ID":"f809c0db-e228-416b-b96c-bdd10e842c22","Type":"ContainerStarted","Data":"bbc09b53a527005fdad8cc8eb2ba7cf3eb132d7eb04af7d46de4e22dd62653da"} Sep 17 01:26:39.069492 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.069466 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz" event={"ID":"ba8140a0-85da-41de-a192-93694fa5b116","Type":"ContainerStarted","Data":"efbcaa7690f2208f17b7541f6f7601b4b4968cd34e34f2487fd03b7b970a0452"} Sep 17 01:26:39.069606 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.069524 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz" event={"ID":"ba8140a0-85da-41de-a192-93694fa5b116","Type":"ContainerStarted","Data":"a9fda33ed30ce7546f881409b44ea4e0307fd0d4badf308db0e7c47640a71170"} Sep 17 01:26:39.069606 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.069553 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz" event={"ID":"ba8140a0-85da-41de-a192-93694fa5b116","Type":"ContainerStarted","Data":"f4799484f5fa2e40f1c5f9a4fc474e5a8735100e63fa29a80783c540ceacf4eb"} Sep 17 01:26:39.361601 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.361558 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 01:26:39.365833 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.365800 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.368752 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.368188 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 17 01:26:39.368752 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.368411 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 17 01:26:39.368752 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.368441 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 17 01:26:39.368752 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.368486 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 17 01:26:39.368752 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.368509 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 17 01:26:39.368752 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.368638 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 17 01:26:39.368752 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.368668 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 17 01:26:39.368752 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.368762 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 17 01:26:39.369286 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.369033 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 17 01:26:39.369286 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.369035 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qzjv6\"" Sep 17 01:26:39.377657 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.377575 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 01:26:39.519015 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.518746 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.519015 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.518792 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b2b71772-281b-438d-a110-36d2050aac21-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.519015 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.518829 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.519015 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.518859 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2b71772-281b-438d-a110-36d2050aac21-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.519015 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.518885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt9dp\" (UniqueName: \"kubernetes.io/projected/b2b71772-281b-438d-a110-36d2050aac21-kube-api-access-kt9dp\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.519015 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.518909 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.519431 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.519094 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-config-volume\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.519431 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.519214 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-web-config\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.519431 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.519245 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.519431 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.519278 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2b71772-281b-438d-a110-36d2050aac21-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.519431 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.519327 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.519431 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.519356 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2b71772-281b-438d-a110-36d2050aac21-config-out\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.519431 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.519382 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b2b71772-281b-438d-a110-36d2050aac21-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.620533 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.620258 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.620533 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.620309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b2b71772-281b-438d-a110-36d2050aac21-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.620533 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.620351 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.620533 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.620382 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2b71772-281b-438d-a110-36d2050aac21-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.620533 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.620409 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt9dp\" (UniqueName: \"kubernetes.io/projected/b2b71772-281b-438d-a110-36d2050aac21-kube-api-access-kt9dp\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.620533 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.620435 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.620533 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.620464 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-config-volume\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.620533 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.620495 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-web-config\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.620533 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.620522 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.621146 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.620557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2b71772-281b-438d-a110-36d2050aac21-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.621146 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.620603 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.621146 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.620633 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2b71772-281b-438d-a110-36d2050aac21-config-out\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.621146 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.620658 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b2b71772-281b-438d-a110-36d2050aac21-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.621146 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.620761 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b2b71772-281b-438d-a110-36d2050aac21-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.621146 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:39.620971 2577 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 17 01:26:39.621146 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:39.621039 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-main-tls podName:b2b71772-281b-438d-a110-36d2050aac21 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:40.121016204 +0000 UTC m=+265.356163376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "b2b71772-281b-438d-a110-36d2050aac21") : secret "alertmanager-main-tls" not found Sep 17 01:26:39.622468 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.622218 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2b71772-281b-438d-a110-36d2050aac21-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.622468 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.622335 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2b71772-281b-438d-a110-36d2050aac21-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.625056 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.625008 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2b71772-281b-438d-a110-36d2050aac21-config-out\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.625165 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.625097 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b2b71772-281b-438d-a110-36d2050aac21-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.625624 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.625569 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.625624 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.625595 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.625852 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.625806 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-web-config\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.626064 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.626040 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.627120 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.627051 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.627666 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.627647 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-config-volume\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:39.631612 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:39.631571 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt9dp\" (UniqueName: \"kubernetes.io/projected/b2b71772-281b-438d-a110-36d2050aac21-kube-api-access-kt9dp\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:40.075892 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:40.075853 2577 generic.go:356] "Generic (PLEG): container finished" podID="f809c0db-e228-416b-b96c-bdd10e842c22" containerID="0c997686b1ecdc79bdf1bf8015924401617bf917097ce3921b7912fd6df41313" exitCode=0 Sep 17 01:26:40.076072 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:40.075915 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-stc9t" event={"ID":"f809c0db-e228-416b-b96c-bdd10e842c22","Type":"ContainerDied","Data":"0c997686b1ecdc79bdf1bf8015924401617bf917097ce3921b7912fd6df41313"} Sep 17 01:26:40.124414 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:40.124336 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:40.124555 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:40.124508 2577 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 17 01:26:40.124652 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:40.124619 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-main-tls podName:b2b71772-281b-438d-a110-36d2050aac21 nodeName:}" failed. No retries permitted until 2026-09-17 01:26:41.124589953 +0000 UTC m=+266.359737132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "b2b71772-281b-438d-a110-36d2050aac21") : secret "alertmanager-main-tls" not found Sep 17 01:26:41.082629 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.082592 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-stc9t" event={"ID":"f809c0db-e228-416b-b96c-bdd10e842c22","Type":"ContainerStarted","Data":"64371a6ca1e30cb6d612f024677787706277c581ea965a0379ae0994d2ba2442"} Sep 17 01:26:41.083111 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.082658 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-stc9t" event={"ID":"f809c0db-e228-416b-b96c-bdd10e842c22","Type":"ContainerStarted","Data":"dd08e7480d93346249f51e64537b332acb90ed4b45f1bd3782c6fc002a71e2ca"} Sep 17 01:26:41.085125 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.085098 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz" event={"ID":"ba8140a0-85da-41de-a192-93694fa5b116","Type":"ContainerStarted","Data":"e5797667ccfd0857297655e6f5e0b96a71e576ff3ab04172b36fbf14ff50c8af"} Sep 17 01:26:41.099976 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.099921 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-stc9t" podStartSLOduration=1.937005106 podStartE2EDuration="3.099896251s" totalImagesPullingTime="1.092024001s" totalInitContainerRuntime="70.867144ms" isStatefulPod=true podCreationTimestamp="2026-09-17 01:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:41.098514535 +0000 UTC m=+266.333661723" watchObservedRunningTime="2026-09-17 01:26:41.099896251 +0000 UTC m=+266.335043440" Sep 17 01:26:41.132622 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.132594 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:41.135421 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.135396 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:41.138491 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.138442 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9jqgz" podStartSLOduration=1.7395526019999998 podStartE2EDuration="3.138425007s" totalImagesPullingTime="1.398872405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:41.136401606 +0000 UTC m=+266.371548796" watchObservedRunningTime="2026-09-17 01:26:41.138425007 +0000 UTC m=+266.373572199" Sep 17 01:26:41.179507 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.179482 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:26:41.302063 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.302033 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-559ccbc595-79zsl"] Sep 17 01:26:41.307981 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.307953 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.310698 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.310316 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-52gmu5o44bjvb\"" Sep 17 01:26:41.310698 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.310561 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 17 01:26:41.312317 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.310883 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 17 01:26:41.312317 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.311125 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7gvwv\"" Sep 17 01:26:41.312317 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.311181 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 17 01:26:41.312317 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.311131 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 17 01:26:41.312317 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.311513 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 17 01:26:41.318237 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.318211 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-559ccbc595-79zsl"] Sep 17 01:26:41.434651 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.434612 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.434845 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.434692 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.434917 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.434850 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-secret-thanos-querier-tls\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.434988 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.434925 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-secret-grpc-tls\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.434988 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.434980 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-metrics-client-ca\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.435100 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.435020 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.435160 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.435109 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.435212 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.435155 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc7s7\" (UniqueName: \"kubernetes.io/projected/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-kube-api-access-kc7s7\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.469612 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.469587 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 01:26:41.536279 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.536241 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-secret-grpc-tls\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.536279 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.536292 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-metrics-client-ca\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.536523 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.536329 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.536523 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.536355 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.536523 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.536381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc7s7\" (UniqueName: \"kubernetes.io/projected/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-kube-api-access-kc7s7\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.536523 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.536432 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.536523 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.536482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.536773 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.536541 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-secret-thanos-querier-tls\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.537448 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.537393 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-metrics-client-ca\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.539586 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.539559 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.539729 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.539683 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-secret-grpc-tls\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.540025 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.539996 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.540171 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.540147 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.540257 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.540153 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-secret-thanos-querier-tls\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.540257 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.540157 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.543682 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.543663 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc7s7\" (UniqueName: \"kubernetes.io/projected/bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49-kube-api-access-kc7s7\") pod \"thanos-querier-559ccbc595-79zsl\" (UID: \"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49\") " pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:41.623814 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:41.623773 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:42.381354 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:42.381318 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:42.381813 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:42.381379 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:42.386458 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:42.386429 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:43.097417 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.097384 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:26:43.505193 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.505096 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6445df8557-7hkg9"] Sep 17 01:26:43.508374 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.508341 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.510605 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.510577 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 17 01:26:43.510706 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.510641 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 17 01:26:43.510752 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.510641 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-6jsjz\"" Sep 17 01:26:43.510827 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.510812 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 17 01:26:43.511039 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.510955 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 17 01:26:43.511208 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.511070 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 17 01:26:43.517317 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.517287 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 17 01:26:43.519637 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.519616 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6445df8557-7hkg9"] Sep 17 01:26:43.655670 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.655638 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-metrics-client-ca\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.655670 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.655673 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-serving-certs-ca-bundle\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.655885 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.655700 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-telemeter-client-tls\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.655885 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.655790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.655885 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.655846 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-federate-client-tls\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.656054 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.655901 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-secret-telemeter-client\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.656054 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.655947 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpbk5\" (UniqueName: \"kubernetes.io/projected/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-kube-api-access-dpbk5\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.656054 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.656013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.757289 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.757212 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-metrics-client-ca\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.757289 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.757252 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-serving-certs-ca-bundle\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.757289 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.757289 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-telemeter-client-tls\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.757590 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.757323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.757590 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.757363 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-federate-client-tls\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.757590 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.757419 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-secret-telemeter-client\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.757590 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.757441 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpbk5\" (UniqueName: \"kubernetes.io/projected/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-kube-api-access-dpbk5\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.757590 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.757493 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.758077 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.758046 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-metrics-client-ca\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.758196 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.758139 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-serving-certs-ca-bundle\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.758551 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.758516 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.760304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.760276 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.760572 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.760549 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-federate-client-tls\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.761086 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.761065 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-telemeter-client-tls\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.761162 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.761105 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-secret-telemeter-client\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.764483 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.764464 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpbk5\" (UniqueName: \"kubernetes.io/projected/a2e8bdcb-05af-4b85-99cb-bf9de2cc2901-kube-api-access-dpbk5\") pod \"telemeter-client-6445df8557-7hkg9\" (UID: \"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901\") " pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:43.821408 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:43.821370 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" Sep 17 01:26:47.094705 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.094669 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c55c57948-kwtrf"] Sep 17 01:26:47.098212 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.098180 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.104985 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.104953 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 17 01:26:47.108688 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.108646 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c55c57948-kwtrf"] Sep 17 01:26:47.288187 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.288145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-service-ca\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.288357 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.288198 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwx9b\" (UniqueName: \"kubernetes.io/projected/ca726db6-ff92-4f48-8a13-552c196c709f-kube-api-access-wwx9b\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.288357 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.288289 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-trusted-ca-bundle\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.288357 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.288312 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-oauth-serving-cert\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.288357 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.288355 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca726db6-ff92-4f48-8a13-552c196c709f-console-serving-cert\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.288550 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.288415 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca726db6-ff92-4f48-8a13-552c196c709f-console-oauth-config\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.288550 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.288537 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-console-config\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.389844 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.389762 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca726db6-ff92-4f48-8a13-552c196c709f-console-serving-cert\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.389844 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.389795 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca726db6-ff92-4f48-8a13-552c196c709f-console-oauth-config\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.389844 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.389834 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-console-config\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.390108 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.389870 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-service-ca\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.390108 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.389900 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwx9b\" (UniqueName: \"kubernetes.io/projected/ca726db6-ff92-4f48-8a13-552c196c709f-kube-api-access-wwx9b\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.390108 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.390058 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-trusted-ca-bundle\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.390108 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.390078 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-oauth-serving-cert\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.390858 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.390669 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-console-config\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.390858 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.390762 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-service-ca\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.391065 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.390978 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-trusted-ca-bundle\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.391065 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.390978 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-oauth-serving-cert\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.392855 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.392834 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca726db6-ff92-4f48-8a13-552c196c709f-console-serving-cert\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.392982 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.392884 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca726db6-ff92-4f48-8a13-552c196c709f-console-oauth-config\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.397507 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.397479 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwx9b\" (UniqueName: \"kubernetes.io/projected/ca726db6-ff92-4f48-8a13-552c196c709f-kube-api-access-wwx9b\") pod \"console-6c55c57948-kwtrf\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:47.410432 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:47.410406 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:48.380653 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:26:48.380612 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b71772_281b_438d_a110_36d2050aac21.slice/crio-0b90ac2e3cf2ee9d4609550cc7b1da05c0bd3e74b4d94875dcbf3a2ac1041eea WatchSource:0}: Error finding container 0b90ac2e3cf2ee9d4609550cc7b1da05c0bd3e74b4d94875dcbf3a2ac1041eea: Status 404 returned error can't find the container with id 0b90ac2e3cf2ee9d4609550cc7b1da05c0bd3e74b4d94875dcbf3a2ac1041eea Sep 17 01:26:48.658082 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:48.658008 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c55c57948-kwtrf"] Sep 17 01:26:48.661304 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:26:48.661270 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca726db6_ff92_4f48_8a13_552c196c709f.slice/crio-87563a65b948d02a22efb73c6f780f1dc11bf95293d303128f21d37290f84e7a WatchSource:0}: Error finding container 87563a65b948d02a22efb73c6f780f1dc11bf95293d303128f21d37290f84e7a: Status 404 returned error can't find the container with id 87563a65b948d02a22efb73c6f780f1dc11bf95293d303128f21d37290f84e7a Sep 17 01:26:48.669180 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:48.669152 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-559ccbc595-79zsl"] Sep 17 01:26:48.675137 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:48.673803 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6445df8557-7hkg9"] Sep 17 01:26:48.678988 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:26:48.678952 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e8bdcb_05af_4b85_99cb_bf9de2cc2901.slice/crio-d08584658595c33916be97c0ea48496e059d3dacc5c38a2d831233de2f56557f WatchSource:0}: Error finding container d08584658595c33916be97c0ea48496e059d3dacc5c38a2d831233de2f56557f: Status 404 returned error can't find the container with id d08584658595c33916be97c0ea48496e059d3dacc5c38a2d831233de2f56557f Sep 17 01:26:49.123113 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:49.123072 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" event={"ID":"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901","Type":"ContainerStarted","Data":"d08584658595c33916be97c0ea48496e059d3dacc5c38a2d831233de2f56557f"} Sep 17 01:26:49.125704 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:49.125668 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-lwhdx" event={"ID":"51861ecb-5643-4b47-9c98-c083d6abec0d","Type":"ContainerStarted","Data":"4ba799505518c0c306fb4bd69628aace6af76b36b0780853b3285031d10acdfe"} Sep 17 01:26:49.126290 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:49.126160 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-lwhdx" Sep 17 01:26:49.128481 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:49.128436 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" event={"ID":"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49","Type":"ContainerStarted","Data":"3a5c6b005291d3898974ea54d60b75f899c198a9b383b09a9a7e2802d65117ae"} Sep 17 01:26:49.130387 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:49.130359 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c55c57948-kwtrf" event={"ID":"ca726db6-ff92-4f48-8a13-552c196c709f","Type":"ContainerStarted","Data":"1746cdc98f0fdf117f89583d746b3f84fbb217e9c13af191af424e06d7590e92"} Sep 17 01:26:49.130493 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:49.130405 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c55c57948-kwtrf" event={"ID":"ca726db6-ff92-4f48-8a13-552c196c709f","Type":"ContainerStarted","Data":"87563a65b948d02a22efb73c6f780f1dc11bf95293d303128f21d37290f84e7a"} Sep 17 01:26:49.131845 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:49.131807 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2b71772-281b-438d-a110-36d2050aac21","Type":"ContainerStarted","Data":"0b90ac2e3cf2ee9d4609550cc7b1da05c0bd3e74b4d94875dcbf3a2ac1041eea"} Sep 17 01:26:49.142366 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:49.142318 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-lwhdx" Sep 17 01:26:49.142496 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:49.142448 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-lwhdx" podStartSLOduration=2.245194689 podStartE2EDuration="19.14243076s" totalImagesPullingTime="16.897236071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:49.140301502 +0000 UTC m=+274.375448691" watchObservedRunningTime="2026-09-17 01:26:49.14243076 +0000 UTC m=+274.377577949" Sep 17 01:26:49.156632 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:49.156585 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c55c57948-kwtrf" podStartSLOduration=2.156568677 podStartE2EDuration="2.156568677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:49.154866267 +0000 UTC m=+274.390013468" watchObservedRunningTime="2026-09-17 01:26:49.156568677 +0000 UTC m=+274.391715867" Sep 17 01:26:50.141225 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:50.140500 2577 generic.go:356] "Generic (PLEG): container finished" podID="b2b71772-281b-438d-a110-36d2050aac21" containerID="022c1b328e2b5391fe82e8b5305c08a0dbeb9a036fe5ce2f74bfc218b9fdf3a1" exitCode=0 Sep 17 01:26:50.141225 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:50.140693 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2b71772-281b-438d-a110-36d2050aac21","Type":"ContainerDied","Data":"022c1b328e2b5391fe82e8b5305c08a0dbeb9a036fe5ce2f74bfc218b9fdf3a1"} Sep 17 01:26:53.162246 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:53.162024 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" event={"ID":"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901","Type":"ContainerStarted","Data":"2c4deaa3760b2c185561126e0b3b24f19ff28edc32a5bcf5b3d0e87282a7f6af"} Sep 17 01:26:53.162246 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:53.162129 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" event={"ID":"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901","Type":"ContainerStarted","Data":"d850bfca000f911acd21131e4a87f59063beb89cb2a83f2acc3f8d7fec95eeb0"} Sep 17 01:26:53.162246 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:53.162162 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" event={"ID":"a2e8bdcb-05af-4b85-99cb-bf9de2cc2901","Type":"ContainerStarted","Data":"07535530cf5a2c6f7ae65616e536b611f558a40e741b12a88ffb2aebf9bb64d7"} Sep 17 01:26:53.164477 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:53.164447 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" event={"ID":"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49","Type":"ContainerStarted","Data":"b637452759a6daffe40c3defc9130e026faff5e4dbf5a0b959f9686b9d703f11"} Sep 17 01:26:53.164603 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:53.164517 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" event={"ID":"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49","Type":"ContainerStarted","Data":"b42a5f6988e115a12cddc32a12b6dff01dfe6f2541a904567c81673c10d8650c"} Sep 17 01:26:53.164603 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:53.164560 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" event={"ID":"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49","Type":"ContainerStarted","Data":"0fd9a008ee6adfb8c0a5427a91b6eb7cefcb131dbb76e4f3f6fd5bb32f9d0c3e"} Sep 17 01:26:53.182041 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:53.181982 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6445df8557-7hkg9" podStartSLOduration=6.788995438 podStartE2EDuration="10.181967111s" totalImagesPullingTime="3.392971673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:53.179648267 +0000 UTC m=+278.414795455" watchObservedRunningTime="2026-09-17 01:26:53.181967111 +0000 UTC m=+278.417114298" Sep 17 01:26:54.174027 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.173981 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2b71772-281b-438d-a110-36d2050aac21","Type":"ContainerStarted","Data":"6147701cf06e3878b04c832b49b88816f6fd5c5c5845e6f0ad907943215f47fb"} Sep 17 01:26:54.174517 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.174079 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2b71772-281b-438d-a110-36d2050aac21","Type":"ContainerStarted","Data":"b422fb9d8d6d446a55b6dde826a8a3c1768ac92ec943ea35ef633d4329a910d3"} Sep 17 01:26:54.174517 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.174115 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2b71772-281b-438d-a110-36d2050aac21","Type":"ContainerStarted","Data":"b56f15a8bea7391aa42d541083912d5aeb155e57425fc938e338ecc16747ff85"} Sep 17 01:26:54.174517 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.174154 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2b71772-281b-438d-a110-36d2050aac21","Type":"ContainerStarted","Data":"1ac3657a12a54c08f96212606de2541ff8f6ac49c3f98bce9f122f40e00b7f70"} Sep 17 01:26:54.174517 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.174193 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2b71772-281b-438d-a110-36d2050aac21","Type":"ContainerStarted","Data":"954d33032a2f574eb83a5ec19df7406d1690e7f468521eaa83fe6783eaf208f3"} Sep 17 01:26:54.283405 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.283366 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c55c57948-kwtrf"] Sep 17 01:26:54.307604 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.307576 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fc6fdf468-czlvg"] Sep 17 01:26:54.327622 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.327588 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fc6fdf468-czlvg"] Sep 17 01:26:54.327780 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.327693 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.358215 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.358186 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2535db49-7b69-4f01-82b3-735111df9bfe-console-oauth-config\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.358367 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.358220 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-trusted-ca-bundle\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.358367 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.358244 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2535db49-7b69-4f01-82b3-735111df9bfe-console-serving-cert\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.358367 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.358281 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-console-config\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.358367 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.358303 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndjhj\" (UniqueName: \"kubernetes.io/projected/2535db49-7b69-4f01-82b3-735111df9bfe-kube-api-access-ndjhj\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.358589 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.358385 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-oauth-serving-cert\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.358589 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.358492 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-service-ca\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.459794 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.459766 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2535db49-7b69-4f01-82b3-735111df9bfe-console-oauth-config\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.459977 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.459800 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-trusted-ca-bundle\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.459977 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.459827 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2535db49-7b69-4f01-82b3-735111df9bfe-console-serving-cert\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.460107 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.460002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-console-config\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.460107 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.460038 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndjhj\" (UniqueName: \"kubernetes.io/projected/2535db49-7b69-4f01-82b3-735111df9bfe-kube-api-access-ndjhj\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.460107 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.460081 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-oauth-serving-cert\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.460257 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.460135 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-service-ca\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.460825 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.460792 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-service-ca\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.460959 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.460792 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-trusted-ca-bundle\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.460959 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.460798 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-console-config\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.461145 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.461113 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-oauth-serving-cert\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.462595 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.462571 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2535db49-7b69-4f01-82b3-735111df9bfe-console-serving-cert\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.463395 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.463371 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2535db49-7b69-4f01-82b3-735111df9bfe-console-oauth-config\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.467489 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.467467 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndjhj\" (UniqueName: \"kubernetes.io/projected/2535db49-7b69-4f01-82b3-735111df9bfe-kube-api-access-ndjhj\") pod \"console-5fc6fdf468-czlvg\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.639899 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.639858 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:26:54.935030 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:54.934954 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fc6fdf468-czlvg"] Sep 17 01:26:54.937418 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:26:54.937387 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2535db49_7b69_4f01_82b3_735111df9bfe.slice/crio-9b479421066d3760291496fa22835e378b728e8237fcac881718afd417cb5487 WatchSource:0}: Error finding container 9b479421066d3760291496fa22835e378b728e8237fcac881718afd417cb5487: Status 404 returned error can't find the container with id 9b479421066d3760291496fa22835e378b728e8237fcac881718afd417cb5487 Sep 17 01:26:55.181315 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:55.181272 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fc6fdf468-czlvg" event={"ID":"2535db49-7b69-4f01-82b3-735111df9bfe","Type":"ContainerStarted","Data":"3cf885c4eaef98dd1ac56c74d36017a910143ed96346a08497cddba6bb11f277"} Sep 17 01:26:55.181763 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:55.181334 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fc6fdf468-czlvg" event={"ID":"2535db49-7b69-4f01-82b3-735111df9bfe","Type":"ContainerStarted","Data":"9b479421066d3760291496fa22835e378b728e8237fcac881718afd417cb5487"} Sep 17 01:26:55.185187 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:55.185149 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" event={"ID":"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49","Type":"ContainerStarted","Data":"42876f6e2869486ced5699adb469565f0910bedbbdd8923f9997ee7228d097d1"} Sep 17 01:26:55.185326 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:55.185221 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" event={"ID":"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49","Type":"ContainerStarted","Data":"74e60496e5d9c19d39246cd1dd24a13fccedb5fd4f9aa2bfc6fde3d1eae53994"} Sep 17 01:26:55.185326 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:55.185264 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" event={"ID":"bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49","Type":"ContainerStarted","Data":"ea8b5ccb6778127dd6a944f8b57c57fdc988b2d3e2d637ca08772df4fbe02f65"} Sep 17 01:26:55.185326 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:55.185306 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:26:55.193176 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:55.193147 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2b71772-281b-438d-a110-36d2050aac21","Type":"ContainerStarted","Data":"17fe6125294cab566f19948756a9aa3b156a3bb15c2c87492defd57533a828fd"} Sep 17 01:26:55.197192 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:55.197145 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fc6fdf468-czlvg" podStartSLOduration=1.197127291 podStartE2EDuration="1.197127291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:55.196570953 +0000 UTC m=+280.431718141" watchObservedRunningTime="2026-09-17 01:26:55.197127291 +0000 UTC m=+280.432274479" Sep 17 01:26:55.216756 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:55.216670 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" podStartSLOduration=9.272021233 podStartE2EDuration="14.216654605s" totalImagesPullingTime="4.944633372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:55.215240923 +0000 UTC m=+280.450388110" watchObservedRunningTime="2026-09-17 01:26:55.216654605 +0000 UTC m=+280.451801792" Sep 17 01:26:55.241075 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:55.241022 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=11.350805054 podStartE2EDuration="16.241001066s" totalImagesPullingTime="4.87162279s" totalInitContainerRuntime="18.573222ms" isStatefulPod=false podCreationTimestamp="2026-09-17 01:26:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:26:55.238814201 +0000 UTC m=+280.473961400" watchObservedRunningTime="2026-09-17 01:26:55.241001066 +0000 UTC m=+280.476148256" Sep 17 01:26:55.746858 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:26:55.746815 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-m4hzm" podUID="942e7bc8-54bc-4e0f-939b-814164a4ed3d" Sep 17 01:26:56.197619 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:56.197583 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m4hzm" Sep 17 01:26:57.411132 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:57.411097 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:26:59.008298 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:59.008257 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:26:59.010530 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:59.010499 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/942e7bc8-54bc-4e0f-939b-814164a4ed3d-metrics-tls\") pod \"dns-default-m4hzm\" (UID: \"942e7bc8-54bc-4e0f-939b-814164a4ed3d\") " pod="openshift-dns/dns-default-m4hzm" Sep 17 01:26:59.200705 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:59.200669 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sdvq5\"" Sep 17 01:26:59.208764 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:59.208737 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m4hzm" Sep 17 01:26:59.506944 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:26:59.506899 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m4hzm"] Sep 17 01:26:59.509366 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:26:59.509341 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942e7bc8_54bc_4e0f_939b_814164a4ed3d.slice/crio-6fbda9c5e4517406739e69bdadc3ca3a3d77294b873b8d934bad9f733a08d1a6 WatchSource:0}: Error finding container 6fbda9c5e4517406739e69bdadc3ca3a3d77294b873b8d934bad9f733a08d1a6: Status 404 returned error can't find the container with id 6fbda9c5e4517406739e69bdadc3ca3a3d77294b873b8d934bad9f733a08d1a6 Sep 17 01:27:00.216915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:00.216877 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m4hzm" event={"ID":"942e7bc8-54bc-4e0f-939b-814164a4ed3d","Type":"ContainerStarted","Data":"6fbda9c5e4517406739e69bdadc3ca3a3d77294b873b8d934bad9f733a08d1a6"} Sep 17 01:27:01.203250 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:01.203223 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-559ccbc595-79zsl" Sep 17 01:27:03.229694 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:03.229653 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m4hzm" event={"ID":"942e7bc8-54bc-4e0f-939b-814164a4ed3d","Type":"ContainerStarted","Data":"f59a55ff5a1ea718f2c568832d8ed7d362c36a91a3b4206f550393fd0f7e3e84"} Sep 17 01:27:03.230126 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:03.229715 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m4hzm" event={"ID":"942e7bc8-54bc-4e0f-939b-814164a4ed3d","Type":"ContainerStarted","Data":"dd2eed7efd191b7849d2cd90fef47e6fe8a37dcf9fa53d0e69dbd6ff22517d92"} Sep 17 01:27:03.230126 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:03.229807 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-m4hzm" Sep 17 01:27:03.245604 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:03.245561 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-m4hzm" podStartSLOduration=251.350205043 podStartE2EDuration="4m14.245544859s" totalImagesPullingTime="2.895339816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:27:03.243648837 +0000 UTC m=+288.478796025" watchObservedRunningTime="2026-09-17 01:27:03.245544859 +0000 UTC m=+288.480692047" Sep 17 01:27:04.640152 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:04.640119 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:27:04.640513 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:04.640170 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:27:04.645017 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:04.644994 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:27:05.241278 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:05.241250 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:27:05.283067 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:05.283035 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-644fc4dcdd-kgghn"] Sep 17 01:27:13.236387 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:13.236355 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-m4hzm" Sep 17 01:27:15.093577 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:15.093547 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 01:27:15.094700 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:15.094663 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 01:27:15.107215 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:15.107196 2577 kubelet.go:1648] "Image garbage collection succeeded" Sep 17 01:27:19.308235 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.308170 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c55c57948-kwtrf" podUID="ca726db6-ff92-4f48-8a13-552c196c709f" containerName="console" containerID="cri-o://1746cdc98f0fdf117f89583d746b3f84fbb217e9c13af191af424e06d7590e92" gracePeriod=15 Sep 17 01:27:19.578877 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.578817 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:27:19.688859 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.688822 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-oauth-serving-cert\") pod \"ca726db6-ff92-4f48-8a13-552c196c709f\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " Sep 17 01:27:19.688859 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.688863 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwx9b\" (UniqueName: \"kubernetes.io/projected/ca726db6-ff92-4f48-8a13-552c196c709f-kube-api-access-wwx9b\") pod \"ca726db6-ff92-4f48-8a13-552c196c709f\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " Sep 17 01:27:19.689105 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.688886 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-trusted-ca-bundle\") pod \"ca726db6-ff92-4f48-8a13-552c196c709f\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " Sep 17 01:27:19.689105 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.688905 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-service-ca\") pod \"ca726db6-ff92-4f48-8a13-552c196c709f\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " Sep 17 01:27:19.689105 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.688960 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-console-config\") pod \"ca726db6-ff92-4f48-8a13-552c196c709f\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " Sep 17 01:27:19.689105 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.689005 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca726db6-ff92-4f48-8a13-552c196c709f-console-oauth-config\") pod \"ca726db6-ff92-4f48-8a13-552c196c709f\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " Sep 17 01:27:19.689105 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.689034 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca726db6-ff92-4f48-8a13-552c196c709f-console-serving-cert\") pod \"ca726db6-ff92-4f48-8a13-552c196c709f\" (UID: \"ca726db6-ff92-4f48-8a13-552c196c709f\") " Sep 17 01:27:19.689387 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.689355 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-service-ca" (OuterVolumeSpecName: "service-ca") pod "ca726db6-ff92-4f48-8a13-552c196c709f" (UID: "ca726db6-ff92-4f48-8a13-552c196c709f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:27:19.689448 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.689365 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ca726db6-ff92-4f48-8a13-552c196c709f" (UID: "ca726db6-ff92-4f48-8a13-552c196c709f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:27:19.689494 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.689442 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-console-config" (OuterVolumeSpecName: "console-config") pod "ca726db6-ff92-4f48-8a13-552c196c709f" (UID: "ca726db6-ff92-4f48-8a13-552c196c709f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:27:19.689694 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.689664 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ca726db6-ff92-4f48-8a13-552c196c709f" (UID: "ca726db6-ff92-4f48-8a13-552c196c709f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:27:19.691252 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.691219 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca726db6-ff92-4f48-8a13-552c196c709f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ca726db6-ff92-4f48-8a13-552c196c709f" (UID: "ca726db6-ff92-4f48-8a13-552c196c709f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:27:19.691343 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.691286 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca726db6-ff92-4f48-8a13-552c196c709f-kube-api-access-wwx9b" (OuterVolumeSpecName: "kube-api-access-wwx9b") pod "ca726db6-ff92-4f48-8a13-552c196c709f" (UID: "ca726db6-ff92-4f48-8a13-552c196c709f"). InnerVolumeSpecName "kube-api-access-wwx9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:27:19.691343 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.691289 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca726db6-ff92-4f48-8a13-552c196c709f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ca726db6-ff92-4f48-8a13-552c196c709f" (UID: "ca726db6-ff92-4f48-8a13-552c196c709f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:27:19.790158 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.790120 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca726db6-ff92-4f48-8a13-552c196c709f-console-oauth-config\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:27:19.790158 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.790151 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca726db6-ff92-4f48-8a13-552c196c709f-console-serving-cert\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:27:19.790158 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.790163 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-oauth-serving-cert\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:27:19.790382 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.790177 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwx9b\" (UniqueName: \"kubernetes.io/projected/ca726db6-ff92-4f48-8a13-552c196c709f-kube-api-access-wwx9b\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:27:19.790382 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.790189 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-trusted-ca-bundle\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:27:19.790382 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.790202 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-service-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:27:19.790382 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:19.790212 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca726db6-ff92-4f48-8a13-552c196c709f-console-config\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:27:20.298057 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:20.298023 2577 generic.go:356] "Generic (PLEG): container finished" podID="ca726db6-ff92-4f48-8a13-552c196c709f" containerID="1746cdc98f0fdf117f89583d746b3f84fbb217e9c13af191af424e06d7590e92" exitCode=0 Sep 17 01:27:20.298204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:20.298065 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c55c57948-kwtrf" Sep 17 01:27:20.298204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:20.298094 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c55c57948-kwtrf" event={"ID":"ca726db6-ff92-4f48-8a13-552c196c709f","Type":"ContainerDied","Data":"1746cdc98f0fdf117f89583d746b3f84fbb217e9c13af191af424e06d7590e92"} Sep 17 01:27:20.298204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:20.298131 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c55c57948-kwtrf" event={"ID":"ca726db6-ff92-4f48-8a13-552c196c709f","Type":"ContainerDied","Data":"87563a65b948d02a22efb73c6f780f1dc11bf95293d303128f21d37290f84e7a"} Sep 17 01:27:20.298204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:20.298150 2577 scope.go:117] "RemoveContainer" containerID="1746cdc98f0fdf117f89583d746b3f84fbb217e9c13af191af424e06d7590e92" Sep 17 01:27:20.306693 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:20.306677 2577 scope.go:117] "RemoveContainer" containerID="1746cdc98f0fdf117f89583d746b3f84fbb217e9c13af191af424e06d7590e92" Sep 17 01:27:20.306944 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:27:20.306913 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1746cdc98f0fdf117f89583d746b3f84fbb217e9c13af191af424e06d7590e92\": container with ID starting with 1746cdc98f0fdf117f89583d746b3f84fbb217e9c13af191af424e06d7590e92 not found: ID does not exist" containerID="1746cdc98f0fdf117f89583d746b3f84fbb217e9c13af191af424e06d7590e92" Sep 17 01:27:20.307028 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:20.306965 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1746cdc98f0fdf117f89583d746b3f84fbb217e9c13af191af424e06d7590e92"} err="failed to get container status \"1746cdc98f0fdf117f89583d746b3f84fbb217e9c13af191af424e06d7590e92\": rpc error: code = NotFound desc = could not find container \"1746cdc98f0fdf117f89583d746b3f84fbb217e9c13af191af424e06d7590e92\": container with ID starting with 1746cdc98f0fdf117f89583d746b3f84fbb217e9c13af191af424e06d7590e92 not found: ID does not exist" Sep 17 01:27:20.316961 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:20.316923 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c55c57948-kwtrf"] Sep 17 01:27:20.322476 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:20.322455 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c55c57948-kwtrf"] Sep 17 01:27:21.211142 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:21.211109 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca726db6-ff92-4f48-8a13-552c196c709f" path="/var/lib/kubelet/pods/ca726db6-ff92-4f48-8a13-552c196c709f/volumes" Sep 17 01:27:29.334707 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:29.334671 2577 generic.go:356] "Generic (PLEG): container finished" podID="24dd28d0-d328-41a2-83bc-d392d123272b" containerID="9eea35f8742d2bc6cc1fcd18f07d33cab59e3b9ca0bd8a441b51eb5b4d403aea" exitCode=0 Sep 17 01:27:29.335138 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:29.334743 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gtwjd" event={"ID":"24dd28d0-d328-41a2-83bc-d392d123272b","Type":"ContainerDied","Data":"9eea35f8742d2bc6cc1fcd18f07d33cab59e3b9ca0bd8a441b51eb5b4d403aea"} Sep 17 01:27:29.335138 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:29.335052 2577 scope.go:117] "RemoveContainer" containerID="9eea35f8742d2bc6cc1fcd18f07d33cab59e3b9ca0bd8a441b51eb5b4d403aea" Sep 17 01:27:30.302990 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.302950 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-644fc4dcdd-kgghn" podUID="930470db-7a0f-4d96-b72e-04833c1f12d8" containerName="console" containerID="cri-o://e7967d20cc56f0e4ca539f6a67f38a3af41831fb35be7de91f3e8e75d5b0d94f" gracePeriod=15 Sep 17 01:27:30.345475 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.345444 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-gtwjd" event={"ID":"24dd28d0-d328-41a2-83bc-d392d123272b","Type":"ContainerStarted","Data":"9de506b3d30176b8b525ba77493847371ce1870a5c73b956185767b205557d77"} Sep 17 01:27:30.572544 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.572471 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:27:30.683519 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.683483 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/930470db-7a0f-4d96-b72e-04833c1f12d8-console-oauth-config\") pod \"930470db-7a0f-4d96-b72e-04833c1f12d8\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " Sep 17 01:27:30.683690 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.683551 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nclsc\" (UniqueName: \"kubernetes.io/projected/930470db-7a0f-4d96-b72e-04833c1f12d8-kube-api-access-nclsc\") pod \"930470db-7a0f-4d96-b72e-04833c1f12d8\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " Sep 17 01:27:30.683690 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.683578 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/930470db-7a0f-4d96-b72e-04833c1f12d8-console-config\") pod \"930470db-7a0f-4d96-b72e-04833c1f12d8\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " Sep 17 01:27:30.683690 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.683616 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/930470db-7a0f-4d96-b72e-04833c1f12d8-oauth-serving-cert\") pod \"930470db-7a0f-4d96-b72e-04833c1f12d8\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " Sep 17 01:27:30.683690 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.683634 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/930470db-7a0f-4d96-b72e-04833c1f12d8-service-ca\") pod \"930470db-7a0f-4d96-b72e-04833c1f12d8\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " Sep 17 01:27:30.683690 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.683669 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/930470db-7a0f-4d96-b72e-04833c1f12d8-console-serving-cert\") pod \"930470db-7a0f-4d96-b72e-04833c1f12d8\" (UID: \"930470db-7a0f-4d96-b72e-04833c1f12d8\") " Sep 17 01:27:30.684104 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.684071 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930470db-7a0f-4d96-b72e-04833c1f12d8-console-config" (OuterVolumeSpecName: "console-config") pod "930470db-7a0f-4d96-b72e-04833c1f12d8" (UID: "930470db-7a0f-4d96-b72e-04833c1f12d8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:27:30.684229 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.684080 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930470db-7a0f-4d96-b72e-04833c1f12d8-service-ca" (OuterVolumeSpecName: "service-ca") pod "930470db-7a0f-4d96-b72e-04833c1f12d8" (UID: "930470db-7a0f-4d96-b72e-04833c1f12d8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:27:30.684343 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.684306 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930470db-7a0f-4d96-b72e-04833c1f12d8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "930470db-7a0f-4d96-b72e-04833c1f12d8" (UID: "930470db-7a0f-4d96-b72e-04833c1f12d8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:27:30.685798 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.685770 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/930470db-7a0f-4d96-b72e-04833c1f12d8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "930470db-7a0f-4d96-b72e-04833c1f12d8" (UID: "930470db-7a0f-4d96-b72e-04833c1f12d8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:27:30.685881 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.685835 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930470db-7a0f-4d96-b72e-04833c1f12d8-kube-api-access-nclsc" (OuterVolumeSpecName: "kube-api-access-nclsc") pod "930470db-7a0f-4d96-b72e-04833c1f12d8" (UID: "930470db-7a0f-4d96-b72e-04833c1f12d8"). InnerVolumeSpecName "kube-api-access-nclsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:27:30.685881 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.685870 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/930470db-7a0f-4d96-b72e-04833c1f12d8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "930470db-7a0f-4d96-b72e-04833c1f12d8" (UID: "930470db-7a0f-4d96-b72e-04833c1f12d8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:27:30.784615 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.784586 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nclsc\" (UniqueName: \"kubernetes.io/projected/930470db-7a0f-4d96-b72e-04833c1f12d8-kube-api-access-nclsc\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:27:30.784615 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.784611 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/930470db-7a0f-4d96-b72e-04833c1f12d8-console-config\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:27:30.784615 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.784621 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/930470db-7a0f-4d96-b72e-04833c1f12d8-oauth-serving-cert\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:27:30.784869 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.784629 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/930470db-7a0f-4d96-b72e-04833c1f12d8-service-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:27:30.784869 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.784638 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/930470db-7a0f-4d96-b72e-04833c1f12d8-console-serving-cert\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:27:30.784869 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:30.784646 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/930470db-7a0f-4d96-b72e-04833c1f12d8-console-oauth-config\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:27:31.350622 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:31.350585 2577 generic.go:356] "Generic (PLEG): container finished" podID="930470db-7a0f-4d96-b72e-04833c1f12d8" containerID="e7967d20cc56f0e4ca539f6a67f38a3af41831fb35be7de91f3e8e75d5b0d94f" exitCode=0 Sep 17 01:27:31.350622 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:31.350626 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-644fc4dcdd-kgghn" Sep 17 01:27:31.351103 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:31.350626 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644fc4dcdd-kgghn" event={"ID":"930470db-7a0f-4d96-b72e-04833c1f12d8","Type":"ContainerDied","Data":"e7967d20cc56f0e4ca539f6a67f38a3af41831fb35be7de91f3e8e75d5b0d94f"} Sep 17 01:27:31.351103 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:31.350754 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-644fc4dcdd-kgghn" event={"ID":"930470db-7a0f-4d96-b72e-04833c1f12d8","Type":"ContainerDied","Data":"bde56d98c3743b51710d4090ff4aa4c73c45c16f15ce8fb9f5325025e1197ed5"} Sep 17 01:27:31.351103 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:31.350780 2577 scope.go:117] "RemoveContainer" containerID="e7967d20cc56f0e4ca539f6a67f38a3af41831fb35be7de91f3e8e75d5b0d94f" Sep 17 01:27:31.358799 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:31.358779 2577 scope.go:117] "RemoveContainer" containerID="e7967d20cc56f0e4ca539f6a67f38a3af41831fb35be7de91f3e8e75d5b0d94f" Sep 17 01:27:31.359093 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:27:31.359075 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7967d20cc56f0e4ca539f6a67f38a3af41831fb35be7de91f3e8e75d5b0d94f\": container with ID starting with e7967d20cc56f0e4ca539f6a67f38a3af41831fb35be7de91f3e8e75d5b0d94f not found: ID does not exist" containerID="e7967d20cc56f0e4ca539f6a67f38a3af41831fb35be7de91f3e8e75d5b0d94f" Sep 17 01:27:31.359147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:31.359101 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7967d20cc56f0e4ca539f6a67f38a3af41831fb35be7de91f3e8e75d5b0d94f"} err="failed to get container status \"e7967d20cc56f0e4ca539f6a67f38a3af41831fb35be7de91f3e8e75d5b0d94f\": rpc error: code = NotFound desc = could not find container \"e7967d20cc56f0e4ca539f6a67f38a3af41831fb35be7de91f3e8e75d5b0d94f\": container with ID starting with e7967d20cc56f0e4ca539f6a67f38a3af41831fb35be7de91f3e8e75d5b0d94f not found: ID does not exist" Sep 17 01:27:31.364489 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:31.364469 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-644fc4dcdd-kgghn"] Sep 17 01:27:31.369778 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:31.369759 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-644fc4dcdd-kgghn"] Sep 17 01:27:33.210412 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:33.210377 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="930470db-7a0f-4d96-b72e-04833c1f12d8" path="/var/lib/kubelet/pods/930470db-7a0f-4d96-b72e-04833c1f12d8/volumes" Sep 17 01:27:58.623423 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:58.623381 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 01:27:58.624046 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:58.623763 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="alertmanager" containerID="cri-o://954d33032a2f574eb83a5ec19df7406d1690e7f468521eaa83fe6783eaf208f3" gracePeriod=120 Sep 17 01:27:58.624046 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:58.623800 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="kube-rbac-proxy-metric" containerID="cri-o://6147701cf06e3878b04c832b49b88816f6fd5c5c5845e6f0ad907943215f47fb" gracePeriod=120 Sep 17 01:27:58.624046 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:58.623823 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="kube-rbac-proxy-web" containerID="cri-o://b56f15a8bea7391aa42d541083912d5aeb155e57425fc938e338ecc16747ff85" gracePeriod=120 Sep 17 01:27:58.624046 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:58.623863 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="prom-label-proxy" containerID="cri-o://17fe6125294cab566f19948756a9aa3b156a3bb15c2c87492defd57533a828fd" gracePeriod=120 Sep 17 01:27:58.624046 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:58.623890 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="kube-rbac-proxy" containerID="cri-o://b422fb9d8d6d446a55b6dde826a8a3c1768ac92ec943ea35ef633d4329a910d3" gracePeriod=120 Sep 17 01:27:58.624046 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:58.623896 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="config-reloader" containerID="cri-o://1ac3657a12a54c08f96212606de2541ff8f6ac49c3f98bce9f122f40e00b7f70" gracePeriod=120 Sep 17 01:27:59.468825 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.468793 2577 generic.go:356] "Generic (PLEG): container finished" podID="b2b71772-281b-438d-a110-36d2050aac21" containerID="17fe6125294cab566f19948756a9aa3b156a3bb15c2c87492defd57533a828fd" exitCode=0 Sep 17 01:27:59.468825 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.468819 2577 generic.go:356] "Generic (PLEG): container finished" podID="b2b71772-281b-438d-a110-36d2050aac21" containerID="6147701cf06e3878b04c832b49b88816f6fd5c5c5845e6f0ad907943215f47fb" exitCode=0 Sep 17 01:27:59.468825 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.468826 2577 generic.go:356] "Generic (PLEG): container finished" podID="b2b71772-281b-438d-a110-36d2050aac21" containerID="b422fb9d8d6d446a55b6dde826a8a3c1768ac92ec943ea35ef633d4329a910d3" exitCode=0 Sep 17 01:27:59.468825 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.468832 2577 generic.go:356] "Generic (PLEG): container finished" podID="b2b71772-281b-438d-a110-36d2050aac21" containerID="1ac3657a12a54c08f96212606de2541ff8f6ac49c3f98bce9f122f40e00b7f70" exitCode=0 Sep 17 01:27:59.468825 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.468837 2577 generic.go:356] "Generic (PLEG): container finished" podID="b2b71772-281b-438d-a110-36d2050aac21" containerID="954d33032a2f574eb83a5ec19df7406d1690e7f468521eaa83fe6783eaf208f3" exitCode=0 Sep 17 01:27:59.469126 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.468869 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2b71772-281b-438d-a110-36d2050aac21","Type":"ContainerDied","Data":"17fe6125294cab566f19948756a9aa3b156a3bb15c2c87492defd57533a828fd"} Sep 17 01:27:59.469126 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.468973 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2b71772-281b-438d-a110-36d2050aac21","Type":"ContainerDied","Data":"6147701cf06e3878b04c832b49b88816f6fd5c5c5845e6f0ad907943215f47fb"} Sep 17 01:27:59.469126 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.469011 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2b71772-281b-438d-a110-36d2050aac21","Type":"ContainerDied","Data":"b422fb9d8d6d446a55b6dde826a8a3c1768ac92ec943ea35ef633d4329a910d3"} Sep 17 01:27:59.469126 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.469036 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2b71772-281b-438d-a110-36d2050aac21","Type":"ContainerDied","Data":"1ac3657a12a54c08f96212606de2541ff8f6ac49c3f98bce9f122f40e00b7f70"} Sep 17 01:27:59.469126 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.469060 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2b71772-281b-438d-a110-36d2050aac21","Type":"ContainerDied","Data":"954d33032a2f574eb83a5ec19df7406d1690e7f468521eaa83fe6783eaf208f3"} Sep 17 01:27:59.861626 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.861602 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:27:59.920671 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.920629 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2b71772-281b-438d-a110-36d2050aac21-metrics-client-ca\") pod \"b2b71772-281b-438d-a110-36d2050aac21\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " Sep 17 01:27:59.920843 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.920686 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b2b71772-281b-438d-a110-36d2050aac21-tls-assets\") pod \"b2b71772-281b-438d-a110-36d2050aac21\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " Sep 17 01:27:59.920843 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.920723 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-config-volume\") pod \"b2b71772-281b-438d-a110-36d2050aac21\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " Sep 17 01:27:59.920843 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.920754 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-kube-rbac-proxy\") pod \"b2b71772-281b-438d-a110-36d2050aac21\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " Sep 17 01:27:59.920843 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.920784 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-web-config\") pod \"b2b71772-281b-438d-a110-36d2050aac21\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " Sep 17 01:27:59.920843 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.920827 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2b71772-281b-438d-a110-36d2050aac21-alertmanager-trusted-ca-bundle\") pod \"b2b71772-281b-438d-a110-36d2050aac21\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " Sep 17 01:27:59.921175 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.920866 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b2b71772-281b-438d-a110-36d2050aac21\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " Sep 17 01:27:59.921175 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.920952 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt9dp\" (UniqueName: \"kubernetes.io/projected/b2b71772-281b-438d-a110-36d2050aac21-kube-api-access-kt9dp\") pod \"b2b71772-281b-438d-a110-36d2050aac21\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " Sep 17 01:27:59.921175 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.920982 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-main-tls\") pod \"b2b71772-281b-438d-a110-36d2050aac21\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " Sep 17 01:27:59.921175 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.921019 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2b71772-281b-438d-a110-36d2050aac21-config-out\") pod \"b2b71772-281b-438d-a110-36d2050aac21\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " Sep 17 01:27:59.921175 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.921052 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-kube-rbac-proxy-web\") pod \"b2b71772-281b-438d-a110-36d2050aac21\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " Sep 17 01:27:59.921175 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.921085 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b2b71772-281b-438d-a110-36d2050aac21-alertmanager-main-db\") pod \"b2b71772-281b-438d-a110-36d2050aac21\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " Sep 17 01:27:59.921175 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.921092 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b71772-281b-438d-a110-36d2050aac21-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "b2b71772-281b-438d-a110-36d2050aac21" (UID: "b2b71772-281b-438d-a110-36d2050aac21"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:27:59.921175 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.921119 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-cluster-tls-config\") pod \"b2b71772-281b-438d-a110-36d2050aac21\" (UID: \"b2b71772-281b-438d-a110-36d2050aac21\") " Sep 17 01:27:59.921585 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.921414 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2b71772-281b-438d-a110-36d2050aac21-metrics-client-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:27:59.921999 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.921966 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b71772-281b-438d-a110-36d2050aac21-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "b2b71772-281b-438d-a110-36d2050aac21" (UID: "b2b71772-281b-438d-a110-36d2050aac21"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:27:59.921999 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.921988 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b71772-281b-438d-a110-36d2050aac21-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "b2b71772-281b-438d-a110-36d2050aac21" (UID: "b2b71772-281b-438d-a110-36d2050aac21"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:27:59.924072 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.924043 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b71772-281b-438d-a110-36d2050aac21-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b2b71772-281b-438d-a110-36d2050aac21" (UID: "b2b71772-281b-438d-a110-36d2050aac21"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:27:59.924420 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.924393 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-config-volume" (OuterVolumeSpecName: "config-volume") pod "b2b71772-281b-438d-a110-36d2050aac21" (UID: "b2b71772-281b-438d-a110-36d2050aac21"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:27:59.924591 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.924554 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "b2b71772-281b-438d-a110-36d2050aac21" (UID: "b2b71772-281b-438d-a110-36d2050aac21"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:27:59.924686 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.924664 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b71772-281b-438d-a110-36d2050aac21-kube-api-access-kt9dp" (OuterVolumeSpecName: "kube-api-access-kt9dp") pod "b2b71772-281b-438d-a110-36d2050aac21" (UID: "b2b71772-281b-438d-a110-36d2050aac21"). InnerVolumeSpecName "kube-api-access-kt9dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:27:59.925543 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.925506 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "b2b71772-281b-438d-a110-36d2050aac21" (UID: "b2b71772-281b-438d-a110-36d2050aac21"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:27:59.925543 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.925524 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "b2b71772-281b-438d-a110-36d2050aac21" (UID: "b2b71772-281b-438d-a110-36d2050aac21"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:27:59.925735 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.925712 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "b2b71772-281b-438d-a110-36d2050aac21" (UID: "b2b71772-281b-438d-a110-36d2050aac21"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:27:59.926029 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.926006 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b71772-281b-438d-a110-36d2050aac21-config-out" (OuterVolumeSpecName: "config-out") pod "b2b71772-281b-438d-a110-36d2050aac21" (UID: "b2b71772-281b-438d-a110-36d2050aac21"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:27:59.929558 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.929533 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "b2b71772-281b-438d-a110-36d2050aac21" (UID: "b2b71772-281b-438d-a110-36d2050aac21"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:27:59.935911 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:27:59.935881 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-web-config" (OuterVolumeSpecName: "web-config") pod "b2b71772-281b-438d-a110-36d2050aac21" (UID: "b2b71772-281b-438d-a110-36d2050aac21"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:28:00.022742 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.022649 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:28:00.022742 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.022692 2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b2b71772-281b-438d-a110-36d2050aac21-alertmanager-main-db\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:28:00.022742 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.022703 2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-cluster-tls-config\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:28:00.022742 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.022712 2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b2b71772-281b-438d-a110-36d2050aac21-tls-assets\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:28:00.022742 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.022720 2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-config-volume\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:28:00.022742 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.022728 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:28:00.022742 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.022738 2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-web-config\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:28:00.022742 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.022747 2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2b71772-281b-438d-a110-36d2050aac21-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:28:00.022742 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.022755 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:28:00.023149 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.022764 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt9dp\" (UniqueName: \"kubernetes.io/projected/b2b71772-281b-438d-a110-36d2050aac21-kube-api-access-kt9dp\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:28:00.023149 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.022773 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b2b71772-281b-438d-a110-36d2050aac21-secret-alertmanager-main-tls\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:28:00.023149 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.022781 2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2b71772-281b-438d-a110-36d2050aac21-config-out\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:28:00.475903 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.475869 2577 generic.go:356] "Generic (PLEG): container finished" podID="b2b71772-281b-438d-a110-36d2050aac21" containerID="b56f15a8bea7391aa42d541083912d5aeb155e57425fc938e338ecc16747ff85" exitCode=0 Sep 17 01:28:00.476079 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.475942 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2b71772-281b-438d-a110-36d2050aac21","Type":"ContainerDied","Data":"b56f15a8bea7391aa42d541083912d5aeb155e57425fc938e338ecc16747ff85"} Sep 17 01:28:00.476079 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.475960 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.476079 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.476010 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2b71772-281b-438d-a110-36d2050aac21","Type":"ContainerDied","Data":"0b90ac2e3cf2ee9d4609550cc7b1da05c0bd3e74b4d94875dcbf3a2ac1041eea"} Sep 17 01:28:00.476079 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.476047 2577 scope.go:117] "RemoveContainer" containerID="17fe6125294cab566f19948756a9aa3b156a3bb15c2c87492defd57533a828fd" Sep 17 01:28:00.484139 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.484119 2577 scope.go:117] "RemoveContainer" containerID="6147701cf06e3878b04c832b49b88816f6fd5c5c5845e6f0ad907943215f47fb" Sep 17 01:28:00.490882 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.490856 2577 scope.go:117] "RemoveContainer" containerID="b422fb9d8d6d446a55b6dde826a8a3c1768ac92ec943ea35ef633d4329a910d3" Sep 17 01:28:00.497975 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.497953 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 01:28:00.498446 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.498427 2577 scope.go:117] "RemoveContainer" containerID="b56f15a8bea7391aa42d541083912d5aeb155e57425fc938e338ecc16747ff85" Sep 17 01:28:00.501063 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.501041 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 01:28:00.505424 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.505409 2577 scope.go:117] "RemoveContainer" containerID="1ac3657a12a54c08f96212606de2541ff8f6ac49c3f98bce9f122f40e00b7f70" Sep 17 01:28:00.512268 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.512250 2577 scope.go:117] "RemoveContainer" containerID="954d33032a2f574eb83a5ec19df7406d1690e7f468521eaa83fe6783eaf208f3" Sep 17 01:28:00.519102 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.519086 2577 scope.go:117] "RemoveContainer" containerID="022c1b328e2b5391fe82e8b5305c08a0dbeb9a036fe5ce2f74bfc218b9fdf3a1" Sep 17 01:28:00.524512 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.524490 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 01:28:00.524919 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.524901 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca726db6-ff92-4f48-8a13-552c196c709f" containerName="console" Sep 17 01:28:00.524994 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.524923 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca726db6-ff92-4f48-8a13-552c196c709f" containerName="console" Sep 17 01:28:00.524994 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.524953 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="kube-rbac-proxy-web" Sep 17 01:28:00.524994 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.524962 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="kube-rbac-proxy-web" Sep 17 01:28:00.524994 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.524984 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="930470db-7a0f-4d96-b72e-04833c1f12d8" containerName="console" Sep 17 01:28:00.524994 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.524992 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="930470db-7a0f-4d96-b72e-04833c1f12d8" containerName="console" Sep 17 01:28:00.525170 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525006 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="prom-label-proxy" Sep 17 01:28:00.525170 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525014 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="prom-label-proxy" Sep 17 01:28:00.525170 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525026 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="kube-rbac-proxy-metric" Sep 17 01:28:00.525170 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525035 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="kube-rbac-proxy-metric" Sep 17 01:28:00.525170 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525048 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="kube-rbac-proxy" Sep 17 01:28:00.525170 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525056 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="kube-rbac-proxy" Sep 17 01:28:00.525170 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525067 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="alertmanager" Sep 17 01:28:00.525170 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525075 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="alertmanager" Sep 17 01:28:00.525170 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525085 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="init-config-reloader" Sep 17 01:28:00.525170 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525093 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="init-config-reloader" Sep 17 01:28:00.525170 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525102 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="config-reloader" Sep 17 01:28:00.525170 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525110 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="config-reloader" Sep 17 01:28:00.525541 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525183 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="prom-label-proxy" Sep 17 01:28:00.525541 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525194 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="kube-rbac-proxy-metric" Sep 17 01:28:00.525541 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525223 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="config-reloader" Sep 17 01:28:00.525541 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525236 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="kube-rbac-proxy-web" Sep 17 01:28:00.525541 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525246 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="930470db-7a0f-4d96-b72e-04833c1f12d8" containerName="console" Sep 17 01:28:00.525541 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525255 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="kube-rbac-proxy" Sep 17 01:28:00.525541 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525265 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca726db6-ff92-4f48-8a13-552c196c709f" containerName="console" Sep 17 01:28:00.525541 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.525272 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2b71772-281b-438d-a110-36d2050aac21" containerName="alertmanager" Sep 17 01:28:00.527084 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.527064 2577 scope.go:117] "RemoveContainer" containerID="17fe6125294cab566f19948756a9aa3b156a3bb15c2c87492defd57533a828fd" Sep 17 01:28:00.527360 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:28:00.527341 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17fe6125294cab566f19948756a9aa3b156a3bb15c2c87492defd57533a828fd\": container with ID starting with 17fe6125294cab566f19948756a9aa3b156a3bb15c2c87492defd57533a828fd not found: ID does not exist" containerID="17fe6125294cab566f19948756a9aa3b156a3bb15c2c87492defd57533a828fd" Sep 17 01:28:00.527417 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.527371 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17fe6125294cab566f19948756a9aa3b156a3bb15c2c87492defd57533a828fd"} err="failed to get container status \"17fe6125294cab566f19948756a9aa3b156a3bb15c2c87492defd57533a828fd\": rpc error: code = NotFound desc = could not find container \"17fe6125294cab566f19948756a9aa3b156a3bb15c2c87492defd57533a828fd\": container with ID starting with 17fe6125294cab566f19948756a9aa3b156a3bb15c2c87492defd57533a828fd not found: ID does not exist" Sep 17 01:28:00.527417 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.527391 2577 scope.go:117] "RemoveContainer" containerID="6147701cf06e3878b04c832b49b88816f6fd5c5c5845e6f0ad907943215f47fb" Sep 17 01:28:00.527614 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:28:00.527597 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6147701cf06e3878b04c832b49b88816f6fd5c5c5845e6f0ad907943215f47fb\": container with ID starting with 6147701cf06e3878b04c832b49b88816f6fd5c5c5845e6f0ad907943215f47fb not found: ID does not exist" containerID="6147701cf06e3878b04c832b49b88816f6fd5c5c5845e6f0ad907943215f47fb" Sep 17 01:28:00.527654 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.527620 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6147701cf06e3878b04c832b49b88816f6fd5c5c5845e6f0ad907943215f47fb"} err="failed to get container status \"6147701cf06e3878b04c832b49b88816f6fd5c5c5845e6f0ad907943215f47fb\": rpc error: code = NotFound desc = could not find container \"6147701cf06e3878b04c832b49b88816f6fd5c5c5845e6f0ad907943215f47fb\": container with ID starting with 6147701cf06e3878b04c832b49b88816f6fd5c5c5845e6f0ad907943215f47fb not found: ID does not exist" Sep 17 01:28:00.527654 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.527634 2577 scope.go:117] "RemoveContainer" containerID="b422fb9d8d6d446a55b6dde826a8a3c1768ac92ec943ea35ef633d4329a910d3" Sep 17 01:28:00.527854 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:28:00.527835 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b422fb9d8d6d446a55b6dde826a8a3c1768ac92ec943ea35ef633d4329a910d3\": container with ID starting with b422fb9d8d6d446a55b6dde826a8a3c1768ac92ec943ea35ef633d4329a910d3 not found: ID does not exist" containerID="b422fb9d8d6d446a55b6dde826a8a3c1768ac92ec943ea35ef633d4329a910d3" Sep 17 01:28:00.527917 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.527864 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b422fb9d8d6d446a55b6dde826a8a3c1768ac92ec943ea35ef633d4329a910d3"} err="failed to get container status \"b422fb9d8d6d446a55b6dde826a8a3c1768ac92ec943ea35ef633d4329a910d3\": rpc error: code = NotFound desc = could not find container \"b422fb9d8d6d446a55b6dde826a8a3c1768ac92ec943ea35ef633d4329a910d3\": container with ID starting with b422fb9d8d6d446a55b6dde826a8a3c1768ac92ec943ea35ef633d4329a910d3 not found: ID does not exist" Sep 17 01:28:00.527917 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.527888 2577 scope.go:117] "RemoveContainer" containerID="b56f15a8bea7391aa42d541083912d5aeb155e57425fc938e338ecc16747ff85" Sep 17 01:28:00.528164 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:28:00.528148 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56f15a8bea7391aa42d541083912d5aeb155e57425fc938e338ecc16747ff85\": container with ID starting with b56f15a8bea7391aa42d541083912d5aeb155e57425fc938e338ecc16747ff85 not found: ID does not exist" containerID="b56f15a8bea7391aa42d541083912d5aeb155e57425fc938e338ecc16747ff85" Sep 17 01:28:00.528204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.528168 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56f15a8bea7391aa42d541083912d5aeb155e57425fc938e338ecc16747ff85"} err="failed to get container status \"b56f15a8bea7391aa42d541083912d5aeb155e57425fc938e338ecc16747ff85\": rpc error: code = NotFound desc = could not find container \"b56f15a8bea7391aa42d541083912d5aeb155e57425fc938e338ecc16747ff85\": container with ID starting with b56f15a8bea7391aa42d541083912d5aeb155e57425fc938e338ecc16747ff85 not found: ID does not exist" Sep 17 01:28:00.528204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.528181 2577 scope.go:117] "RemoveContainer" containerID="1ac3657a12a54c08f96212606de2541ff8f6ac49c3f98bce9f122f40e00b7f70" Sep 17 01:28:00.528434 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:28:00.528416 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac3657a12a54c08f96212606de2541ff8f6ac49c3f98bce9f122f40e00b7f70\": container with ID starting with 1ac3657a12a54c08f96212606de2541ff8f6ac49c3f98bce9f122f40e00b7f70 not found: ID does not exist" containerID="1ac3657a12a54c08f96212606de2541ff8f6ac49c3f98bce9f122f40e00b7f70" Sep 17 01:28:00.528500 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.528440 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac3657a12a54c08f96212606de2541ff8f6ac49c3f98bce9f122f40e00b7f70"} err="failed to get container status \"1ac3657a12a54c08f96212606de2541ff8f6ac49c3f98bce9f122f40e00b7f70\": rpc error: code = NotFound desc = could not find container \"1ac3657a12a54c08f96212606de2541ff8f6ac49c3f98bce9f122f40e00b7f70\": container with ID starting with 1ac3657a12a54c08f96212606de2541ff8f6ac49c3f98bce9f122f40e00b7f70 not found: ID does not exist" Sep 17 01:28:00.528500 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.528476 2577 scope.go:117] "RemoveContainer" containerID="954d33032a2f574eb83a5ec19df7406d1690e7f468521eaa83fe6783eaf208f3" Sep 17 01:28:00.528690 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:28:00.528674 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"954d33032a2f574eb83a5ec19df7406d1690e7f468521eaa83fe6783eaf208f3\": container with ID starting with 954d33032a2f574eb83a5ec19df7406d1690e7f468521eaa83fe6783eaf208f3 not found: ID does not exist" containerID="954d33032a2f574eb83a5ec19df7406d1690e7f468521eaa83fe6783eaf208f3" Sep 17 01:28:00.528728 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.528694 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"954d33032a2f574eb83a5ec19df7406d1690e7f468521eaa83fe6783eaf208f3"} err="failed to get container status \"954d33032a2f574eb83a5ec19df7406d1690e7f468521eaa83fe6783eaf208f3\": rpc error: code = NotFound desc = could not find container \"954d33032a2f574eb83a5ec19df7406d1690e7f468521eaa83fe6783eaf208f3\": container with ID starting with 954d33032a2f574eb83a5ec19df7406d1690e7f468521eaa83fe6783eaf208f3 not found: ID does not exist" Sep 17 01:28:00.528728 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.528707 2577 scope.go:117] "RemoveContainer" containerID="022c1b328e2b5391fe82e8b5305c08a0dbeb9a036fe5ce2f74bfc218b9fdf3a1" Sep 17 01:28:00.528913 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:28:00.528896 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"022c1b328e2b5391fe82e8b5305c08a0dbeb9a036fe5ce2f74bfc218b9fdf3a1\": container with ID starting with 022c1b328e2b5391fe82e8b5305c08a0dbeb9a036fe5ce2f74bfc218b9fdf3a1 not found: ID does not exist" containerID="022c1b328e2b5391fe82e8b5305c08a0dbeb9a036fe5ce2f74bfc218b9fdf3a1" Sep 17 01:28:00.529002 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.528919 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"022c1b328e2b5391fe82e8b5305c08a0dbeb9a036fe5ce2f74bfc218b9fdf3a1"} err="failed to get container status \"022c1b328e2b5391fe82e8b5305c08a0dbeb9a036fe5ce2f74bfc218b9fdf3a1\": rpc error: code = NotFound desc = could not find container \"022c1b328e2b5391fe82e8b5305c08a0dbeb9a036fe5ce2f74bfc218b9fdf3a1\": container with ID starting with 022c1b328e2b5391fe82e8b5305c08a0dbeb9a036fe5ce2f74bfc218b9fdf3a1 not found: ID does not exist" Sep 17 01:28:00.530894 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.530876 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.533129 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.533100 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 17 01:28:00.533218 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.533162 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 17 01:28:00.533218 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.533182 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 17 01:28:00.533628 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.533599 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 17 01:28:00.533726 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.533633 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 17 01:28:00.533726 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.533690 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 17 01:28:00.533866 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.533800 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qzjv6\"" Sep 17 01:28:00.533866 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.533808 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 17 01:28:00.534162 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.534122 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 17 01:28:00.537919 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.537898 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 17 01:28:00.540747 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.540726 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 01:28:00.627745 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.627711 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.627905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.627752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-web-config\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.627905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.627782 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-config-volume\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.627905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.627828 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.627905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.627854 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60997e32-a723-4ea7-9716-66971576bb36-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.628112 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.627902 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60997e32-a723-4ea7-9716-66971576bb36-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.628112 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.627927 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbxgf\" (UniqueName: \"kubernetes.io/projected/60997e32-a723-4ea7-9716-66971576bb36-kube-api-access-mbxgf\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.628112 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.627969 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.628112 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.627998 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60997e32-a723-4ea7-9716-66971576bb36-config-out\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.628112 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.628016 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/60997e32-a723-4ea7-9716-66971576bb36-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.628112 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.628033 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.628112 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.628055 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.628337 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.628131 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60997e32-a723-4ea7-9716-66971576bb36-tls-assets\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.729201 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.729149 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.729201 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.729180 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-web-config\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.729394 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.729202 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-config-volume\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.729394 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.729226 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.729394 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.729243 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60997e32-a723-4ea7-9716-66971576bb36-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.729394 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.729265 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60997e32-a723-4ea7-9716-66971576bb36-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.729394 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.729289 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbxgf\" (UniqueName: \"kubernetes.io/projected/60997e32-a723-4ea7-9716-66971576bb36-kube-api-access-mbxgf\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.729394 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.729320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.729394 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.729351 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60997e32-a723-4ea7-9716-66971576bb36-config-out\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.729394 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.729376 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/60997e32-a723-4ea7-9716-66971576bb36-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.729707 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.729399 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.729707 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.729435 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.729707 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.729482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60997e32-a723-4ea7-9716-66971576bb36-tls-assets\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.730064 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.730040 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60997e32-a723-4ea7-9716-66971576bb36-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.730171 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.730140 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/60997e32-a723-4ea7-9716-66971576bb36-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.730553 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.730525 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60997e32-a723-4ea7-9716-66971576bb36-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.732516 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.732485 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.732608 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.732491 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60997e32-a723-4ea7-9716-66971576bb36-tls-assets\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.732755 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.732720 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-web-config\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.732755 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.732746 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.732946 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.732903 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.733209 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.733188 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.733209 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.733199 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.733309 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.733267 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/60997e32-a723-4ea7-9716-66971576bb36-config-volume\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.733992 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.733972 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60997e32-a723-4ea7-9716-66971576bb36-config-out\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.736295 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.736278 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbxgf\" (UniqueName: \"kubernetes.io/projected/60997e32-a723-4ea7-9716-66971576bb36-kube-api-access-mbxgf\") pod \"alertmanager-main-0\" (UID: \"60997e32-a723-4ea7-9716-66971576bb36\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:00.842029 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:00.842006 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 01:28:01.118342 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:01.118313 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 01:28:01.120144 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:28:01.120110 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60997e32_a723_4ea7_9716_66971576bb36.slice/crio-a30b6f9fb0d66149daaed995851e5547cea13765b10c51a43714de762e3fb5c4 WatchSource:0}: Error finding container a30b6f9fb0d66149daaed995851e5547cea13765b10c51a43714de762e3fb5c4: Status 404 returned error can't find the container with id a30b6f9fb0d66149daaed995851e5547cea13765b10c51a43714de762e3fb5c4 Sep 17 01:28:01.210205 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:01.210180 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2b71772-281b-438d-a110-36d2050aac21" path="/var/lib/kubelet/pods/b2b71772-281b-438d-a110-36d2050aac21/volumes" Sep 17 01:28:01.482363 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:01.482273 2577 generic.go:356] "Generic (PLEG): container finished" podID="60997e32-a723-4ea7-9716-66971576bb36" containerID="68e9b4d29ee0f4acdb33a114c122ce2464d24157fb34d997c80e1a6a5bcc88df" exitCode=0 Sep 17 01:28:01.482521 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:01.482365 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60997e32-a723-4ea7-9716-66971576bb36","Type":"ContainerDied","Data":"68e9b4d29ee0f4acdb33a114c122ce2464d24157fb34d997c80e1a6a5bcc88df"} Sep 17 01:28:01.482521 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:01.482480 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60997e32-a723-4ea7-9716-66971576bb36","Type":"ContainerStarted","Data":"a30b6f9fb0d66149daaed995851e5547cea13765b10c51a43714de762e3fb5c4"} Sep 17 01:28:02.492298 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:02.492261 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60997e32-a723-4ea7-9716-66971576bb36","Type":"ContainerStarted","Data":"42b69de4484bdcbb686e6e74ab060a1601ca02af91edd70adf7d4976f6b8d29b"} Sep 17 01:28:02.492692 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:02.492326 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60997e32-a723-4ea7-9716-66971576bb36","Type":"ContainerStarted","Data":"a3a99a898ab6d00b23ed8ab0f63792cb7633e98e8563adef1e565c3602684720"} Sep 17 01:28:02.492692 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:02.492355 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60997e32-a723-4ea7-9716-66971576bb36","Type":"ContainerStarted","Data":"497c1890fbde8616e5543db753d0bf1894e41348a745d61c3dafc28837c246e5"} Sep 17 01:28:02.492692 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:02.492386 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60997e32-a723-4ea7-9716-66971576bb36","Type":"ContainerStarted","Data":"b8d350b3d4136ec111c7aa2bb122eadd4b23d34a48709f59c4e1757e38f3c203"} Sep 17 01:28:02.492692 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:02.492410 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60997e32-a723-4ea7-9716-66971576bb36","Type":"ContainerStarted","Data":"0b769900f5d6ff9a1fc5609ff780858b7ee16f8cc271448dd43ef047ca60da6b"} Sep 17 01:28:02.492692 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:02.492440 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60997e32-a723-4ea7-9716-66971576bb36","Type":"ContainerStarted","Data":"8faa102e763c1d63bbdbcf3207c31f874a28dcb0209f94c2dca90aba9fb90b79"} Sep 17 01:28:02.516370 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:02.516326 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.501963552 podStartE2EDuration="2.516310227s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.346675ms" isStatefulPod=false podCreationTimestamp="2026-09-17 01:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:28:02.514208345 +0000 UTC m=+347.749355534" watchObservedRunningTime="2026-09-17 01:28:02.516310227 +0000 UTC m=+347.751457414" Sep 17 01:28:19.099672 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.099626 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fdc664585-s8pnk"] Sep 17 01:28:19.104665 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.104644 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.113251 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.113222 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fdc664585-s8pnk"] Sep 17 01:28:19.191498 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.191460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-trusted-ca-bundle\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.191498 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.191501 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-console-config\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.191708 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.191517 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phdvc\" (UniqueName: \"kubernetes.io/projected/ff0a23a2-2849-4b7f-ae09-2e29651811e4-kube-api-access-phdvc\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.191708 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.191537 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff0a23a2-2849-4b7f-ae09-2e29651811e4-console-oauth-config\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.191708 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.191591 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff0a23a2-2849-4b7f-ae09-2e29651811e4-console-serving-cert\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.191708 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.191640 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-service-ca\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.191708 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.191663 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-oauth-serving-cert\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.292703 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.292671 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-trusted-ca-bundle\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.292703 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.292711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-console-config\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.292965 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.292726 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phdvc\" (UniqueName: \"kubernetes.io/projected/ff0a23a2-2849-4b7f-ae09-2e29651811e4-kube-api-access-phdvc\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.292965 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.292747 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff0a23a2-2849-4b7f-ae09-2e29651811e4-console-oauth-config\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.292965 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.292762 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff0a23a2-2849-4b7f-ae09-2e29651811e4-console-serving-cert\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.292965 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.292790 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-service-ca\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.292965 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.292818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-oauth-serving-cert\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.293531 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.293499 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-console-config\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.293635 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.293516 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-oauth-serving-cert\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.293635 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.293558 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-service-ca\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.293806 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.293780 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-trusted-ca-bundle\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.295454 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.295420 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff0a23a2-2849-4b7f-ae09-2e29651811e4-console-oauth-config\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.295556 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.295502 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff0a23a2-2849-4b7f-ae09-2e29651811e4-console-serving-cert\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.299840 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.299816 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phdvc\" (UniqueName: \"kubernetes.io/projected/ff0a23a2-2849-4b7f-ae09-2e29651811e4-kube-api-access-phdvc\") pod \"console-6fdc664585-s8pnk\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.415027 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.414942 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:19.687090 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:19.686912 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fdc664585-s8pnk"] Sep 17 01:28:19.689674 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:28:19.689640 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0a23a2_2849_4b7f_ae09_2e29651811e4.slice/crio-e6dd29fd1425ca7d481bd7b954fba9fc769272083f49b11839d621d7da20256c WatchSource:0}: Error finding container e6dd29fd1425ca7d481bd7b954fba9fc769272083f49b11839d621d7da20256c: Status 404 returned error can't find the container with id e6dd29fd1425ca7d481bd7b954fba9fc769272083f49b11839d621d7da20256c Sep 17 01:28:20.573926 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:20.573888 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fdc664585-s8pnk" event={"ID":"ff0a23a2-2849-4b7f-ae09-2e29651811e4","Type":"ContainerStarted","Data":"d3dfef928563146aa892ee4967079813a5257b580409b91a78fd68197763ec1e"} Sep 17 01:28:20.574342 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:20.573955 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fdc664585-s8pnk" event={"ID":"ff0a23a2-2849-4b7f-ae09-2e29651811e4","Type":"ContainerStarted","Data":"e6dd29fd1425ca7d481bd7b954fba9fc769272083f49b11839d621d7da20256c"} Sep 17 01:28:20.590657 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:20.590606 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fdc664585-s8pnk" podStartSLOduration=1.590589754 podStartE2EDuration="1.590589754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:28:20.588996362 +0000 UTC m=+365.824143550" watchObservedRunningTime="2026-09-17 01:28:20.590589754 +0000 UTC m=+365.825736999" Sep 17 01:28:29.415480 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:29.415432 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:29.415879 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:29.415497 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:29.420167 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:29.420143 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:29.612252 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:29.612219 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:28:29.652749 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:29.652713 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fc6fdf468-czlvg"] Sep 17 01:28:54.679060 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:54.679019 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5fc6fdf468-czlvg" podUID="2535db49-7b69-4f01-82b3-735111df9bfe" containerName="console" containerID="cri-o://3cf885c4eaef98dd1ac56c74d36017a910143ed96346a08497cddba6bb11f277" gracePeriod=15 Sep 17 01:28:54.919284 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:54.919259 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:28:54.982042 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:54.981971 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2535db49-7b69-4f01-82b3-735111df9bfe-console-serving-cert\") pod \"2535db49-7b69-4f01-82b3-735111df9bfe\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " Sep 17 01:28:54.982042 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:54.982017 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndjhj\" (UniqueName: \"kubernetes.io/projected/2535db49-7b69-4f01-82b3-735111df9bfe-kube-api-access-ndjhj\") pod \"2535db49-7b69-4f01-82b3-735111df9bfe\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " Sep 17 01:28:54.982042 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:54.982041 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-oauth-serving-cert\") pod \"2535db49-7b69-4f01-82b3-735111df9bfe\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " Sep 17 01:28:54.982279 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:54.982067 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-service-ca\") pod \"2535db49-7b69-4f01-82b3-735111df9bfe\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " Sep 17 01:28:54.982279 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:54.982098 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-console-config\") pod \"2535db49-7b69-4f01-82b3-735111df9bfe\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " Sep 17 01:28:54.982279 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:54.982115 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-trusted-ca-bundle\") pod \"2535db49-7b69-4f01-82b3-735111df9bfe\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " Sep 17 01:28:54.982279 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:54.982131 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2535db49-7b69-4f01-82b3-735111df9bfe-console-oauth-config\") pod \"2535db49-7b69-4f01-82b3-735111df9bfe\" (UID: \"2535db49-7b69-4f01-82b3-735111df9bfe\") " Sep 17 01:28:54.982569 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:54.982542 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2535db49-7b69-4f01-82b3-735111df9bfe" (UID: "2535db49-7b69-4f01-82b3-735111df9bfe"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:28:54.982667 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:54.982615 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-console-config" (OuterVolumeSpecName: "console-config") pod "2535db49-7b69-4f01-82b3-735111df9bfe" (UID: "2535db49-7b69-4f01-82b3-735111df9bfe"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:28:54.982667 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:54.982625 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2535db49-7b69-4f01-82b3-735111df9bfe" (UID: "2535db49-7b69-4f01-82b3-735111df9bfe"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:28:54.982667 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:54.982618 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-service-ca" (OuterVolumeSpecName: "service-ca") pod "2535db49-7b69-4f01-82b3-735111df9bfe" (UID: "2535db49-7b69-4f01-82b3-735111df9bfe"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:28:54.984368 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:54.984343 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2535db49-7b69-4f01-82b3-735111df9bfe-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2535db49-7b69-4f01-82b3-735111df9bfe" (UID: "2535db49-7b69-4f01-82b3-735111df9bfe"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:28:54.984614 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:54.984592 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2535db49-7b69-4f01-82b3-735111df9bfe-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2535db49-7b69-4f01-82b3-735111df9bfe" (UID: "2535db49-7b69-4f01-82b3-735111df9bfe"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:28:54.984614 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:54.984603 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2535db49-7b69-4f01-82b3-735111df9bfe-kube-api-access-ndjhj" (OuterVolumeSpecName: "kube-api-access-ndjhj") pod "2535db49-7b69-4f01-82b3-735111df9bfe" (UID: "2535db49-7b69-4f01-82b3-735111df9bfe"). InnerVolumeSpecName "kube-api-access-ndjhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:28:55.082840 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:55.082808 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndjhj\" (UniqueName: \"kubernetes.io/projected/2535db49-7b69-4f01-82b3-735111df9bfe-kube-api-access-ndjhj\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:28:55.082840 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:55.082834 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-oauth-serving-cert\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:28:55.083050 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:55.082847 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-service-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:28:55.083050 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:55.082860 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-console-config\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:28:55.083050 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:55.082873 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2535db49-7b69-4f01-82b3-735111df9bfe-trusted-ca-bundle\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:28:55.083050 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:55.082884 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2535db49-7b69-4f01-82b3-735111df9bfe-console-oauth-config\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:28:55.083050 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:55.082896 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2535db49-7b69-4f01-82b3-735111df9bfe-console-serving-cert\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:28:55.715952 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:55.715896 2577 generic.go:356] "Generic (PLEG): container finished" podID="2535db49-7b69-4f01-82b3-735111df9bfe" containerID="3cf885c4eaef98dd1ac56c74d36017a910143ed96346a08497cddba6bb11f277" exitCode=0 Sep 17 01:28:55.716383 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:55.715966 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fc6fdf468-czlvg" Sep 17 01:28:55.716383 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:55.715986 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fc6fdf468-czlvg" event={"ID":"2535db49-7b69-4f01-82b3-735111df9bfe","Type":"ContainerDied","Data":"3cf885c4eaef98dd1ac56c74d36017a910143ed96346a08497cddba6bb11f277"} Sep 17 01:28:55.716383 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:55.716045 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fc6fdf468-czlvg" event={"ID":"2535db49-7b69-4f01-82b3-735111df9bfe","Type":"ContainerDied","Data":"9b479421066d3760291496fa22835e378b728e8237fcac881718afd417cb5487"} Sep 17 01:28:55.716383 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:55.716061 2577 scope.go:117] "RemoveContainer" containerID="3cf885c4eaef98dd1ac56c74d36017a910143ed96346a08497cddba6bb11f277" Sep 17 01:28:55.724412 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:55.724397 2577 scope.go:117] "RemoveContainer" containerID="3cf885c4eaef98dd1ac56c74d36017a910143ed96346a08497cddba6bb11f277" Sep 17 01:28:55.724670 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:28:55.724645 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf885c4eaef98dd1ac56c74d36017a910143ed96346a08497cddba6bb11f277\": container with ID starting with 3cf885c4eaef98dd1ac56c74d36017a910143ed96346a08497cddba6bb11f277 not found: ID does not exist" containerID="3cf885c4eaef98dd1ac56c74d36017a910143ed96346a08497cddba6bb11f277" Sep 17 01:28:55.724774 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:55.724672 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf885c4eaef98dd1ac56c74d36017a910143ed96346a08497cddba6bb11f277"} err="failed to get container status \"3cf885c4eaef98dd1ac56c74d36017a910143ed96346a08497cddba6bb11f277\": rpc error: code = NotFound desc = could not find container \"3cf885c4eaef98dd1ac56c74d36017a910143ed96346a08497cddba6bb11f277\": container with ID starting with 3cf885c4eaef98dd1ac56c74d36017a910143ed96346a08497cddba6bb11f277 not found: ID does not exist" Sep 17 01:28:55.729803 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:55.729783 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fc6fdf468-czlvg"] Sep 17 01:28:55.733307 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:55.733284 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fc6fdf468-czlvg"] Sep 17 01:28:57.211393 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:28:57.211356 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2535db49-7b69-4f01-82b3-735111df9bfe" path="/var/lib/kubelet/pods/2535db49-7b69-4f01-82b3-735111df9bfe/volumes" Sep 17 01:29:03.492763 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:03.492725 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9"] Sep 17 01:29:03.493229 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:03.493068 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2535db49-7b69-4f01-82b3-735111df9bfe" containerName="console" Sep 17 01:29:03.493229 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:03.493081 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2535db49-7b69-4f01-82b3-735111df9bfe" containerName="console" Sep 17 01:29:03.493229 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:03.493137 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2535db49-7b69-4f01-82b3-735111df9bfe" containerName="console" Sep 17 01:29:03.497632 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:03.497615 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9" Sep 17 01:29:03.499903 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:03.499878 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 01:29:03.500036 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:03.499884 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 01:29:03.500743 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:03.500729 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7clft\"" Sep 17 01:29:03.504156 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:03.504130 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9"] Sep 17 01:29:03.648718 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:03.648673 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/872f6348-b8fa-4ffa-a622-a0f99a3ffe22-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9\" (UID: \"872f6348-b8fa-4ffa-a622-a0f99a3ffe22\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9" Sep 17 01:29:03.648718 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:03.648724 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/872f6348-b8fa-4ffa-a622-a0f99a3ffe22-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9\" (UID: \"872f6348-b8fa-4ffa-a622-a0f99a3ffe22\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9" Sep 17 01:29:03.648957 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:03.648796 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnbrs\" (UniqueName: \"kubernetes.io/projected/872f6348-b8fa-4ffa-a622-a0f99a3ffe22-kube-api-access-fnbrs\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9\" (UID: \"872f6348-b8fa-4ffa-a622-a0f99a3ffe22\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9" Sep 17 01:29:03.749581 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:03.749512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/872f6348-b8fa-4ffa-a622-a0f99a3ffe22-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9\" (UID: \"872f6348-b8fa-4ffa-a622-a0f99a3ffe22\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9" Sep 17 01:29:03.749581 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:03.749546 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/872f6348-b8fa-4ffa-a622-a0f99a3ffe22-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9\" (UID: \"872f6348-b8fa-4ffa-a622-a0f99a3ffe22\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9" Sep 17 01:29:03.749734 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:03.749586 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnbrs\" (UniqueName: \"kubernetes.io/projected/872f6348-b8fa-4ffa-a622-a0f99a3ffe22-kube-api-access-fnbrs\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9\" (UID: \"872f6348-b8fa-4ffa-a622-a0f99a3ffe22\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9" Sep 17 01:29:03.749886 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:03.749869 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/872f6348-b8fa-4ffa-a622-a0f99a3ffe22-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9\" (UID: \"872f6348-b8fa-4ffa-a622-a0f99a3ffe22\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9" Sep 17 01:29:03.749980 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:03.749960 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/872f6348-b8fa-4ffa-a622-a0f99a3ffe22-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9\" (UID: \"872f6348-b8fa-4ffa-a622-a0f99a3ffe22\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9" Sep 17 01:29:03.757177 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:03.757152 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnbrs\" (UniqueName: \"kubernetes.io/projected/872f6348-b8fa-4ffa-a622-a0f99a3ffe22-kube-api-access-fnbrs\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9\" (UID: \"872f6348-b8fa-4ffa-a622-a0f99a3ffe22\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9" Sep 17 01:29:03.808052 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:03.808028 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9" Sep 17 01:29:04.075132 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:04.075105 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9"] Sep 17 01:29:04.077756 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:29:04.077723 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872f6348_b8fa_4ffa_a622_a0f99a3ffe22.slice/crio-0de1245e7fb40cc1fd8050dd9e021c942038ad25d111266c5f40adcc8110e32f WatchSource:0}: Error finding container 0de1245e7fb40cc1fd8050dd9e021c942038ad25d111266c5f40adcc8110e32f: Status 404 returned error can't find the container with id 0de1245e7fb40cc1fd8050dd9e021c942038ad25d111266c5f40adcc8110e32f Sep 17 01:29:04.079727 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:04.079710 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 01:29:04.754512 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:04.754474 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9" event={"ID":"872f6348-b8fa-4ffa-a622-a0f99a3ffe22","Type":"ContainerStarted","Data":"0de1245e7fb40cc1fd8050dd9e021c942038ad25d111266c5f40adcc8110e32f"} Sep 17 01:29:10.782506 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:10.782467 2577 generic.go:356] "Generic (PLEG): container finished" podID="872f6348-b8fa-4ffa-a622-a0f99a3ffe22" containerID="2e9616d0a24fa4334295ee7e27e7e312291aa1f3d26bf1c1fdcae75a90455d2c" exitCode=0 Sep 17 01:29:10.782898 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:10.782554 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9" event={"ID":"872f6348-b8fa-4ffa-a622-a0f99a3ffe22","Type":"ContainerDied","Data":"2e9616d0a24fa4334295ee7e27e7e312291aa1f3d26bf1c1fdcae75a90455d2c"} Sep 17 01:29:14.801396 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:14.801354 2577 generic.go:356] "Generic (PLEG): container finished" podID="872f6348-b8fa-4ffa-a622-a0f99a3ffe22" containerID="f33b2f62930064f7098c4c4a29dcfe50a71960eace0157844275247e83ddfe8a" exitCode=0 Sep 17 01:29:14.801769 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:14.801436 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9" event={"ID":"872f6348-b8fa-4ffa-a622-a0f99a3ffe22","Type":"ContainerDied","Data":"f33b2f62930064f7098c4c4a29dcfe50a71960eace0157844275247e83ddfe8a"} Sep 17 01:29:22.838788 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:22.838700 2577 generic.go:356] "Generic (PLEG): container finished" podID="872f6348-b8fa-4ffa-a622-a0f99a3ffe22" containerID="ca9bb0f8c6ceaaa564130bc41f4bfe566d7158073e8e608a102a71793ab0f236" exitCode=0 Sep 17 01:29:22.839164 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:22.838787 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9" event={"ID":"872f6348-b8fa-4ffa-a622-a0f99a3ffe22","Type":"ContainerDied","Data":"ca9bb0f8c6ceaaa564130bc41f4bfe566d7158073e8e608a102a71793ab0f236"} Sep 17 01:29:23.963247 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:23.963219 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9" Sep 17 01:29:24.028324 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:24.028296 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/872f6348-b8fa-4ffa-a622-a0f99a3ffe22-util\") pod \"872f6348-b8fa-4ffa-a622-a0f99a3ffe22\" (UID: \"872f6348-b8fa-4ffa-a622-a0f99a3ffe22\") " Sep 17 01:29:24.028490 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:24.028345 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/872f6348-b8fa-4ffa-a622-a0f99a3ffe22-bundle\") pod \"872f6348-b8fa-4ffa-a622-a0f99a3ffe22\" (UID: \"872f6348-b8fa-4ffa-a622-a0f99a3ffe22\") " Sep 17 01:29:24.028490 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:24.028371 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnbrs\" (UniqueName: \"kubernetes.io/projected/872f6348-b8fa-4ffa-a622-a0f99a3ffe22-kube-api-access-fnbrs\") pod \"872f6348-b8fa-4ffa-a622-a0f99a3ffe22\" (UID: \"872f6348-b8fa-4ffa-a622-a0f99a3ffe22\") " Sep 17 01:29:24.029047 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:24.029023 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872f6348-b8fa-4ffa-a622-a0f99a3ffe22-bundle" (OuterVolumeSpecName: "bundle") pod "872f6348-b8fa-4ffa-a622-a0f99a3ffe22" (UID: "872f6348-b8fa-4ffa-a622-a0f99a3ffe22"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:29:24.030529 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:24.030510 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872f6348-b8fa-4ffa-a622-a0f99a3ffe22-kube-api-access-fnbrs" (OuterVolumeSpecName: "kube-api-access-fnbrs") pod "872f6348-b8fa-4ffa-a622-a0f99a3ffe22" (UID: "872f6348-b8fa-4ffa-a622-a0f99a3ffe22"). InnerVolumeSpecName "kube-api-access-fnbrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:29:24.032285 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:24.032265 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872f6348-b8fa-4ffa-a622-a0f99a3ffe22-util" (OuterVolumeSpecName: "util") pod "872f6348-b8fa-4ffa-a622-a0f99a3ffe22" (UID: "872f6348-b8fa-4ffa-a622-a0f99a3ffe22"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:29:24.129822 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:24.129736 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/872f6348-b8fa-4ffa-a622-a0f99a3ffe22-util\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:29:24.129822 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:24.129774 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/872f6348-b8fa-4ffa-a622-a0f99a3ffe22-bundle\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:29:24.129822 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:24.129786 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnbrs\" (UniqueName: \"kubernetes.io/projected/872f6348-b8fa-4ffa-a622-a0f99a3ffe22-kube-api-access-fnbrs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:29:24.848390 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:24.848358 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9" Sep 17 01:29:24.848581 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:24.848358 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754048xc9" event={"ID":"872f6348-b8fa-4ffa-a622-a0f99a3ffe22","Type":"ContainerDied","Data":"0de1245e7fb40cc1fd8050dd9e021c942038ad25d111266c5f40adcc8110e32f"} Sep 17 01:29:24.848581 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:24.848484 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0de1245e7fb40cc1fd8050dd9e021c942038ad25d111266c5f40adcc8110e32f" Sep 17 01:29:31.450255 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.450220 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hltpg"] Sep 17 01:29:31.450623 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.450535 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872f6348-b8fa-4ffa-a622-a0f99a3ffe22" containerName="pull" Sep 17 01:29:31.450623 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.450545 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="872f6348-b8fa-4ffa-a622-a0f99a3ffe22" containerName="pull" Sep 17 01:29:31.450623 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.450556 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872f6348-b8fa-4ffa-a622-a0f99a3ffe22" containerName="extract" Sep 17 01:29:31.450623 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.450561 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="872f6348-b8fa-4ffa-a622-a0f99a3ffe22" containerName="extract" Sep 17 01:29:31.450623 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.450582 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872f6348-b8fa-4ffa-a622-a0f99a3ffe22" containerName="util" Sep 17 01:29:31.450623 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.450588 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="872f6348-b8fa-4ffa-a622-a0f99a3ffe22" containerName="util" Sep 17 01:29:31.450844 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.450637 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="872f6348-b8fa-4ffa-a622-a0f99a3ffe22" containerName="extract" Sep 17 01:29:31.453763 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.453746 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hltpg" Sep 17 01:29:31.455840 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.455810 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9pt9z\"" Sep 17 01:29:31.455995 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.455820 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 17 01:29:31.455995 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.455966 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 17 01:29:31.463983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.463961 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hltpg"] Sep 17 01:29:31.487054 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.487028 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f505e6d3-19ac-49f5-87d5-ef3cd8ca1213-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hltpg\" (UID: \"f505e6d3-19ac-49f5-87d5-ef3cd8ca1213\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hltpg" Sep 17 01:29:31.487172 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.487075 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spr65\" (UniqueName: \"kubernetes.io/projected/f505e6d3-19ac-49f5-87d5-ef3cd8ca1213-kube-api-access-spr65\") pod \"cert-manager-operator-controller-manager-55c444bf86-hltpg\" (UID: \"f505e6d3-19ac-49f5-87d5-ef3cd8ca1213\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hltpg" Sep 17 01:29:31.587946 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.587893 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f505e6d3-19ac-49f5-87d5-ef3cd8ca1213-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hltpg\" (UID: \"f505e6d3-19ac-49f5-87d5-ef3cd8ca1213\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hltpg" Sep 17 01:29:31.588115 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.587986 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spr65\" (UniqueName: \"kubernetes.io/projected/f505e6d3-19ac-49f5-87d5-ef3cd8ca1213-kube-api-access-spr65\") pod \"cert-manager-operator-controller-manager-55c444bf86-hltpg\" (UID: \"f505e6d3-19ac-49f5-87d5-ef3cd8ca1213\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hltpg" Sep 17 01:29:31.588314 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.588293 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f505e6d3-19ac-49f5-87d5-ef3cd8ca1213-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hltpg\" (UID: \"f505e6d3-19ac-49f5-87d5-ef3cd8ca1213\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hltpg" Sep 17 01:29:31.596522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.596499 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spr65\" (UniqueName: \"kubernetes.io/projected/f505e6d3-19ac-49f5-87d5-ef3cd8ca1213-kube-api-access-spr65\") pod \"cert-manager-operator-controller-manager-55c444bf86-hltpg\" (UID: \"f505e6d3-19ac-49f5-87d5-ef3cd8ca1213\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hltpg" Sep 17 01:29:31.762812 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:31.762730 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hltpg" Sep 17 01:29:32.039293 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:32.039267 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hltpg"] Sep 17 01:29:32.041699 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:29:32.041671 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf505e6d3_19ac_49f5_87d5_ef3cd8ca1213.slice/crio-a8780bde4e4959a4853fb1873c03c694aa1f7759c6ed75c900edfb0c0c821faa WatchSource:0}: Error finding container a8780bde4e4959a4853fb1873c03c694aa1f7759c6ed75c900edfb0c0c821faa: Status 404 returned error can't find the container with id a8780bde4e4959a4853fb1873c03c694aa1f7759c6ed75c900edfb0c0c821faa Sep 17 01:29:32.880693 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:32.880653 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hltpg" event={"ID":"f505e6d3-19ac-49f5-87d5-ef3cd8ca1213","Type":"ContainerStarted","Data":"a8780bde4e4959a4853fb1873c03c694aa1f7759c6ed75c900edfb0c0c821faa"} Sep 17 01:29:34.891950 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:34.891902 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hltpg" event={"ID":"f505e6d3-19ac-49f5-87d5-ef3cd8ca1213","Type":"ContainerStarted","Data":"d4792733f8181b2bfc0a8504f049a9ab7fdff6a1a766786630c7d377088fe6f3"} Sep 17 01:29:34.911890 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:34.911833 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hltpg" podStartSLOduration=2.100810329 podStartE2EDuration="3.91181412s" totalImagesPullingTime="1.811003791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:29:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:29:34.910948366 +0000 UTC m=+440.146095545" watchObservedRunningTime="2026-09-17 01:29:34.91181412 +0000 UTC m=+440.146961309" Sep 17 01:29:35.093144 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.093106 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-95dct"] Sep 17 01:29:35.096550 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.096526 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-95dct" Sep 17 01:29:35.098707 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.098692 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 17 01:29:35.098794 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.098692 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 17 01:29:35.098794 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.098747 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-d6942\"" Sep 17 01:29:35.105597 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.105572 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-95dct"] Sep 17 01:29:35.119556 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.119526 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dffef6f-2d29-4d8e-9a61-cfefcfe2cdec-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-95dct\" (UID: \"4dffef6f-2d29-4d8e-9a61-cfefcfe2cdec\") " pod="cert-manager/cert-manager-webhook-6486ffd997-95dct" Sep 17 01:29:35.119692 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.119666 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7nw8\" (UniqueName: \"kubernetes.io/projected/4dffef6f-2d29-4d8e-9a61-cfefcfe2cdec-kube-api-access-r7nw8\") pod \"cert-manager-webhook-6486ffd997-95dct\" (UID: \"4dffef6f-2d29-4d8e-9a61-cfefcfe2cdec\") " pod="cert-manager/cert-manager-webhook-6486ffd997-95dct" Sep 17 01:29:35.220850 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.220761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7nw8\" (UniqueName: \"kubernetes.io/projected/4dffef6f-2d29-4d8e-9a61-cfefcfe2cdec-kube-api-access-r7nw8\") pod \"cert-manager-webhook-6486ffd997-95dct\" (UID: \"4dffef6f-2d29-4d8e-9a61-cfefcfe2cdec\") " pod="cert-manager/cert-manager-webhook-6486ffd997-95dct" Sep 17 01:29:35.220850 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.220813 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dffef6f-2d29-4d8e-9a61-cfefcfe2cdec-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-95dct\" (UID: \"4dffef6f-2d29-4d8e-9a61-cfefcfe2cdec\") " pod="cert-manager/cert-manager-webhook-6486ffd997-95dct" Sep 17 01:29:35.229981 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.229945 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7nw8\" (UniqueName: \"kubernetes.io/projected/4dffef6f-2d29-4d8e-9a61-cfefcfe2cdec-kube-api-access-r7nw8\") pod \"cert-manager-webhook-6486ffd997-95dct\" (UID: \"4dffef6f-2d29-4d8e-9a61-cfefcfe2cdec\") " pod="cert-manager/cert-manager-webhook-6486ffd997-95dct" Sep 17 01:29:35.230138 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.230024 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dffef6f-2d29-4d8e-9a61-cfefcfe2cdec-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-95dct\" (UID: \"4dffef6f-2d29-4d8e-9a61-cfefcfe2cdec\") " pod="cert-manager/cert-manager-webhook-6486ffd997-95dct" Sep 17 01:29:35.417061 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.417025 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-95dct" Sep 17 01:29:35.686712 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.686683 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-95dct"] Sep 17 01:29:35.688815 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:29:35.688785 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4dffef6f_2d29_4d8e_9a61_cfefcfe2cdec.slice/crio-a49b3e00d49c61be2e98ef00bf07fdd218c1a69df6516b0ee1546d63cf5e3a9d WatchSource:0}: Error finding container a49b3e00d49c61be2e98ef00bf07fdd218c1a69df6516b0ee1546d63cf5e3a9d: Status 404 returned error can't find the container with id a49b3e00d49c61be2e98ef00bf07fdd218c1a69df6516b0ee1546d63cf5e3a9d Sep 17 01:29:35.900834 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.900798 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-95dct" event={"ID":"4dffef6f-2d29-4d8e-9a61-cfefcfe2cdec","Type":"ContainerStarted","Data":"a49b3e00d49c61be2e98ef00bf07fdd218c1a69df6516b0ee1546d63cf5e3a9d"} Sep 17 01:29:35.927794 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.927760 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5"] Sep 17 01:29:35.932467 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.932448 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5" Sep 17 01:29:35.934578 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.934554 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 01:29:35.934692 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.934641 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 01:29:35.935279 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.935263 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7clft\"" Sep 17 01:29:35.937992 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:35.937921 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5"] Sep 17 01:29:36.027086 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:36.027043 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd2gr\" (UniqueName: \"kubernetes.io/projected/13ae1180-44a1-4b0e-aa06-a4b777813356-kube-api-access-zd2gr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5\" (UID: \"13ae1180-44a1-4b0e-aa06-a4b777813356\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5" Sep 17 01:29:36.027282 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:36.027101 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ae1180-44a1-4b0e-aa06-a4b777813356-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5\" (UID: \"13ae1180-44a1-4b0e-aa06-a4b777813356\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5" Sep 17 01:29:36.027282 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:36.027180 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13ae1180-44a1-4b0e-aa06-a4b777813356-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5\" (UID: \"13ae1180-44a1-4b0e-aa06-a4b777813356\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5" Sep 17 01:29:36.128175 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:36.128134 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd2gr\" (UniqueName: \"kubernetes.io/projected/13ae1180-44a1-4b0e-aa06-a4b777813356-kube-api-access-zd2gr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5\" (UID: \"13ae1180-44a1-4b0e-aa06-a4b777813356\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5" Sep 17 01:29:36.128332 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:36.128211 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ae1180-44a1-4b0e-aa06-a4b777813356-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5\" (UID: \"13ae1180-44a1-4b0e-aa06-a4b777813356\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5" Sep 17 01:29:36.128332 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:36.128249 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13ae1180-44a1-4b0e-aa06-a4b777813356-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5\" (UID: \"13ae1180-44a1-4b0e-aa06-a4b777813356\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5" Sep 17 01:29:36.128662 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:36.128635 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ae1180-44a1-4b0e-aa06-a4b777813356-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5\" (UID: \"13ae1180-44a1-4b0e-aa06-a4b777813356\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5" Sep 17 01:29:36.128722 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:36.128691 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13ae1180-44a1-4b0e-aa06-a4b777813356-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5\" (UID: \"13ae1180-44a1-4b0e-aa06-a4b777813356\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5" Sep 17 01:29:36.135903 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:36.135883 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd2gr\" (UniqueName: \"kubernetes.io/projected/13ae1180-44a1-4b0e-aa06-a4b777813356-kube-api-access-zd2gr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5\" (UID: \"13ae1180-44a1-4b0e-aa06-a4b777813356\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5" Sep 17 01:29:36.243566 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:36.243490 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5" Sep 17 01:29:36.516236 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:36.516010 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5"] Sep 17 01:29:36.518576 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:29:36.518547 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ae1180_44a1_4b0e_aa06_a4b777813356.slice/crio-ae6461e1e5c760ba3fc97fd02e0769c84893734f3770aa95651c0bdb9ba8b653 WatchSource:0}: Error finding container ae6461e1e5c760ba3fc97fd02e0769c84893734f3770aa95651c0bdb9ba8b653: Status 404 returned error can't find the container with id ae6461e1e5c760ba3fc97fd02e0769c84893734f3770aa95651c0bdb9ba8b653 Sep 17 01:29:36.908012 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:36.907972 2577 generic.go:356] "Generic (PLEG): container finished" podID="13ae1180-44a1-4b0e-aa06-a4b777813356" containerID="1850af48b047997f55530e5087be133a947ec97a9508b655585f9d5117509ebc" exitCode=0 Sep 17 01:29:36.908406 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:36.908032 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5" event={"ID":"13ae1180-44a1-4b0e-aa06-a4b777813356","Type":"ContainerDied","Data":"1850af48b047997f55530e5087be133a947ec97a9508b655585f9d5117509ebc"} Sep 17 01:29:36.908406 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:36.908071 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5" event={"ID":"13ae1180-44a1-4b0e-aa06-a4b777813356","Type":"ContainerStarted","Data":"ae6461e1e5c760ba3fc97fd02e0769c84893734f3770aa95651c0bdb9ba8b653"} Sep 17 01:29:37.879347 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:37.879306 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8lst7"] Sep 17 01:29:37.882923 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:37.882901 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8lst7" Sep 17 01:29:37.884945 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:37.884905 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-kdmhg\"" Sep 17 01:29:37.893688 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:37.891005 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8lst7"] Sep 17 01:29:37.944487 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:37.944303 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07750d2e-bed2-4e5f-bbb3-e971c4073a8b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8lst7\" (UID: \"07750d2e-bed2-4e5f-bbb3-e971c4073a8b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8lst7" Sep 17 01:29:37.944487 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:37.944367 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l66zj\" (UniqueName: \"kubernetes.io/projected/07750d2e-bed2-4e5f-bbb3-e971c4073a8b-kube-api-access-l66zj\") pod \"cert-manager-cainjector-559cdd8c87-8lst7\" (UID: \"07750d2e-bed2-4e5f-bbb3-e971c4073a8b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8lst7" Sep 17 01:29:38.045727 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:38.045684 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07750d2e-bed2-4e5f-bbb3-e971c4073a8b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8lst7\" (UID: \"07750d2e-bed2-4e5f-bbb3-e971c4073a8b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8lst7" Sep 17 01:29:38.045895 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:38.045755 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l66zj\" (UniqueName: \"kubernetes.io/projected/07750d2e-bed2-4e5f-bbb3-e971c4073a8b-kube-api-access-l66zj\") pod \"cert-manager-cainjector-559cdd8c87-8lst7\" (UID: \"07750d2e-bed2-4e5f-bbb3-e971c4073a8b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8lst7" Sep 17 01:29:38.055812 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:38.055783 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07750d2e-bed2-4e5f-bbb3-e971c4073a8b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8lst7\" (UID: \"07750d2e-bed2-4e5f-bbb3-e971c4073a8b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8lst7" Sep 17 01:29:38.055953 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:38.055886 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l66zj\" (UniqueName: \"kubernetes.io/projected/07750d2e-bed2-4e5f-bbb3-e971c4073a8b-kube-api-access-l66zj\") pod \"cert-manager-cainjector-559cdd8c87-8lst7\" (UID: \"07750d2e-bed2-4e5f-bbb3-e971c4073a8b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8lst7" Sep 17 01:29:38.200632 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:38.200543 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8lst7" Sep 17 01:29:38.479750 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:38.479661 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8lst7"] Sep 17 01:29:38.483233 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:29:38.483202 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07750d2e_bed2_4e5f_bbb3_e971c4073a8b.slice/crio-d52afa39402f4e872f5cbd32ba74c17ba10a3d0ce6b3e773a2a2f1d49513f0e2 WatchSource:0}: Error finding container d52afa39402f4e872f5cbd32ba74c17ba10a3d0ce6b3e773a2a2f1d49513f0e2: Status 404 returned error can't find the container with id d52afa39402f4e872f5cbd32ba74c17ba10a3d0ce6b3e773a2a2f1d49513f0e2 Sep 17 01:29:38.920455 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:38.920405 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8lst7" event={"ID":"07750d2e-bed2-4e5f-bbb3-e971c4073a8b","Type":"ContainerStarted","Data":"d52afa39402f4e872f5cbd32ba74c17ba10a3d0ce6b3e773a2a2f1d49513f0e2"} Sep 17 01:29:40.933227 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:40.933185 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-95dct" event={"ID":"4dffef6f-2d29-4d8e-9a61-cfefcfe2cdec","Type":"ContainerStarted","Data":"950a2744f9b163902bed72647f80a0a482e1ee167a3c3abfc06891e9a0dc86ec"} Sep 17 01:29:40.933643 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:40.933238 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-95dct" Sep 17 01:29:40.934818 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:40.934785 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8lst7" event={"ID":"07750d2e-bed2-4e5f-bbb3-e971c4073a8b","Type":"ContainerStarted","Data":"78b5b2e1e154ab21c428ca473e9cb2daa8f9cf8318cdc31d5e85a3be72071aa2"} Sep 17 01:29:40.936637 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:40.936613 2577 generic.go:356] "Generic (PLEG): container finished" podID="13ae1180-44a1-4b0e-aa06-a4b777813356" containerID="b3f999502d7471434f2de6d6125118c8d4393b6a4062b4cd9994d1f27bb0719e" exitCode=0 Sep 17 01:29:40.936755 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:40.936658 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5" event={"ID":"13ae1180-44a1-4b0e-aa06-a4b777813356","Type":"ContainerDied","Data":"b3f999502d7471434f2de6d6125118c8d4393b6a4062b4cd9994d1f27bb0719e"} Sep 17 01:29:40.954125 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:40.954082 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-95dct" podStartSLOduration=1.481198877 podStartE2EDuration="5.954064528s" totalImagesPullingTime="4.472865651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:29:40.953139582 +0000 UTC m=+446.188286770" watchObservedRunningTime="2026-09-17 01:29:40.954064528 +0000 UTC m=+446.189211715" Sep 17 01:29:40.966612 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:40.966563 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8lst7" podStartSLOduration=2.223659005 podStartE2EDuration="3.966543498s" totalImagesPullingTime="1.742884493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:29:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:29:40.965901695 +0000 UTC m=+446.201048883" watchObservedRunningTime="2026-09-17 01:29:40.966543498 +0000 UTC m=+446.201690685" Sep 17 01:29:41.943978 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:41.943946 2577 generic.go:356] "Generic (PLEG): container finished" podID="13ae1180-44a1-4b0e-aa06-a4b777813356" containerID="b001b5932a8e23aa912f3f825c09480b5bda5049487e7d6b9f38ea4ac244a2a0" exitCode=0 Sep 17 01:29:41.944380 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:41.944043 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5" event={"ID":"13ae1180-44a1-4b0e-aa06-a4b777813356","Type":"ContainerDied","Data":"b001b5932a8e23aa912f3f825c09480b5bda5049487e7d6b9f38ea4ac244a2a0"} Sep 17 01:29:43.076280 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:43.076257 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5" Sep 17 01:29:43.198095 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:43.198026 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13ae1180-44a1-4b0e-aa06-a4b777813356-util\") pod \"13ae1180-44a1-4b0e-aa06-a4b777813356\" (UID: \"13ae1180-44a1-4b0e-aa06-a4b777813356\") " Sep 17 01:29:43.198095 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:43.198100 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ae1180-44a1-4b0e-aa06-a4b777813356-bundle\") pod \"13ae1180-44a1-4b0e-aa06-a4b777813356\" (UID: \"13ae1180-44a1-4b0e-aa06-a4b777813356\") " Sep 17 01:29:43.198314 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:43.198127 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd2gr\" (UniqueName: \"kubernetes.io/projected/13ae1180-44a1-4b0e-aa06-a4b777813356-kube-api-access-zd2gr\") pod \"13ae1180-44a1-4b0e-aa06-a4b777813356\" (UID: \"13ae1180-44a1-4b0e-aa06-a4b777813356\") " Sep 17 01:29:43.198956 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:43.198895 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ae1180-44a1-4b0e-aa06-a4b777813356-bundle" (OuterVolumeSpecName: "bundle") pod "13ae1180-44a1-4b0e-aa06-a4b777813356" (UID: "13ae1180-44a1-4b0e-aa06-a4b777813356"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:29:43.200556 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:43.200489 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ae1180-44a1-4b0e-aa06-a4b777813356-kube-api-access-zd2gr" (OuterVolumeSpecName: "kube-api-access-zd2gr") pod "13ae1180-44a1-4b0e-aa06-a4b777813356" (UID: "13ae1180-44a1-4b0e-aa06-a4b777813356"). InnerVolumeSpecName "kube-api-access-zd2gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:29:43.203287 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:43.203261 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13ae1180-44a1-4b0e-aa06-a4b777813356-util" (OuterVolumeSpecName: "util") pod "13ae1180-44a1-4b0e-aa06-a4b777813356" (UID: "13ae1180-44a1-4b0e-aa06-a4b777813356"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:29:43.299695 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:43.299647 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13ae1180-44a1-4b0e-aa06-a4b777813356-util\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:29:43.299695 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:43.299686 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13ae1180-44a1-4b0e-aa06-a4b777813356-bundle\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:29:43.299695 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:43.299695 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd2gr\" (UniqueName: \"kubernetes.io/projected/13ae1180-44a1-4b0e-aa06-a4b777813356-kube-api-access-zd2gr\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:29:43.955377 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:43.955347 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5" Sep 17 01:29:43.955377 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:43.955371 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz7kv5" event={"ID":"13ae1180-44a1-4b0e-aa06-a4b777813356","Type":"ContainerDied","Data":"ae6461e1e5c760ba3fc97fd02e0769c84893734f3770aa95651c0bdb9ba8b653"} Sep 17 01:29:43.955570 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:43.955420 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae6461e1e5c760ba3fc97fd02e0769c84893734f3770aa95651c0bdb9ba8b653" Sep 17 01:29:46.947301 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:46.947269 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-95dct" Sep 17 01:29:53.730354 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:53.730320 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b7z29"] Sep 17 01:29:53.730723 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:53.730647 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13ae1180-44a1-4b0e-aa06-a4b777813356" containerName="pull" Sep 17 01:29:53.730723 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:53.730658 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="13ae1180-44a1-4b0e-aa06-a4b777813356" containerName="pull" Sep 17 01:29:53.730723 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:53.730674 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13ae1180-44a1-4b0e-aa06-a4b777813356" containerName="util" Sep 17 01:29:53.730723 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:53.730681 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="13ae1180-44a1-4b0e-aa06-a4b777813356" containerName="util" Sep 17 01:29:53.730723 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:53.730701 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13ae1180-44a1-4b0e-aa06-a4b777813356" containerName="extract" Sep 17 01:29:53.730723 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:53.730706 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="13ae1180-44a1-4b0e-aa06-a4b777813356" containerName="extract" Sep 17 01:29:53.730952 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:53.730761 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="13ae1180-44a1-4b0e-aa06-a4b777813356" containerName="extract" Sep 17 01:29:53.737182 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:53.737162 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b7z29" Sep 17 01:29:53.740916 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:53.740890 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 17 01:29:53.741073 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:53.741044 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-2j5bg\"" Sep 17 01:29:53.741441 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:53.741418 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 17 01:29:53.747282 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:53.747261 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b7z29"] Sep 17 01:29:53.884732 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:53.884695 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bdc6fe7-6652-4190-bfc0-b1030e58e632-tmp\") pod \"openshift-lws-operator-fd8ccff4c-b7z29\" (UID: \"0bdc6fe7-6652-4190-bfc0-b1030e58e632\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b7z29" Sep 17 01:29:53.884958 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:53.884812 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frwnl\" (UniqueName: \"kubernetes.io/projected/0bdc6fe7-6652-4190-bfc0-b1030e58e632-kube-api-access-frwnl\") pod \"openshift-lws-operator-fd8ccff4c-b7z29\" (UID: \"0bdc6fe7-6652-4190-bfc0-b1030e58e632\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b7z29" Sep 17 01:29:53.985521 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:53.985422 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwnl\" (UniqueName: \"kubernetes.io/projected/0bdc6fe7-6652-4190-bfc0-b1030e58e632-kube-api-access-frwnl\") pod \"openshift-lws-operator-fd8ccff4c-b7z29\" (UID: \"0bdc6fe7-6652-4190-bfc0-b1030e58e632\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b7z29" Sep 17 01:29:53.985521 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:53.985500 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bdc6fe7-6652-4190-bfc0-b1030e58e632-tmp\") pod \"openshift-lws-operator-fd8ccff4c-b7z29\" (UID: \"0bdc6fe7-6652-4190-bfc0-b1030e58e632\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b7z29" Sep 17 01:29:53.985861 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:53.985842 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bdc6fe7-6652-4190-bfc0-b1030e58e632-tmp\") pod \"openshift-lws-operator-fd8ccff4c-b7z29\" (UID: \"0bdc6fe7-6652-4190-bfc0-b1030e58e632\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b7z29" Sep 17 01:29:53.992617 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:53.992597 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frwnl\" (UniqueName: \"kubernetes.io/projected/0bdc6fe7-6652-4190-bfc0-b1030e58e632-kube-api-access-frwnl\") pod \"openshift-lws-operator-fd8ccff4c-b7z29\" (UID: \"0bdc6fe7-6652-4190-bfc0-b1030e58e632\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b7z29" Sep 17 01:29:54.046596 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:54.046564 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b7z29" Sep 17 01:29:54.312229 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:54.312206 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b7z29"] Sep 17 01:29:54.314075 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:29:54.314048 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bdc6fe7_6652_4190_bfc0_b1030e58e632.slice/crio-8e51fe0f3d361a8d83dcb2aae42ebe402debc14ee7800204a3cf5d7bd580308f WatchSource:0}: Error finding container 8e51fe0f3d361a8d83dcb2aae42ebe402debc14ee7800204a3cf5d7bd580308f: Status 404 returned error can't find the container with id 8e51fe0f3d361a8d83dcb2aae42ebe402debc14ee7800204a3cf5d7bd580308f Sep 17 01:29:55.006811 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:55.006777 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b7z29" event={"ID":"0bdc6fe7-6652-4190-bfc0-b1030e58e632","Type":"ContainerStarted","Data":"8e51fe0f3d361a8d83dcb2aae42ebe402debc14ee7800204a3cf5d7bd580308f"} Sep 17 01:29:57.018685 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:57.018651 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b7z29" event={"ID":"0bdc6fe7-6652-4190-bfc0-b1030e58e632","Type":"ContainerStarted","Data":"f1daa6475d6f724118c93fd3b740e88b8c5b3159ea2b49f3f53702dd56d93004"} Sep 17 01:29:57.033011 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:29:57.032963 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b7z29" podStartSLOduration=1.491187842 podStartE2EDuration="4.032922634s" totalImagesPullingTime="2.541734792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:29:57.032353805 +0000 UTC m=+462.267500995" watchObservedRunningTime="2026-09-17 01:29:57.032922634 +0000 UTC m=+462.268069823" Sep 17 01:30:14.516237 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:14.516199 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq"] Sep 17 01:30:14.522386 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:14.522364 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq" Sep 17 01:30:14.524857 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:14.524830 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7clft\"" Sep 17 01:30:14.525011 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:14.524857 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 01:30:14.525594 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:14.525560 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 01:30:14.526003 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:14.525980 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq"] Sep 17 01:30:14.551896 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:14.551871 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq\" (UID: \"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq" Sep 17 01:30:14.552042 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:14.551909 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq59p\" (UniqueName: \"kubernetes.io/projected/36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd-kube-api-access-lq59p\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq\" (UID: \"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq" Sep 17 01:30:14.552042 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:14.551994 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq\" (UID: \"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq" Sep 17 01:30:14.652719 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:14.652688 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq\" (UID: \"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq" Sep 17 01:30:14.652868 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:14.652746 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq\" (UID: \"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq" Sep 17 01:30:14.652868 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:14.652773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq59p\" (UniqueName: \"kubernetes.io/projected/36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd-kube-api-access-lq59p\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq\" (UID: \"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq" Sep 17 01:30:14.653117 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:14.653095 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq\" (UID: \"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq" Sep 17 01:30:14.653166 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:14.653107 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq\" (UID: \"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq" Sep 17 01:30:14.659982 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:14.659952 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq59p\" (UniqueName: \"kubernetes.io/projected/36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd-kube-api-access-lq59p\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq\" (UID: \"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq" Sep 17 01:30:14.832707 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:14.832673 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq" Sep 17 01:30:15.101420 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:15.101392 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq"] Sep 17 01:30:15.103485 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:30:15.103460 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c35d4b_ccf0_48f3_bfc8_bce4eccfaabd.slice/crio-5675a8f6ce14d6fade0460387e388f1573ab557f25b5762241bd4dce4fb42a22 WatchSource:0}: Error finding container 5675a8f6ce14d6fade0460387e388f1573ab557f25b5762241bd4dce4fb42a22: Status 404 returned error can't find the container with id 5675a8f6ce14d6fade0460387e388f1573ab557f25b5762241bd4dce4fb42a22 Sep 17 01:30:16.110195 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:16.110157 2577 generic.go:356] "Generic (PLEG): container finished" podID="36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd" containerID="8a0a80917dbed2101611478eeffac523b46bfd10e67c1e9bde260ab2ed44b4d3" exitCode=0 Sep 17 01:30:16.110619 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:16.110254 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq" event={"ID":"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd","Type":"ContainerDied","Data":"8a0a80917dbed2101611478eeffac523b46bfd10e67c1e9bde260ab2ed44b4d3"} Sep 17 01:30:16.110619 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:16.110357 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq" event={"ID":"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd","Type":"ContainerStarted","Data":"5675a8f6ce14d6fade0460387e388f1573ab557f25b5762241bd4dce4fb42a22"} Sep 17 01:30:17.116663 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:17.116624 2577 generic.go:356] "Generic (PLEG): container finished" podID="36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd" containerID="f32dead21d0cf85b96dd17a108a01a2c6303d39435311ac192ce881658f9794e" exitCode=0 Sep 17 01:30:17.117094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:17.116671 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq" event={"ID":"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd","Type":"ContainerDied","Data":"f32dead21d0cf85b96dd17a108a01a2c6303d39435311ac192ce881658f9794e"} Sep 17 01:30:18.123250 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.123214 2577 generic.go:356] "Generic (PLEG): container finished" podID="36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd" containerID="429d707b84064a321401aeb524d1d1256d59901ebe5eeae0376dd1187b26125c" exitCode=0 Sep 17 01:30:18.123720 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.123299 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq" event={"ID":"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd","Type":"ContainerDied","Data":"429d707b84064a321401aeb524d1d1256d59901ebe5eeae0376dd1187b26125c"} Sep 17 01:30:18.371547 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.371508 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9"] Sep 17 01:30:18.374810 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.374756 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9" Sep 17 01:30:18.377105 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.377075 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 17 01:30:18.377105 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.377085 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 17 01:30:18.377274 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.377162 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-29l8d\"" Sep 17 01:30:18.377328 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.377277 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 17 01:30:18.380592 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.380569 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/635e80dd-2525-4e5b-9a0c-e8ff2db390fa-metrics-cert\") pod \"lws-controller-manager-5749dbf97f-f95v9\" (UID: \"635e80dd-2525-4e5b-9a0c-e8ff2db390fa\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9" Sep 17 01:30:18.380697 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.380600 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/635e80dd-2525-4e5b-9a0c-e8ff2db390fa-manager-config\") pod \"lws-controller-manager-5749dbf97f-f95v9\" (UID: \"635e80dd-2525-4e5b-9a0c-e8ff2db390fa\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9" Sep 17 01:30:18.380697 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.380664 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/635e80dd-2525-4e5b-9a0c-e8ff2db390fa-cert\") pod \"lws-controller-manager-5749dbf97f-f95v9\" (UID: \"635e80dd-2525-4e5b-9a0c-e8ff2db390fa\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9" Sep 17 01:30:18.380804 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.380763 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtpwm\" (UniqueName: \"kubernetes.io/projected/635e80dd-2525-4e5b-9a0c-e8ff2db390fa-kube-api-access-xtpwm\") pod \"lws-controller-manager-5749dbf97f-f95v9\" (UID: \"635e80dd-2525-4e5b-9a0c-e8ff2db390fa\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9" Sep 17 01:30:18.384436 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.384414 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9"] Sep 17 01:30:18.481636 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.481601 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/635e80dd-2525-4e5b-9a0c-e8ff2db390fa-metrics-cert\") pod \"lws-controller-manager-5749dbf97f-f95v9\" (UID: \"635e80dd-2525-4e5b-9a0c-e8ff2db390fa\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9" Sep 17 01:30:18.481636 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.481633 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/635e80dd-2525-4e5b-9a0c-e8ff2db390fa-manager-config\") pod \"lws-controller-manager-5749dbf97f-f95v9\" (UID: \"635e80dd-2525-4e5b-9a0c-e8ff2db390fa\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9" Sep 17 01:30:18.481849 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.481667 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/635e80dd-2525-4e5b-9a0c-e8ff2db390fa-cert\") pod \"lws-controller-manager-5749dbf97f-f95v9\" (UID: \"635e80dd-2525-4e5b-9a0c-e8ff2db390fa\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9" Sep 17 01:30:18.481849 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.481707 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtpwm\" (UniqueName: \"kubernetes.io/projected/635e80dd-2525-4e5b-9a0c-e8ff2db390fa-kube-api-access-xtpwm\") pod \"lws-controller-manager-5749dbf97f-f95v9\" (UID: \"635e80dd-2525-4e5b-9a0c-e8ff2db390fa\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9" Sep 17 01:30:18.482296 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.482274 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/635e80dd-2525-4e5b-9a0c-e8ff2db390fa-manager-config\") pod \"lws-controller-manager-5749dbf97f-f95v9\" (UID: \"635e80dd-2525-4e5b-9a0c-e8ff2db390fa\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9" Sep 17 01:30:18.484236 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.484207 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/635e80dd-2525-4e5b-9a0c-e8ff2db390fa-cert\") pod \"lws-controller-manager-5749dbf97f-f95v9\" (UID: \"635e80dd-2525-4e5b-9a0c-e8ff2db390fa\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9" Sep 17 01:30:18.484350 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.484243 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/635e80dd-2525-4e5b-9a0c-e8ff2db390fa-metrics-cert\") pod \"lws-controller-manager-5749dbf97f-f95v9\" (UID: \"635e80dd-2525-4e5b-9a0c-e8ff2db390fa\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9" Sep 17 01:30:18.490096 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.490074 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtpwm\" (UniqueName: \"kubernetes.io/projected/635e80dd-2525-4e5b-9a0c-e8ff2db390fa-kube-api-access-xtpwm\") pod \"lws-controller-manager-5749dbf97f-f95v9\" (UID: \"635e80dd-2525-4e5b-9a0c-e8ff2db390fa\") " pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9" Sep 17 01:30:18.684980 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.684878 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9" Sep 17 01:30:18.958371 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:18.958338 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9"] Sep 17 01:30:18.959011 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:30:18.958973 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635e80dd_2525_4e5b_9a0c_e8ff2db390fa.slice/crio-abc331638a0897dd7a86abd2ef675c206b80f3e7a0a20b27308d0aef97c04a40 WatchSource:0}: Error finding container abc331638a0897dd7a86abd2ef675c206b80f3e7a0a20b27308d0aef97c04a40: Status 404 returned error can't find the container with id abc331638a0897dd7a86abd2ef675c206b80f3e7a0a20b27308d0aef97c04a40 Sep 17 01:30:19.129228 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:19.129193 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9" event={"ID":"635e80dd-2525-4e5b-9a0c-e8ff2db390fa","Type":"ContainerStarted","Data":"abc331638a0897dd7a86abd2ef675c206b80f3e7a0a20b27308d0aef97c04a40"} Sep 17 01:30:19.246979 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:19.246953 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq" Sep 17 01:30:19.388844 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:19.388810 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lq59p\" (UniqueName: \"kubernetes.io/projected/36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd-kube-api-access-lq59p\") pod \"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd\" (UID: \"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd\") " Sep 17 01:30:19.389054 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:19.388856 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd-bundle\") pod \"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd\" (UID: \"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd\") " Sep 17 01:30:19.389054 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:19.388884 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd-util\") pod \"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd\" (UID: \"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd\") " Sep 17 01:30:19.389720 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:19.389692 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd-bundle" (OuterVolumeSpecName: "bundle") pod "36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd" (UID: "36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:30:19.390968 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:19.390944 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd-kube-api-access-lq59p" (OuterVolumeSpecName: "kube-api-access-lq59p") pod "36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd" (UID: "36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd"). InnerVolumeSpecName "kube-api-access-lq59p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:30:19.394515 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:19.394492 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd-util" (OuterVolumeSpecName: "util") pod "36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd" (UID: "36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:30:19.489764 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:19.489733 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd-util\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:30:19.489764 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:19.489763 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq59p\" (UniqueName: \"kubernetes.io/projected/36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd-kube-api-access-lq59p\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:30:19.489917 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:19.489778 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd-bundle\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:30:20.136902 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:20.136864 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq" event={"ID":"36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd","Type":"ContainerDied","Data":"5675a8f6ce14d6fade0460387e388f1573ab557f25b5762241bd4dce4fb42a22"} Sep 17 01:30:20.136902 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:20.136904 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhjhhq" Sep 17 01:30:20.137351 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:20.136955 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5675a8f6ce14d6fade0460387e388f1573ab557f25b5762241bd4dce4fb42a22" Sep 17 01:30:22.149203 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:22.149160 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9" event={"ID":"635e80dd-2525-4e5b-9a0c-e8ff2db390fa","Type":"ContainerStarted","Data":"16bef93798bf6bffbfcc665836b843dcf901c87613ef2127abcb1dff998f0604"} Sep 17 01:30:22.149608 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:22.149326 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9" Sep 17 01:30:22.164223 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:22.164177 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9" podStartSLOduration=1.695786993 podStartE2EDuration="4.164162211s" totalImagesPullingTime="2.468375218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:30:22.162521348 +0000 UTC m=+487.397668538" watchObservedRunningTime="2026-09-17 01:30:22.164162211 +0000 UTC m=+487.399309400" Sep 17 01:30:28.958857 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:28.958818 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4"] Sep 17 01:30:28.959260 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:28.959193 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd" containerName="util" Sep 17 01:30:28.959260 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:28.959206 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd" containerName="util" Sep 17 01:30:28.959260 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:28.959215 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd" containerName="extract" Sep 17 01:30:28.959260 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:28.959222 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd" containerName="extract" Sep 17 01:30:28.959260 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:28.959243 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd" containerName="pull" Sep 17 01:30:28.959260 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:28.959249 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd" containerName="pull" Sep 17 01:30:28.959448 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:28.959302 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="36c35d4b-ccf0-48f3-bfc8-bce4eccfaabd" containerName="extract" Sep 17 01:30:28.962548 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:28.962531 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4" Sep 17 01:30:28.964752 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:28.964728 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7clft\"" Sep 17 01:30:28.964752 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:28.964742 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 01:30:28.964956 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:28.964774 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 01:30:28.971012 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:28.970983 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4"] Sep 17 01:30:29.068373 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:29.068333 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftzsz\" (UniqueName: \"kubernetes.io/projected/2172cba1-81a1-489a-b546-b6b2b4a4d944-kube-api-access-ftzsz\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4\" (UID: \"2172cba1-81a1-489a-b546-b6b2b4a4d944\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4" Sep 17 01:30:29.068373 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:29.068373 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2172cba1-81a1-489a-b546-b6b2b4a4d944-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4\" (UID: \"2172cba1-81a1-489a-b546-b6b2b4a4d944\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4" Sep 17 01:30:29.068572 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:29.068463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2172cba1-81a1-489a-b546-b6b2b4a4d944-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4\" (UID: \"2172cba1-81a1-489a-b546-b6b2b4a4d944\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4" Sep 17 01:30:29.169722 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:29.169684 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2172cba1-81a1-489a-b546-b6b2b4a4d944-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4\" (UID: \"2172cba1-81a1-489a-b546-b6b2b4a4d944\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4" Sep 17 01:30:29.169886 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:29.169749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftzsz\" (UniqueName: \"kubernetes.io/projected/2172cba1-81a1-489a-b546-b6b2b4a4d944-kube-api-access-ftzsz\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4\" (UID: \"2172cba1-81a1-489a-b546-b6b2b4a4d944\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4" Sep 17 01:30:29.169886 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:29.169772 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2172cba1-81a1-489a-b546-b6b2b4a4d944-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4\" (UID: \"2172cba1-81a1-489a-b546-b6b2b4a4d944\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4" Sep 17 01:30:29.170187 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:29.170166 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2172cba1-81a1-489a-b546-b6b2b4a4d944-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4\" (UID: \"2172cba1-81a1-489a-b546-b6b2b4a4d944\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4" Sep 17 01:30:29.170233 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:29.170173 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2172cba1-81a1-489a-b546-b6b2b4a4d944-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4\" (UID: \"2172cba1-81a1-489a-b546-b6b2b4a4d944\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4" Sep 17 01:30:29.181812 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:29.181776 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftzsz\" (UniqueName: \"kubernetes.io/projected/2172cba1-81a1-489a-b546-b6b2b4a4d944-kube-api-access-ftzsz\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4\" (UID: \"2172cba1-81a1-489a-b546-b6b2b4a4d944\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4" Sep 17 01:30:29.272999 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:29.272900 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4" Sep 17 01:30:29.548063 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:29.548033 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4"] Sep 17 01:30:29.550676 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:30:29.550647 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2172cba1_81a1_489a_b546_b6b2b4a4d944.slice/crio-d9c593c7364f46c74d501d8105381f1005f1f8d75d17c2ae3478b8c83e46861b WatchSource:0}: Error finding container d9c593c7364f46c74d501d8105381f1005f1f8d75d17c2ae3478b8c83e46861b: Status 404 returned error can't find the container with id d9c593c7364f46c74d501d8105381f1005f1f8d75d17c2ae3478b8c83e46861b Sep 17 01:30:30.188809 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:30.188781 2577 generic.go:356] "Generic (PLEG): container finished" podID="2172cba1-81a1-489a-b546-b6b2b4a4d944" containerID="dcdcfd7354eff69136572c43f2a4ebb747709a08c582611b5f07cabdfe378055" exitCode=0 Sep 17 01:30:30.189174 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:30.188866 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4" event={"ID":"2172cba1-81a1-489a-b546-b6b2b4a4d944","Type":"ContainerDied","Data":"dcdcfd7354eff69136572c43f2a4ebb747709a08c582611b5f07cabdfe378055"} Sep 17 01:30:30.189174 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:30.188950 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4" event={"ID":"2172cba1-81a1-489a-b546-b6b2b4a4d944","Type":"ContainerStarted","Data":"d9c593c7364f46c74d501d8105381f1005f1f8d75d17c2ae3478b8c83e46861b"} Sep 17 01:30:31.017836 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.017796 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t"] Sep 17 01:30:31.021596 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.021568 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" Sep 17 01:30:31.023801 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.023773 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 17 01:30:31.023801 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.023798 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 17 01:30:31.023984 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.023797 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 17 01:30:31.023984 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.023831 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-x4nvm\"" Sep 17 01:30:31.023984 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.023846 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 17 01:30:31.041437 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.041160 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t"] Sep 17 01:30:31.086497 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.086459 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24d8f922-bd35-4b68-b549-e801ac074c1a-tmp\") pod \"opendatahub-operator-controller-manager-57c6978585-vxh8t\" (UID: \"24d8f922-bd35-4b68-b549-e801ac074c1a\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" Sep 17 01:30:31.086673 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.086510 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/24d8f922-bd35-4b68-b549-e801ac074c1a-manifests\") pod \"opendatahub-operator-controller-manager-57c6978585-vxh8t\" (UID: \"24d8f922-bd35-4b68-b549-e801ac074c1a\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" Sep 17 01:30:31.086673 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.086588 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24d8f922-bd35-4b68-b549-e801ac074c1a-webhook-cert\") pod \"opendatahub-operator-controller-manager-57c6978585-vxh8t\" (UID: \"24d8f922-bd35-4b68-b549-e801ac074c1a\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" Sep 17 01:30:31.086673 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.086622 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdckp\" (UniqueName: \"kubernetes.io/projected/24d8f922-bd35-4b68-b549-e801ac074c1a-kube-api-access-qdckp\") pod \"opendatahub-operator-controller-manager-57c6978585-vxh8t\" (UID: \"24d8f922-bd35-4b68-b549-e801ac074c1a\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" Sep 17 01:30:31.086673 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.086654 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/24d8f922-bd35-4b68-b549-e801ac074c1a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57c6978585-vxh8t\" (UID: \"24d8f922-bd35-4b68-b549-e801ac074c1a\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" Sep 17 01:30:31.187180 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.187141 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/24d8f922-bd35-4b68-b549-e801ac074c1a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57c6978585-vxh8t\" (UID: \"24d8f922-bd35-4b68-b549-e801ac074c1a\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" Sep 17 01:30:31.187350 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.187224 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24d8f922-bd35-4b68-b549-e801ac074c1a-tmp\") pod \"opendatahub-operator-controller-manager-57c6978585-vxh8t\" (UID: \"24d8f922-bd35-4b68-b549-e801ac074c1a\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" Sep 17 01:30:31.187350 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.187275 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/24d8f922-bd35-4b68-b549-e801ac074c1a-manifests\") pod \"opendatahub-operator-controller-manager-57c6978585-vxh8t\" (UID: \"24d8f922-bd35-4b68-b549-e801ac074c1a\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" Sep 17 01:30:31.187350 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.187317 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24d8f922-bd35-4b68-b549-e801ac074c1a-webhook-cert\") pod \"opendatahub-operator-controller-manager-57c6978585-vxh8t\" (UID: \"24d8f922-bd35-4b68-b549-e801ac074c1a\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" Sep 17 01:30:31.187350 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.187342 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdckp\" (UniqueName: \"kubernetes.io/projected/24d8f922-bd35-4b68-b549-e801ac074c1a-kube-api-access-qdckp\") pod \"opendatahub-operator-controller-manager-57c6978585-vxh8t\" (UID: \"24d8f922-bd35-4b68-b549-e801ac074c1a\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" Sep 17 01:30:31.187761 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.187728 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/24d8f922-bd35-4b68-b549-e801ac074c1a-manifests\") pod \"opendatahub-operator-controller-manager-57c6978585-vxh8t\" (UID: \"24d8f922-bd35-4b68-b549-e801ac074c1a\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" Sep 17 01:30:31.189773 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.189741 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24d8f922-bd35-4b68-b549-e801ac074c1a-tmp\") pod \"opendatahub-operator-controller-manager-57c6978585-vxh8t\" (UID: \"24d8f922-bd35-4b68-b549-e801ac074c1a\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" Sep 17 01:30:31.190175 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.189904 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/24d8f922-bd35-4b68-b549-e801ac074c1a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57c6978585-vxh8t\" (UID: \"24d8f922-bd35-4b68-b549-e801ac074c1a\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" Sep 17 01:30:31.190175 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.189925 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24d8f922-bd35-4b68-b549-e801ac074c1a-webhook-cert\") pod \"opendatahub-operator-controller-manager-57c6978585-vxh8t\" (UID: \"24d8f922-bd35-4b68-b549-e801ac074c1a\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" Sep 17 01:30:31.196170 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.196150 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdckp\" (UniqueName: \"kubernetes.io/projected/24d8f922-bd35-4b68-b549-e801ac074c1a-kube-api-access-qdckp\") pod \"opendatahub-operator-controller-manager-57c6978585-vxh8t\" (UID: \"24d8f922-bd35-4b68-b549-e801ac074c1a\") " pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" Sep 17 01:30:31.196264 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.196202 2577 generic.go:356] "Generic (PLEG): container finished" podID="2172cba1-81a1-489a-b546-b6b2b4a4d944" containerID="8231898d4c65b6e93cd89d1a904daf7ffde6b1214f6499146f49e7328b29cfab" exitCode=0 Sep 17 01:30:31.196327 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.196310 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4" event={"ID":"2172cba1-81a1-489a-b546-b6b2b4a4d944","Type":"ContainerDied","Data":"8231898d4c65b6e93cd89d1a904daf7ffde6b1214f6499146f49e7328b29cfab"} Sep 17 01:30:31.332224 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.332190 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" Sep 17 01:30:31.612558 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:31.612531 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t"] Sep 17 01:30:31.614664 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:30:31.614636 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d8f922_bd35_4b68_b549_e801ac074c1a.slice/crio-8dad2903399ec6c4606dc924efac95735d65a4b65f900730ce24b7315b5b2bb5 WatchSource:0}: Error finding container 8dad2903399ec6c4606dc924efac95735d65a4b65f900730ce24b7315b5b2bb5: Status 404 returned error can't find the container with id 8dad2903399ec6c4606dc924efac95735d65a4b65f900730ce24b7315b5b2bb5 Sep 17 01:30:32.206401 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:32.206316 2577 generic.go:356] "Generic (PLEG): container finished" podID="2172cba1-81a1-489a-b546-b6b2b4a4d944" containerID="74370996c0c56ce1f80c0a3e03a534647e7e3024fca4527d7f140ca52d87f1dd" exitCode=0 Sep 17 01:30:32.206887 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:32.206401 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4" event={"ID":"2172cba1-81a1-489a-b546-b6b2b4a4d944","Type":"ContainerDied","Data":"74370996c0c56ce1f80c0a3e03a534647e7e3024fca4527d7f140ca52d87f1dd"} Sep 17 01:30:32.208747 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:32.208698 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" event={"ID":"24d8f922-bd35-4b68-b549-e801ac074c1a","Type":"ContainerStarted","Data":"8dad2903399ec6c4606dc924efac95735d65a4b65f900730ce24b7315b5b2bb5"} Sep 17 01:30:33.156525 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:33.156484 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5749dbf97f-f95v9" Sep 17 01:30:34.542164 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:34.542141 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4" Sep 17 01:30:34.619025 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:34.618982 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2172cba1-81a1-489a-b546-b6b2b4a4d944-util\") pod \"2172cba1-81a1-489a-b546-b6b2b4a4d944\" (UID: \"2172cba1-81a1-489a-b546-b6b2b4a4d944\") " Sep 17 01:30:34.619109 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:34.619084 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2172cba1-81a1-489a-b546-b6b2b4a4d944-bundle\") pod \"2172cba1-81a1-489a-b546-b6b2b4a4d944\" (UID: \"2172cba1-81a1-489a-b546-b6b2b4a4d944\") " Sep 17 01:30:34.619174 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:34.619146 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftzsz\" (UniqueName: \"kubernetes.io/projected/2172cba1-81a1-489a-b546-b6b2b4a4d944-kube-api-access-ftzsz\") pod \"2172cba1-81a1-489a-b546-b6b2b4a4d944\" (UID: \"2172cba1-81a1-489a-b546-b6b2b4a4d944\") " Sep 17 01:30:34.620116 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:34.620080 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2172cba1-81a1-489a-b546-b6b2b4a4d944-bundle" (OuterVolumeSpecName: "bundle") pod "2172cba1-81a1-489a-b546-b6b2b4a4d944" (UID: "2172cba1-81a1-489a-b546-b6b2b4a4d944"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:30:34.621501 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:34.621469 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2172cba1-81a1-489a-b546-b6b2b4a4d944-kube-api-access-ftzsz" (OuterVolumeSpecName: "kube-api-access-ftzsz") pod "2172cba1-81a1-489a-b546-b6b2b4a4d944" (UID: "2172cba1-81a1-489a-b546-b6b2b4a4d944"). InnerVolumeSpecName "kube-api-access-ftzsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:30:34.626083 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:34.626060 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2172cba1-81a1-489a-b546-b6b2b4a4d944-util" (OuterVolumeSpecName: "util") pod "2172cba1-81a1-489a-b546-b6b2b4a4d944" (UID: "2172cba1-81a1-489a-b546-b6b2b4a4d944"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:30:34.720219 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:34.720182 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftzsz\" (UniqueName: \"kubernetes.io/projected/2172cba1-81a1-489a-b546-b6b2b4a4d944-kube-api-access-ftzsz\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:30:34.720219 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:34.720215 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2172cba1-81a1-489a-b546-b6b2b4a4d944-util\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:30:34.720219 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:34.720227 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2172cba1-81a1-489a-b546-b6b2b4a4d944-bundle\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:30:35.235211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:35.235169 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" event={"ID":"24d8f922-bd35-4b68-b549-e801ac074c1a","Type":"ContainerStarted","Data":"a1140c6063ff54f20d4653eb5dac9d263adbaff4b9019016552a15ace843620c"} Sep 17 01:30:35.237270 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:35.237246 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4" Sep 17 01:30:35.237398 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:35.237249 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0x78h4" event={"ID":"2172cba1-81a1-489a-b546-b6b2b4a4d944","Type":"ContainerDied","Data":"d9c593c7364f46c74d501d8105381f1005f1f8d75d17c2ae3478b8c83e46861b"} Sep 17 01:30:35.237398 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:35.237357 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9c593c7364f46c74d501d8105381f1005f1f8d75d17c2ae3478b8c83e46861b" Sep 17 01:30:36.244210 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:36.244178 2577 generic.go:356] "Generic (PLEG): container finished" podID="24d8f922-bd35-4b68-b549-e801ac074c1a" containerID="a1140c6063ff54f20d4653eb5dac9d263adbaff4b9019016552a15ace843620c" exitCode=0 Sep 17 01:30:36.244604 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:36.244260 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" event={"ID":"24d8f922-bd35-4b68-b549-e801ac074c1a","Type":"ContainerDied","Data":"a1140c6063ff54f20d4653eb5dac9d263adbaff4b9019016552a15ace843620c"} Sep 17 01:30:37.251719 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:37.251684 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" event={"ID":"24d8f922-bd35-4b68-b549-e801ac074c1a","Type":"ContainerStarted","Data":"eb872d16b3f4fdbc2de6ff2f805109fc4b975532b58ec3d191e4d9f9bb61c623"} Sep 17 01:30:37.252164 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:37.251797 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" Sep 17 01:30:37.273099 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:37.273047 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" podStartSLOduration=2.802979865 podStartE2EDuration="7.273025383s" totalImagesPullingTime="3.290181559s" totalInitContainerRuntime="1.179863959s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:30:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:30:37.270673497 +0000 UTC m=+502.505820685" watchObservedRunningTime="2026-09-17 01:30:37.273025383 +0000 UTC m=+502.508172572" Sep 17 01:30:48.259200 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:48.259163 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-57c6978585-vxh8t" Sep 17 01:30:57.291003 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.290967 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn"] Sep 17 01:30:57.291478 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.291319 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2172cba1-81a1-489a-b546-b6b2b4a4d944" containerName="pull" Sep 17 01:30:57.291478 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.291330 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2172cba1-81a1-489a-b546-b6b2b4a4d944" containerName="pull" Sep 17 01:30:57.291478 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.291348 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2172cba1-81a1-489a-b546-b6b2b4a4d944" containerName="extract" Sep 17 01:30:57.291478 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.291354 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2172cba1-81a1-489a-b546-b6b2b4a4d944" containerName="extract" Sep 17 01:30:57.291478 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.291362 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2172cba1-81a1-489a-b546-b6b2b4a4d944" containerName="util" Sep 17 01:30:57.291478 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.291367 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2172cba1-81a1-489a-b546-b6b2b4a4d944" containerName="util" Sep 17 01:30:57.291478 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.291445 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2172cba1-81a1-489a-b546-b6b2b4a4d944" containerName="extract" Sep 17 01:30:57.300058 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.300031 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.302083 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.302059 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 17 01:30:57.302220 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.302140 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-mzmww\"" Sep 17 01:30:57.306294 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.306268 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn"] Sep 17 01:30:57.417628 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.417590 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad0996a8-d893-486b-a3ee-4ee366bc558b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.417808 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.417651 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad0996a8-d893-486b-a3ee-4ee366bc558b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.417808 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.417706 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0996a8-d893-486b-a3ee-4ee366bc558b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.417808 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.417740 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79zbn\" (UniqueName: \"kubernetes.io/projected/ad0996a8-d893-486b-a3ee-4ee366bc558b-kube-api-access-79zbn\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.417808 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.417769 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad0996a8-d893-486b-a3ee-4ee366bc558b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.417808 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.417798 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad0996a8-d893-486b-a3ee-4ee366bc558b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.418069 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.417826 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad0996a8-d893-486b-a3ee-4ee366bc558b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.418069 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.417849 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0996a8-d893-486b-a3ee-4ee366bc558b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.418069 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.417873 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad0996a8-d893-486b-a3ee-4ee366bc558b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.518689 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.518653 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad0996a8-d893-486b-a3ee-4ee366bc558b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.518873 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.518711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad0996a8-d893-486b-a3ee-4ee366bc558b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.518873 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.518733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0996a8-d893-486b-a3ee-4ee366bc558b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.518873 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.518749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79zbn\" (UniqueName: \"kubernetes.io/projected/ad0996a8-d893-486b-a3ee-4ee366bc558b-kube-api-access-79zbn\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.518873 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.518782 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad0996a8-d893-486b-a3ee-4ee366bc558b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.518873 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.518821 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad0996a8-d893-486b-a3ee-4ee366bc558b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.518873 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.518851 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad0996a8-d893-486b-a3ee-4ee366bc558b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.519211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.518878 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0996a8-d893-486b-a3ee-4ee366bc558b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.519211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.518914 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad0996a8-d893-486b-a3ee-4ee366bc558b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.519211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.519185 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad0996a8-d893-486b-a3ee-4ee366bc558b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.519372 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.519285 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0996a8-d893-486b-a3ee-4ee366bc558b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.519372 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.519327 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad0996a8-d893-486b-a3ee-4ee366bc558b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.519480 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.519383 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0996a8-d893-486b-a3ee-4ee366bc558b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.519745 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.519720 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad0996a8-d893-486b-a3ee-4ee366bc558b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.521240 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.521216 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad0996a8-d893-486b-a3ee-4ee366bc558b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.521616 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.521592 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad0996a8-d893-486b-a3ee-4ee366bc558b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.525965 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.525922 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad0996a8-d893-486b-a3ee-4ee366bc558b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.526160 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.526139 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79zbn\" (UniqueName: \"kubernetes.io/projected/ad0996a8-d893-486b-a3ee-4ee366bc558b-kube-api-access-79zbn\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn\" (UID: \"ad0996a8-d893-486b-a3ee-4ee366bc558b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.611725 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.611684 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:30:57.883903 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:57.883874 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn"] Sep 17 01:30:57.885471 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:30:57.885438 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0996a8_d893_486b_a3ee_4ee366bc558b.slice/crio-6e2363f6086e4d9467fb8167e665b21f3ec1defc02b1c6f213fb1f796189c0bc WatchSource:0}: Error finding container 6e2363f6086e4d9467fb8167e665b21f3ec1defc02b1c6f213fb1f796189c0bc: Status 404 returned error can't find the container with id 6e2363f6086e4d9467fb8167e665b21f3ec1defc02b1c6f213fb1f796189c0bc Sep 17 01:30:58.354680 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:30:58.354638 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" event={"ID":"ad0996a8-d893-486b-a3ee-4ee366bc558b","Type":"ContainerStarted","Data":"6e2363f6086e4d9467fb8167e665b21f3ec1defc02b1c6f213fb1f796189c0bc"} Sep 17 01:31:01.103920 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.103881 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:31:01.104180 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.103975 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:31:01.104180 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.104004 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:31:01.157708 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.157679 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9x5vs"] Sep 17 01:31:01.163109 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.163089 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9x5vs" Sep 17 01:31:01.166083 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.166062 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 17 01:31:01.169327 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.169280 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-gdh67\"" Sep 17 01:31:01.178474 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.178412 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9x5vs"] Sep 17 01:31:01.256922 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.256882 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d0d0ca1-0f07-4b70-b26c-57821ec3e927-cert\") pod \"odh-model-controller-7998c67dc4-9x5vs\" (UID: \"6d0d0ca1-0f07-4b70-b26c-57821ec3e927\") " pod="opendatahub/odh-model-controller-7998c67dc4-9x5vs" Sep 17 01:31:01.257107 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.257048 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgjm4\" (UniqueName: \"kubernetes.io/projected/6d0d0ca1-0f07-4b70-b26c-57821ec3e927-kube-api-access-wgjm4\") pod \"odh-model-controller-7998c67dc4-9x5vs\" (UID: \"6d0d0ca1-0f07-4b70-b26c-57821ec3e927\") " pod="opendatahub/odh-model-controller-7998c67dc4-9x5vs" Sep 17 01:31:01.357716 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.357636 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgjm4\" (UniqueName: \"kubernetes.io/projected/6d0d0ca1-0f07-4b70-b26c-57821ec3e927-kube-api-access-wgjm4\") pod \"odh-model-controller-7998c67dc4-9x5vs\" (UID: \"6d0d0ca1-0f07-4b70-b26c-57821ec3e927\") " pod="opendatahub/odh-model-controller-7998c67dc4-9x5vs" Sep 17 01:31:01.357716 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.357686 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d0d0ca1-0f07-4b70-b26c-57821ec3e927-cert\") pod \"odh-model-controller-7998c67dc4-9x5vs\" (UID: \"6d0d0ca1-0f07-4b70-b26c-57821ec3e927\") " pod="opendatahub/odh-model-controller-7998c67dc4-9x5vs" Sep 17 01:31:01.360078 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.360050 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d0d0ca1-0f07-4b70-b26c-57821ec3e927-cert\") pod \"odh-model-controller-7998c67dc4-9x5vs\" (UID: \"6d0d0ca1-0f07-4b70-b26c-57821ec3e927\") " pod="opendatahub/odh-model-controller-7998c67dc4-9x5vs" Sep 17 01:31:01.364955 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.364920 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgjm4\" (UniqueName: \"kubernetes.io/projected/6d0d0ca1-0f07-4b70-b26c-57821ec3e927-kube-api-access-wgjm4\") pod \"odh-model-controller-7998c67dc4-9x5vs\" (UID: \"6d0d0ca1-0f07-4b70-b26c-57821ec3e927\") " pod="opendatahub/odh-model-controller-7998c67dc4-9x5vs" Sep 17 01:31:01.373228 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.373199 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" event={"ID":"ad0996a8-d893-486b-a3ee-4ee366bc558b","Type":"ContainerStarted","Data":"fd566e99658045d3b7b9f4f8f8366bc8c3a1cc6581fd9529ea2e0a88b80eb9c0"} Sep 17 01:31:01.394146 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.394095 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" podStartSLOduration=1.17818646 podStartE2EDuration="4.39407269s" totalImagesPullingTime="3.21588623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:31:01.392395967 +0000 UTC m=+526.627543157" watchObservedRunningTime="2026-09-17 01:31:01.39407269 +0000 UTC m=+526.629219878" Sep 17 01:31:01.461876 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.461845 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-69vh5"] Sep 17 01:31:01.465521 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.465503 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-69vh5" Sep 17 01:31:01.467366 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.467339 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 17 01:31:01.467495 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.467477 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-mzmff\"" Sep 17 01:31:01.472545 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.472518 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9x5vs" Sep 17 01:31:01.475091 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.475063 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-69vh5"] Sep 17 01:31:01.558796 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.558760 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6d5x\" (UniqueName: \"kubernetes.io/projected/5ea401cd-94c4-4dba-94a7-b08cd781569b-kube-api-access-p6d5x\") pod \"model-serving-api-54d89bd5fc-69vh5\" (UID: \"5ea401cd-94c4-4dba-94a7-b08cd781569b\") " pod="opendatahub/model-serving-api-54d89bd5fc-69vh5" Sep 17 01:31:01.558991 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.558859 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea401cd-94c4-4dba-94a7-b08cd781569b-tls-certs\") pod \"model-serving-api-54d89bd5fc-69vh5\" (UID: \"5ea401cd-94c4-4dba-94a7-b08cd781569b\") " pod="opendatahub/model-serving-api-54d89bd5fc-69vh5" Sep 17 01:31:01.611953 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.611857 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:31:01.659352 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.659319 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6d5x\" (UniqueName: \"kubernetes.io/projected/5ea401cd-94c4-4dba-94a7-b08cd781569b-kube-api-access-p6d5x\") pod \"model-serving-api-54d89bd5fc-69vh5\" (UID: \"5ea401cd-94c4-4dba-94a7-b08cd781569b\") " pod="opendatahub/model-serving-api-54d89bd5fc-69vh5" Sep 17 01:31:01.659511 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.659373 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea401cd-94c4-4dba-94a7-b08cd781569b-tls-certs\") pod \"model-serving-api-54d89bd5fc-69vh5\" (UID: \"5ea401cd-94c4-4dba-94a7-b08cd781569b\") " pod="opendatahub/model-serving-api-54d89bd5fc-69vh5" Sep 17 01:31:01.659511 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:31:01.659505 2577 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 17 01:31:01.659614 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:31:01.659557 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ea401cd-94c4-4dba-94a7-b08cd781569b-tls-certs podName:5ea401cd-94c4-4dba-94a7-b08cd781569b nodeName:}" failed. No retries permitted until 2026-09-17 01:31:02.159541603 +0000 UTC m=+527.394688769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5ea401cd-94c4-4dba-94a7-b08cd781569b-tls-certs") pod "model-serving-api-54d89bd5fc-69vh5" (UID: "5ea401cd-94c4-4dba-94a7-b08cd781569b") : secret "model-serving-api-tls" not found Sep 17 01:31:01.667534 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.667511 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6d5x\" (UniqueName: \"kubernetes.io/projected/5ea401cd-94c4-4dba-94a7-b08cd781569b-kube-api-access-p6d5x\") pod \"model-serving-api-54d89bd5fc-69vh5\" (UID: \"5ea401cd-94c4-4dba-94a7-b08cd781569b\") " pod="opendatahub/model-serving-api-54d89bd5fc-69vh5" Sep 17 01:31:01.743917 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:01.743885 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9x5vs"] Sep 17 01:31:01.745777 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:31:01.745755 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0d0ca1_0f07_4b70_b26c_57821ec3e927.slice/crio-ec8188e93c27435bdb37db827d879dbd198bb095c636b767874aded35f51836b WatchSource:0}: Error finding container ec8188e93c27435bdb37db827d879dbd198bb095c636b767874aded35f51836b: Status 404 returned error can't find the container with id ec8188e93c27435bdb37db827d879dbd198bb095c636b767874aded35f51836b Sep 17 01:31:02.163698 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:02.163658 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea401cd-94c4-4dba-94a7-b08cd781569b-tls-certs\") pod \"model-serving-api-54d89bd5fc-69vh5\" (UID: \"5ea401cd-94c4-4dba-94a7-b08cd781569b\") " pod="opendatahub/model-serving-api-54d89bd5fc-69vh5" Sep 17 01:31:02.166139 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:02.166116 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea401cd-94c4-4dba-94a7-b08cd781569b-tls-certs\") pod \"model-serving-api-54d89bd5fc-69vh5\" (UID: \"5ea401cd-94c4-4dba-94a7-b08cd781569b\") " pod="opendatahub/model-serving-api-54d89bd5fc-69vh5" Sep 17 01:31:02.376831 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:02.376795 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-69vh5" Sep 17 01:31:02.379581 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:02.379547 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9x5vs" event={"ID":"6d0d0ca1-0f07-4b70-b26c-57821ec3e927","Type":"ContainerStarted","Data":"ec8188e93c27435bdb37db827d879dbd198bb095c636b767874aded35f51836b"} Sep 17 01:31:02.618828 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:02.618795 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:31:02.650263 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:02.650234 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-69vh5"] Sep 17 01:31:02.652782 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:31:02.652748 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea401cd_94c4_4dba_94a7_b08cd781569b.slice/crio-b0a2e5e47a7fd0314409b4ae626ed2a8464d8642537e6c5d1aeada501dbf852f WatchSource:0}: Error finding container b0a2e5e47a7fd0314409b4ae626ed2a8464d8642537e6c5d1aeada501dbf852f: Status 404 returned error can't find the container with id b0a2e5e47a7fd0314409b4ae626ed2a8464d8642537e6c5d1aeada501dbf852f Sep 17 01:31:03.391676 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:03.391631 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-69vh5" event={"ID":"5ea401cd-94c4-4dba-94a7-b08cd781569b","Type":"ContainerStarted","Data":"b0a2e5e47a7fd0314409b4ae626ed2a8464d8642537e6c5d1aeada501dbf852f"} Sep 17 01:31:03.392178 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:03.392133 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:31:03.393358 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:03.393335 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn" Sep 17 01:31:06.409623 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:06.409585 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-69vh5" event={"ID":"5ea401cd-94c4-4dba-94a7-b08cd781569b","Type":"ContainerStarted","Data":"634a2822015096757b1d428486d27a6ec6eef3876e9f185f496b54c6c2e65efe"} Sep 17 01:31:06.410060 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:06.409688 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-69vh5" Sep 17 01:31:06.411129 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:06.411110 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9x5vs" event={"ID":"6d0d0ca1-0f07-4b70-b26c-57821ec3e927","Type":"ContainerStarted","Data":"d7fae59e66541c1d084eb2d939cb20b670a677a58ad92a1f73870e245143a478"} Sep 17 01:31:06.411194 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:06.411151 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-9x5vs" Sep 17 01:31:06.424847 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:06.424801 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-69vh5" podStartSLOduration=2.190039644 podStartE2EDuration="5.424789233s" totalImagesPullingTime="3.234749589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:31:06.42285985 +0000 UTC m=+531.658007039" watchObservedRunningTime="2026-09-17 01:31:06.424789233 +0000 UTC m=+531.659936420" Sep 17 01:31:06.436562 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:06.436509 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-9x5vs" podStartSLOduration=1.288085551 podStartE2EDuration="5.436496152s" totalImagesPullingTime="4.148410601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:31:06.436237535 +0000 UTC m=+531.671384726" watchObservedRunningTime="2026-09-17 01:31:06.436496152 +0000 UTC m=+531.671643377" Sep 17 01:31:17.420339 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:17.420306 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-9x5vs" Sep 17 01:31:17.422359 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:17.422335 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-69vh5" Sep 17 01:31:43.830406 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:43.830357 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c95c4f55d-sg92x"] Sep 17 01:31:43.834454 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:43.834428 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:43.844131 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:43.844102 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c95c4f55d-sg92x"] Sep 17 01:31:43.932421 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:43.932380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd98a571-4bfe-44d5-973b-74a3758758f7-service-ca\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:43.932421 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:43.932424 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd98a571-4bfe-44d5-973b-74a3758758f7-oauth-serving-cert\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:43.932646 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:43.932516 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd98a571-4bfe-44d5-973b-74a3758758f7-trusted-ca-bundle\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:43.932646 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:43.932563 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd98a571-4bfe-44d5-973b-74a3758758f7-console-serving-cert\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:43.932646 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:43.932582 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd98a571-4bfe-44d5-973b-74a3758758f7-console-oauth-config\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:43.932646 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:43.932599 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-558hx\" (UniqueName: \"kubernetes.io/projected/dd98a571-4bfe-44d5-973b-74a3758758f7-kube-api-access-558hx\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:43.932820 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:43.932666 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd98a571-4bfe-44d5-973b-74a3758758f7-console-config\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:44.033568 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:44.033518 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd98a571-4bfe-44d5-973b-74a3758758f7-trusted-ca-bundle\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:44.033568 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:44.033577 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd98a571-4bfe-44d5-973b-74a3758758f7-console-serving-cert\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:44.033834 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:44.033595 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd98a571-4bfe-44d5-973b-74a3758758f7-console-oauth-config\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:44.033834 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:44.033617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-558hx\" (UniqueName: \"kubernetes.io/projected/dd98a571-4bfe-44d5-973b-74a3758758f7-kube-api-access-558hx\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:44.033834 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:44.033648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd98a571-4bfe-44d5-973b-74a3758758f7-console-config\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:44.033834 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:44.033728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd98a571-4bfe-44d5-973b-74a3758758f7-service-ca\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:44.033834 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:44.033754 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd98a571-4bfe-44d5-973b-74a3758758f7-oauth-serving-cert\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:44.034519 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:44.034495 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd98a571-4bfe-44d5-973b-74a3758758f7-oauth-serving-cert\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:44.034616 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:44.034497 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd98a571-4bfe-44d5-973b-74a3758758f7-trusted-ca-bundle\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:44.034616 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:44.034535 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd98a571-4bfe-44d5-973b-74a3758758f7-console-config\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:44.034685 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:44.034634 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd98a571-4bfe-44d5-973b-74a3758758f7-service-ca\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:44.036505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:44.036481 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd98a571-4bfe-44d5-973b-74a3758758f7-console-oauth-config\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:44.036598 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:44.036571 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd98a571-4bfe-44d5-973b-74a3758758f7-console-serving-cert\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:44.041033 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:44.041016 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-558hx\" (UniqueName: \"kubernetes.io/projected/dd98a571-4bfe-44d5-973b-74a3758758f7-kube-api-access-558hx\") pod \"console-6c95c4f55d-sg92x\" (UID: \"dd98a571-4bfe-44d5-973b-74a3758758f7\") " pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:44.146996 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:44.146951 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:44.416500 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:44.416468 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c95c4f55d-sg92x"] Sep 17 01:31:44.418147 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:31:44.418118 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd98a571_4bfe_44d5_973b_74a3758758f7.slice/crio-f3faeb020e5c3db7379ece40165c54e39751fa6fd37d9edeea4f4c6689bc5533 WatchSource:0}: Error finding container f3faeb020e5c3db7379ece40165c54e39751fa6fd37d9edeea4f4c6689bc5533: Status 404 returned error can't find the container with id f3faeb020e5c3db7379ece40165c54e39751fa6fd37d9edeea4f4c6689bc5533 Sep 17 01:31:44.605190 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:44.605153 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c95c4f55d-sg92x" event={"ID":"dd98a571-4bfe-44d5-973b-74a3758758f7","Type":"ContainerStarted","Data":"6675743236deac7db070a150979da80376a73f45abc86296bd5c631a8a9eb6db"} Sep 17 01:31:44.605370 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:44.605201 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c95c4f55d-sg92x" event={"ID":"dd98a571-4bfe-44d5-973b-74a3758758f7","Type":"ContainerStarted","Data":"f3faeb020e5c3db7379ece40165c54e39751fa6fd37d9edeea4f4c6689bc5533"} Sep 17 01:31:44.623978 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:44.623917 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c95c4f55d-sg92x" podStartSLOduration=1.6239014489999999 podStartE2EDuration="1.623901449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:31:44.62117748 +0000 UTC m=+569.856324678" watchObservedRunningTime="2026-09-17 01:31:44.623901449 +0000 UTC m=+569.859048636" Sep 17 01:31:54.147588 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:54.147546 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:54.148097 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:54.147617 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:54.152521 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:54.152497 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:54.662216 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:54.662186 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c95c4f55d-sg92x" Sep 17 01:31:54.708253 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:31:54.708204 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fdc664585-s8pnk"] Sep 17 01:32:05.870101 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:05.870064 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz"] Sep 17 01:32:05.874962 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:05.874188 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz" Sep 17 01:32:05.877081 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:05.877042 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 01:32:05.877538 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:05.877509 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7clft\"" Sep 17 01:32:05.877808 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:05.877787 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 01:32:05.882555 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:05.882530 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz"] Sep 17 01:32:05.940242 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:05.940204 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c3c83f0-258f-4539-99c8-2ee09a09c000-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz\" (UID: \"3c3c83f0-258f-4539-99c8-2ee09a09c000\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz" Sep 17 01:32:05.940465 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:05.940256 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c3c83f0-258f-4539-99c8-2ee09a09c000-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz\" (UID: \"3c3c83f0-258f-4539-99c8-2ee09a09c000\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz" Sep 17 01:32:05.940465 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:05.940338 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x5rz\" (UniqueName: \"kubernetes.io/projected/3c3c83f0-258f-4539-99c8-2ee09a09c000-kube-api-access-9x5rz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz\" (UID: \"3c3c83f0-258f-4539-99c8-2ee09a09c000\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz" Sep 17 01:32:05.971408 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:05.971375 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn"] Sep 17 01:32:05.975113 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:05.975093 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn" Sep 17 01:32:05.983588 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:05.983561 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn"] Sep 17 01:32:06.041385 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.041341 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9799bbd1-9fd6-4b9c-a724-c3545b2aadb2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn\" (UID: \"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn" Sep 17 01:32:06.041590 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.041430 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c3c83f0-258f-4539-99c8-2ee09a09c000-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz\" (UID: \"3c3c83f0-258f-4539-99c8-2ee09a09c000\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz" Sep 17 01:32:06.041590 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.041461 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c3c83f0-258f-4539-99c8-2ee09a09c000-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz\" (UID: \"3c3c83f0-258f-4539-99c8-2ee09a09c000\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz" Sep 17 01:32:06.041590 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.041479 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p46zr\" (UniqueName: \"kubernetes.io/projected/9799bbd1-9fd6-4b9c-a724-c3545b2aadb2-kube-api-access-p46zr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn\" (UID: \"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn" Sep 17 01:32:06.041590 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.041543 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x5rz\" (UniqueName: \"kubernetes.io/projected/3c3c83f0-258f-4539-99c8-2ee09a09c000-kube-api-access-9x5rz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz\" (UID: \"3c3c83f0-258f-4539-99c8-2ee09a09c000\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz" Sep 17 01:32:06.041590 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.041566 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9799bbd1-9fd6-4b9c-a724-c3545b2aadb2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn\" (UID: \"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn" Sep 17 01:32:06.041884 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.041864 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c3c83f0-258f-4539-99c8-2ee09a09c000-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz\" (UID: \"3c3c83f0-258f-4539-99c8-2ee09a09c000\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz" Sep 17 01:32:06.041976 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.041919 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c3c83f0-258f-4539-99c8-2ee09a09c000-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz\" (UID: \"3c3c83f0-258f-4539-99c8-2ee09a09c000\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz" Sep 17 01:32:06.053221 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.053190 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x5rz\" (UniqueName: \"kubernetes.io/projected/3c3c83f0-258f-4539-99c8-2ee09a09c000-kube-api-access-9x5rz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz\" (UID: \"3c3c83f0-258f-4539-99c8-2ee09a09c000\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz" Sep 17 01:32:06.066557 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.066529 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk"] Sep 17 01:32:06.070325 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.070308 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk" Sep 17 01:32:06.078428 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.078404 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk"] Sep 17 01:32:06.142891 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.142803 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wnxl\" (UniqueName: \"kubernetes.io/projected/dbc122b0-4d70-4cc7-ad24-1ffab04cd039-kube-api-access-2wnxl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk\" (UID: \"dbc122b0-4d70-4cc7-ad24-1ffab04cd039\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk" Sep 17 01:32:06.142891 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.142852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbc122b0-4d70-4cc7-ad24-1ffab04cd039-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk\" (UID: \"dbc122b0-4d70-4cc7-ad24-1ffab04cd039\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk" Sep 17 01:32:06.143117 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.142901 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p46zr\" (UniqueName: \"kubernetes.io/projected/9799bbd1-9fd6-4b9c-a724-c3545b2aadb2-kube-api-access-p46zr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn\" (UID: \"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn" Sep 17 01:32:06.143117 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.142972 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbc122b0-4d70-4cc7-ad24-1ffab04cd039-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk\" (UID: \"dbc122b0-4d70-4cc7-ad24-1ffab04cd039\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk" Sep 17 01:32:06.143117 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.143020 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9799bbd1-9fd6-4b9c-a724-c3545b2aadb2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn\" (UID: \"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn" Sep 17 01:32:06.143117 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.143053 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9799bbd1-9fd6-4b9c-a724-c3545b2aadb2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn\" (UID: \"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn" Sep 17 01:32:06.143388 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.143370 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9799bbd1-9fd6-4b9c-a724-c3545b2aadb2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn\" (UID: \"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn" Sep 17 01:32:06.143427 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.143396 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9799bbd1-9fd6-4b9c-a724-c3545b2aadb2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn\" (UID: \"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn" Sep 17 01:32:06.153184 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.153160 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p46zr\" (UniqueName: \"kubernetes.io/projected/9799bbd1-9fd6-4b9c-a724-c3545b2aadb2-kube-api-access-p46zr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn\" (UID: \"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn" Sep 17 01:32:06.172570 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.172538 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8"] Sep 17 01:32:06.176405 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.176387 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8" Sep 17 01:32:06.185453 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.185423 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8"] Sep 17 01:32:06.189907 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.189885 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz" Sep 17 01:32:06.244354 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.244311 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wnxl\" (UniqueName: \"kubernetes.io/projected/dbc122b0-4d70-4cc7-ad24-1ffab04cd039-kube-api-access-2wnxl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk\" (UID: \"dbc122b0-4d70-4cc7-ad24-1ffab04cd039\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk" Sep 17 01:32:06.244561 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.244539 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62352577-a39b-49c3-b12f-079a94e4e342-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8\" (UID: \"62352577-a39b-49c3-b12f-079a94e4e342\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8" Sep 17 01:32:06.244747 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.244721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbc122b0-4d70-4cc7-ad24-1ffab04cd039-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk\" (UID: \"dbc122b0-4d70-4cc7-ad24-1ffab04cd039\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk" Sep 17 01:32:06.244883 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.244773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbc122b0-4d70-4cc7-ad24-1ffab04cd039-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk\" (UID: \"dbc122b0-4d70-4cc7-ad24-1ffab04cd039\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk" Sep 17 01:32:06.244883 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.244865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62352577-a39b-49c3-b12f-079a94e4e342-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8\" (UID: \"62352577-a39b-49c3-b12f-079a94e4e342\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8" Sep 17 01:32:06.245402 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.244918 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf5lf\" (UniqueName: \"kubernetes.io/projected/62352577-a39b-49c3-b12f-079a94e4e342-kube-api-access-rf5lf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8\" (UID: \"62352577-a39b-49c3-b12f-079a94e4e342\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8" Sep 17 01:32:06.245402 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.245188 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbc122b0-4d70-4cc7-ad24-1ffab04cd039-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk\" (UID: \"dbc122b0-4d70-4cc7-ad24-1ffab04cd039\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk" Sep 17 01:32:06.245402 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.245287 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbc122b0-4d70-4cc7-ad24-1ffab04cd039-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk\" (UID: \"dbc122b0-4d70-4cc7-ad24-1ffab04cd039\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk" Sep 17 01:32:06.258647 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.258238 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wnxl\" (UniqueName: \"kubernetes.io/projected/dbc122b0-4d70-4cc7-ad24-1ffab04cd039-kube-api-access-2wnxl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk\" (UID: \"dbc122b0-4d70-4cc7-ad24-1ffab04cd039\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk" Sep 17 01:32:06.285879 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.285852 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn" Sep 17 01:32:06.345587 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.345560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62352577-a39b-49c3-b12f-079a94e4e342-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8\" (UID: \"62352577-a39b-49c3-b12f-079a94e4e342\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8" Sep 17 01:32:06.345722 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.345669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62352577-a39b-49c3-b12f-079a94e4e342-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8\" (UID: \"62352577-a39b-49c3-b12f-079a94e4e342\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8" Sep 17 01:32:06.345722 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.345703 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf5lf\" (UniqueName: \"kubernetes.io/projected/62352577-a39b-49c3-b12f-079a94e4e342-kube-api-access-rf5lf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8\" (UID: \"62352577-a39b-49c3-b12f-079a94e4e342\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8" Sep 17 01:32:06.346099 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.346066 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62352577-a39b-49c3-b12f-079a94e4e342-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8\" (UID: \"62352577-a39b-49c3-b12f-079a94e4e342\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8" Sep 17 01:32:06.346203 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.346180 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62352577-a39b-49c3-b12f-079a94e4e342-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8\" (UID: \"62352577-a39b-49c3-b12f-079a94e4e342\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8" Sep 17 01:32:06.357747 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.357720 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf5lf\" (UniqueName: \"kubernetes.io/projected/62352577-a39b-49c3-b12f-079a94e4e342-kube-api-access-rf5lf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8\" (UID: \"62352577-a39b-49c3-b12f-079a94e4e342\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8" Sep 17 01:32:06.380584 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.380559 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk" Sep 17 01:32:06.464149 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.464120 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz"] Sep 17 01:32:06.465407 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:32:06.465377 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3c83f0_258f_4539_99c8_2ee09a09c000.slice/crio-96c1382964d26cba9bffde1c8851509323a1a19147fa64f1f245719bb12ea797 WatchSource:0}: Error finding container 96c1382964d26cba9bffde1c8851509323a1a19147fa64f1f245719bb12ea797: Status 404 returned error can't find the container with id 96c1382964d26cba9bffde1c8851509323a1a19147fa64f1f245719bb12ea797 Sep 17 01:32:06.486071 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.486046 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8" Sep 17 01:32:06.557495 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.557473 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn"] Sep 17 01:32:06.560242 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:32:06.560208 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9799bbd1_9fd6_4b9c_a724_c3545b2aadb2.slice/crio-d14dd48ea9c213e9b9ed0fc4f50ff91896b90e5b68a6d4fff64e8287cc2af336 WatchSource:0}: Error finding container d14dd48ea9c213e9b9ed0fc4f50ff91896b90e5b68a6d4fff64e8287cc2af336: Status 404 returned error can't find the container with id d14dd48ea9c213e9b9ed0fc4f50ff91896b90e5b68a6d4fff64e8287cc2af336 Sep 17 01:32:06.650487 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.650461 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk"] Sep 17 01:32:06.651538 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:32:06.651508 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc122b0_4d70_4cc7_ad24_1ffab04cd039.slice/crio-525c3b336d5deedae9656aa4eb7d8ca5ac39c7fc6821936a03e9d85d406bef9c WatchSource:0}: Error finding container 525c3b336d5deedae9656aa4eb7d8ca5ac39c7fc6821936a03e9d85d406bef9c: Status 404 returned error can't find the container with id 525c3b336d5deedae9656aa4eb7d8ca5ac39c7fc6821936a03e9d85d406bef9c Sep 17 01:32:06.722053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.722027 2577 generic.go:356] "Generic (PLEG): container finished" podID="3c3c83f0-258f-4539-99c8-2ee09a09c000" containerID="99bba712b69d989e83ee2de7852f3f888326c45f96a383fbfa246edfb223cb07" exitCode=0 Sep 17 01:32:06.722158 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.722110 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz" event={"ID":"3c3c83f0-258f-4539-99c8-2ee09a09c000","Type":"ContainerDied","Data":"99bba712b69d989e83ee2de7852f3f888326c45f96a383fbfa246edfb223cb07"} Sep 17 01:32:06.722211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.722166 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz" event={"ID":"3c3c83f0-258f-4539-99c8-2ee09a09c000","Type":"ContainerStarted","Data":"96c1382964d26cba9bffde1c8851509323a1a19147fa64f1f245719bb12ea797"} Sep 17 01:32:06.724123 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.723908 2577 generic.go:356] "Generic (PLEG): container finished" podID="9799bbd1-9fd6-4b9c-a724-c3545b2aadb2" containerID="26d069caeb3845db02fa589c67ba5022704088097e201c45fae2be66592c2d6a" exitCode=0 Sep 17 01:32:06.724123 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.723968 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn" event={"ID":"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2","Type":"ContainerDied","Data":"26d069caeb3845db02fa589c67ba5022704088097e201c45fae2be66592c2d6a"} Sep 17 01:32:06.724123 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.724009 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn" event={"ID":"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2","Type":"ContainerStarted","Data":"d14dd48ea9c213e9b9ed0fc4f50ff91896b90e5b68a6d4fff64e8287cc2af336"} Sep 17 01:32:06.726063 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.726040 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk" event={"ID":"dbc122b0-4d70-4cc7-ad24-1ffab04cd039","Type":"ContainerStarted","Data":"a06b3d16b0f6059c4adf541d03ce6a867d2ce47d8eab0f6b62fced493cbcc210"} Sep 17 01:32:06.726149 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.726085 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk" event={"ID":"dbc122b0-4d70-4cc7-ad24-1ffab04cd039","Type":"ContainerStarted","Data":"525c3b336d5deedae9656aa4eb7d8ca5ac39c7fc6821936a03e9d85d406bef9c"} Sep 17 01:32:06.756739 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:06.756704 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8"] Sep 17 01:32:06.800820 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:32:06.800787 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62352577_a39b_49c3_b12f_079a94e4e342.slice/crio-b271b623fdf7961c8a45d5f7fc4e28a8a685b5837acb9f5e6501454bfe3a8e73 WatchSource:0}: Error finding container b271b623fdf7961c8a45d5f7fc4e28a8a685b5837acb9f5e6501454bfe3a8e73: Status 404 returned error can't find the container with id b271b623fdf7961c8a45d5f7fc4e28a8a685b5837acb9f5e6501454bfe3a8e73 Sep 17 01:32:07.733269 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:07.733239 2577 generic.go:356] "Generic (PLEG): container finished" podID="62352577-a39b-49c3-b12f-079a94e4e342" containerID="880b627a9a82c5f13aba523b728eedd78375c991fcc2840729aae457b0d4f69f" exitCode=0 Sep 17 01:32:07.733606 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:07.733364 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8" event={"ID":"62352577-a39b-49c3-b12f-079a94e4e342","Type":"ContainerDied","Data":"880b627a9a82c5f13aba523b728eedd78375c991fcc2840729aae457b0d4f69f"} Sep 17 01:32:07.733606 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:07.733449 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8" event={"ID":"62352577-a39b-49c3-b12f-079a94e4e342","Type":"ContainerStarted","Data":"b271b623fdf7961c8a45d5f7fc4e28a8a685b5837acb9f5e6501454bfe3a8e73"} Sep 17 01:32:07.735338 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:07.735320 2577 generic.go:356] "Generic (PLEG): container finished" podID="9799bbd1-9fd6-4b9c-a724-c3545b2aadb2" containerID="5cede30dbb6d58ce4ce051f6aad219e3cdba7ccd5de4d6e79184bdabb86019f9" exitCode=0 Sep 17 01:32:07.735430 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:07.735409 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn" event={"ID":"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2","Type":"ContainerDied","Data":"5cede30dbb6d58ce4ce051f6aad219e3cdba7ccd5de4d6e79184bdabb86019f9"} Sep 17 01:32:07.737084 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:07.737052 2577 generic.go:356] "Generic (PLEG): container finished" podID="dbc122b0-4d70-4cc7-ad24-1ffab04cd039" containerID="a06b3d16b0f6059c4adf541d03ce6a867d2ce47d8eab0f6b62fced493cbcc210" exitCode=0 Sep 17 01:32:07.737164 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:07.737100 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk" event={"ID":"dbc122b0-4d70-4cc7-ad24-1ffab04cd039","Type":"ContainerDied","Data":"a06b3d16b0f6059c4adf541d03ce6a867d2ce47d8eab0f6b62fced493cbcc210"} Sep 17 01:32:08.746501 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:08.746475 2577 generic.go:356] "Generic (PLEG): container finished" podID="9799bbd1-9fd6-4b9c-a724-c3545b2aadb2" containerID="c29f32e2521d5d7c2b605ea722ea7d24dfcdc7b2ed34e08d048da38cbd495cc1" exitCode=0 Sep 17 01:32:08.746815 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:08.746560 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn" event={"ID":"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2","Type":"ContainerDied","Data":"c29f32e2521d5d7c2b605ea722ea7d24dfcdc7b2ed34e08d048da38cbd495cc1"} Sep 17 01:32:08.748304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:08.748282 2577 generic.go:356] "Generic (PLEG): container finished" podID="dbc122b0-4d70-4cc7-ad24-1ffab04cd039" containerID="367e8fdac7fbef0038da7e872b584ef2f8c59c9541ef43f0ecc08ab13a6c5f72" exitCode=0 Sep 17 01:32:08.748377 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:08.748362 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk" event={"ID":"dbc122b0-4d70-4cc7-ad24-1ffab04cd039","Type":"ContainerDied","Data":"367e8fdac7fbef0038da7e872b584ef2f8c59c9541ef43f0ecc08ab13a6c5f72"} Sep 17 01:32:08.750231 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:08.750205 2577 generic.go:356] "Generic (PLEG): container finished" podID="3c3c83f0-258f-4539-99c8-2ee09a09c000" containerID="492c8b1195b27910ca363d67414bda9500bc5c0cd1f2a6faaf99eb2a8494fd5c" exitCode=0 Sep 17 01:32:08.750322 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:08.750251 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz" event={"ID":"3c3c83f0-258f-4539-99c8-2ee09a09c000","Type":"ContainerDied","Data":"492c8b1195b27910ca363d67414bda9500bc5c0cd1f2a6faaf99eb2a8494fd5c"} Sep 17 01:32:09.758174 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:09.758132 2577 generic.go:356] "Generic (PLEG): container finished" podID="dbc122b0-4d70-4cc7-ad24-1ffab04cd039" containerID="603eb3934dc09b65d640b46a6d8c047642f1deb4691f09973293c399421ee3ba" exitCode=0 Sep 17 01:32:09.758603 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:09.758226 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk" event={"ID":"dbc122b0-4d70-4cc7-ad24-1ffab04cd039","Type":"ContainerDied","Data":"603eb3934dc09b65d640b46a6d8c047642f1deb4691f09973293c399421ee3ba"} Sep 17 01:32:09.760420 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:09.760397 2577 generic.go:356] "Generic (PLEG): container finished" podID="3c3c83f0-258f-4539-99c8-2ee09a09c000" containerID="7c77298c94c0f0d14daa9c31b726d46da78464120b708ce58423d1462fb89d10" exitCode=0 Sep 17 01:32:09.760536 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:09.760475 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz" event={"ID":"3c3c83f0-258f-4539-99c8-2ee09a09c000","Type":"ContainerDied","Data":"7c77298c94c0f0d14daa9c31b726d46da78464120b708ce58423d1462fb89d10"} Sep 17 01:32:09.762213 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:09.762196 2577 generic.go:356] "Generic (PLEG): container finished" podID="62352577-a39b-49c3-b12f-079a94e4e342" containerID="6e2f0852c5f44c5fb1f3e332144739cab2af7ffdd0df7cacba5b23f7647816d7" exitCode=0 Sep 17 01:32:09.762280 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:09.762255 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8" event={"ID":"62352577-a39b-49c3-b12f-079a94e4e342","Type":"ContainerDied","Data":"6e2f0852c5f44c5fb1f3e332144739cab2af7ffdd0df7cacba5b23f7647816d7"} Sep 17 01:32:09.895745 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:09.895721 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn" Sep 17 01:32:09.980763 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:09.980734 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p46zr\" (UniqueName: \"kubernetes.io/projected/9799bbd1-9fd6-4b9c-a724-c3545b2aadb2-kube-api-access-p46zr\") pod \"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2\" (UID: \"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2\") " Sep 17 01:32:09.980896 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:09.980801 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9799bbd1-9fd6-4b9c-a724-c3545b2aadb2-bundle\") pod \"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2\" (UID: \"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2\") " Sep 17 01:32:09.980896 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:09.980864 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9799bbd1-9fd6-4b9c-a724-c3545b2aadb2-util\") pod \"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2\" (UID: \"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2\") " Sep 17 01:32:09.981414 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:09.981388 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9799bbd1-9fd6-4b9c-a724-c3545b2aadb2-bundle" (OuterVolumeSpecName: "bundle") pod "9799bbd1-9fd6-4b9c-a724-c3545b2aadb2" (UID: "9799bbd1-9fd6-4b9c-a724-c3545b2aadb2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:32:09.982813 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:09.982786 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9799bbd1-9fd6-4b9c-a724-c3545b2aadb2-kube-api-access-p46zr" (OuterVolumeSpecName: "kube-api-access-p46zr") pod "9799bbd1-9fd6-4b9c-a724-c3545b2aadb2" (UID: "9799bbd1-9fd6-4b9c-a724-c3545b2aadb2"). InnerVolumeSpecName "kube-api-access-p46zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:32:09.985636 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:09.985602 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9799bbd1-9fd6-4b9c-a724-c3545b2aadb2-util" (OuterVolumeSpecName: "util") pod "9799bbd1-9fd6-4b9c-a724-c3545b2aadb2" (UID: "9799bbd1-9fd6-4b9c-a724-c3545b2aadb2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:32:10.082196 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.082123 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p46zr\" (UniqueName: \"kubernetes.io/projected/9799bbd1-9fd6-4b9c-a724-c3545b2aadb2-kube-api-access-p46zr\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:10.082196 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.082152 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9799bbd1-9fd6-4b9c-a724-c3545b2aadb2-bundle\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:10.082196 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.082163 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9799bbd1-9fd6-4b9c-a724-c3545b2aadb2-util\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:10.770405 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.770366 2577 generic.go:356] "Generic (PLEG): container finished" podID="62352577-a39b-49c3-b12f-079a94e4e342" containerID="0a7333fd4f88ed287aa67ad68dfb23a18e5c1f59788bb416cf1b6f11e7dc4df9" exitCode=0 Sep 17 01:32:10.770862 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.770448 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8" event={"ID":"62352577-a39b-49c3-b12f-079a94e4e342","Type":"ContainerDied","Data":"0a7333fd4f88ed287aa67ad68dfb23a18e5c1f59788bb416cf1b6f11e7dc4df9"} Sep 17 01:32:10.772565 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.772543 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn" Sep 17 01:32:10.772666 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.772576 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avqvtn" event={"ID":"9799bbd1-9fd6-4b9c-a724-c3545b2aadb2","Type":"ContainerDied","Data":"d14dd48ea9c213e9b9ed0fc4f50ff91896b90e5b68a6d4fff64e8287cc2af336"} Sep 17 01:32:10.772666 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.772619 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d14dd48ea9c213e9b9ed0fc4f50ff91896b90e5b68a6d4fff64e8287cc2af336" Sep 17 01:32:10.909212 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.909188 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz" Sep 17 01:32:10.946222 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.946196 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk" Sep 17 01:32:10.990016 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.989976 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x5rz\" (UniqueName: \"kubernetes.io/projected/3c3c83f0-258f-4539-99c8-2ee09a09c000-kube-api-access-9x5rz\") pod \"3c3c83f0-258f-4539-99c8-2ee09a09c000\" (UID: \"3c3c83f0-258f-4539-99c8-2ee09a09c000\") " Sep 17 01:32:10.990168 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.990051 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c3c83f0-258f-4539-99c8-2ee09a09c000-bundle\") pod \"3c3c83f0-258f-4539-99c8-2ee09a09c000\" (UID: \"3c3c83f0-258f-4539-99c8-2ee09a09c000\") " Sep 17 01:32:10.990232 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.990210 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c3c83f0-258f-4539-99c8-2ee09a09c000-util\") pod \"3c3c83f0-258f-4539-99c8-2ee09a09c000\" (UID: \"3c3c83f0-258f-4539-99c8-2ee09a09c000\") " Sep 17 01:32:10.990311 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.990299 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbc122b0-4d70-4cc7-ad24-1ffab04cd039-bundle\") pod \"dbc122b0-4d70-4cc7-ad24-1ffab04cd039\" (UID: \"dbc122b0-4d70-4cc7-ad24-1ffab04cd039\") " Sep 17 01:32:10.990360 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.990320 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbc122b0-4d70-4cc7-ad24-1ffab04cd039-util\") pod \"dbc122b0-4d70-4cc7-ad24-1ffab04cd039\" (UID: \"dbc122b0-4d70-4cc7-ad24-1ffab04cd039\") " Sep 17 01:32:10.990360 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.990347 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wnxl\" (UniqueName: \"kubernetes.io/projected/dbc122b0-4d70-4cc7-ad24-1ffab04cd039-kube-api-access-2wnxl\") pod \"dbc122b0-4d70-4cc7-ad24-1ffab04cd039\" (UID: \"dbc122b0-4d70-4cc7-ad24-1ffab04cd039\") " Sep 17 01:32:10.990603 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.990574 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3c83f0-258f-4539-99c8-2ee09a09c000-bundle" (OuterVolumeSpecName: "bundle") pod "3c3c83f0-258f-4539-99c8-2ee09a09c000" (UID: "3c3c83f0-258f-4539-99c8-2ee09a09c000"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:32:10.990706 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.990687 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c3c83f0-258f-4539-99c8-2ee09a09c000-bundle\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:10.991035 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.991011 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc122b0-4d70-4cc7-ad24-1ffab04cd039-bundle" (OuterVolumeSpecName: "bundle") pod "dbc122b0-4d70-4cc7-ad24-1ffab04cd039" (UID: "dbc122b0-4d70-4cc7-ad24-1ffab04cd039"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:32:10.992711 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.992679 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc122b0-4d70-4cc7-ad24-1ffab04cd039-kube-api-access-2wnxl" (OuterVolumeSpecName: "kube-api-access-2wnxl") pod "dbc122b0-4d70-4cc7-ad24-1ffab04cd039" (UID: "dbc122b0-4d70-4cc7-ad24-1ffab04cd039"). InnerVolumeSpecName "kube-api-access-2wnxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:32:10.992711 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.992713 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3c83f0-258f-4539-99c8-2ee09a09c000-kube-api-access-9x5rz" (OuterVolumeSpecName: "kube-api-access-9x5rz") pod "3c3c83f0-258f-4539-99c8-2ee09a09c000" (UID: "3c3c83f0-258f-4539-99c8-2ee09a09c000"). InnerVolumeSpecName "kube-api-access-9x5rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:32:10.995873 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.995851 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc122b0-4d70-4cc7-ad24-1ffab04cd039-util" (OuterVolumeSpecName: "util") pod "dbc122b0-4d70-4cc7-ad24-1ffab04cd039" (UID: "dbc122b0-4d70-4cc7-ad24-1ffab04cd039"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:32:10.996586 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:10.996565 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3c83f0-258f-4539-99c8-2ee09a09c000-util" (OuterVolumeSpecName: "util") pod "3c3c83f0-258f-4539-99c8-2ee09a09c000" (UID: "3c3c83f0-258f-4539-99c8-2ee09a09c000"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:32:11.092049 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:11.092017 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x5rz\" (UniqueName: \"kubernetes.io/projected/3c3c83f0-258f-4539-99c8-2ee09a09c000-kube-api-access-9x5rz\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:11.092049 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:11.092050 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c3c83f0-258f-4539-99c8-2ee09a09c000-util\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:11.092214 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:11.092060 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbc122b0-4d70-4cc7-ad24-1ffab04cd039-bundle\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:11.092214 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:11.092068 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbc122b0-4d70-4cc7-ad24-1ffab04cd039-util\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:11.092214 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:11.092075 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wnxl\" (UniqueName: \"kubernetes.io/projected/dbc122b0-4d70-4cc7-ad24-1ffab04cd039-kube-api-access-2wnxl\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:11.780049 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:11.780014 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz" Sep 17 01:32:11.780530 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:11.780045 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mhsxz" event={"ID":"3c3c83f0-258f-4539-99c8-2ee09a09c000","Type":"ContainerDied","Data":"96c1382964d26cba9bffde1c8851509323a1a19147fa64f1f245719bb12ea797"} Sep 17 01:32:11.780530 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:11.780094 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96c1382964d26cba9bffde1c8851509323a1a19147fa64f1f245719bb12ea797" Sep 17 01:32:11.782192 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:11.782142 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk" Sep 17 01:32:11.782192 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:11.782155 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pcxnk" event={"ID":"dbc122b0-4d70-4cc7-ad24-1ffab04cd039","Type":"ContainerDied","Data":"525c3b336d5deedae9656aa4eb7d8ca5ac39c7fc6821936a03e9d85d406bef9c"} Sep 17 01:32:11.782393 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:11.782215 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="525c3b336d5deedae9656aa4eb7d8ca5ac39c7fc6821936a03e9d85d406bef9c" Sep 17 01:32:11.910667 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:11.910636 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8" Sep 17 01:32:12.001204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:12.001164 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62352577-a39b-49c3-b12f-079a94e4e342-bundle\") pod \"62352577-a39b-49c3-b12f-079a94e4e342\" (UID: \"62352577-a39b-49c3-b12f-079a94e4e342\") " Sep 17 01:32:12.001372 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:12.001214 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62352577-a39b-49c3-b12f-079a94e4e342-util\") pod \"62352577-a39b-49c3-b12f-079a94e4e342\" (UID: \"62352577-a39b-49c3-b12f-079a94e4e342\") " Sep 17 01:32:12.001372 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:12.001280 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf5lf\" (UniqueName: \"kubernetes.io/projected/62352577-a39b-49c3-b12f-079a94e4e342-kube-api-access-rf5lf\") pod \"62352577-a39b-49c3-b12f-079a94e4e342\" (UID: \"62352577-a39b-49c3-b12f-079a94e4e342\") " Sep 17 01:32:12.001855 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:12.001823 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62352577-a39b-49c3-b12f-079a94e4e342-bundle" (OuterVolumeSpecName: "bundle") pod "62352577-a39b-49c3-b12f-079a94e4e342" (UID: "62352577-a39b-49c3-b12f-079a94e4e342"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:32:12.003479 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:12.003448 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62352577-a39b-49c3-b12f-079a94e4e342-kube-api-access-rf5lf" (OuterVolumeSpecName: "kube-api-access-rf5lf") pod "62352577-a39b-49c3-b12f-079a94e4e342" (UID: "62352577-a39b-49c3-b12f-079a94e4e342"). InnerVolumeSpecName "kube-api-access-rf5lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:32:12.007010 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:12.006978 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62352577-a39b-49c3-b12f-079a94e4e342-util" (OuterVolumeSpecName: "util") pod "62352577-a39b-49c3-b12f-079a94e4e342" (UID: "62352577-a39b-49c3-b12f-079a94e4e342"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:32:12.102311 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:12.102280 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf5lf\" (UniqueName: \"kubernetes.io/projected/62352577-a39b-49c3-b12f-079a94e4e342-kube-api-access-rf5lf\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:12.102311 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:12.102310 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62352577-a39b-49c3-b12f-079a94e4e342-bundle\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:12.102487 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:12.102321 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62352577-a39b-49c3-b12f-079a94e4e342-util\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:12.789602 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:12.789565 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8" event={"ID":"62352577-a39b-49c3-b12f-079a94e4e342","Type":"ContainerDied","Data":"b271b623fdf7961c8a45d5f7fc4e28a8a685b5837acb9f5e6501454bfe3a8e73"} Sep 17 01:32:12.789996 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:12.789620 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b271b623fdf7961c8a45d5f7fc4e28a8a685b5837acb9f5e6501454bfe3a8e73" Sep 17 01:32:12.789996 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:12.789582 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpwhs8" Sep 17 01:32:15.141293 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:15.141258 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 01:32:15.141742 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:15.141722 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 01:32:19.732715 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:19.732653 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6fdc664585-s8pnk" podUID="ff0a23a2-2849-4b7f-ae09-2e29651811e4" containerName="console" containerID="cri-o://d3dfef928563146aa892ee4967079813a5257b580409b91a78fd68197763ec1e" gracePeriod=15 Sep 17 01:32:19.984908 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:19.984845 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:32:20.074353 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.074309 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phdvc\" (UniqueName: \"kubernetes.io/projected/ff0a23a2-2849-4b7f-ae09-2e29651811e4-kube-api-access-phdvc\") pod \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " Sep 17 01:32:20.074550 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.074366 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-oauth-serving-cert\") pod \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " Sep 17 01:32:20.074550 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.074400 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff0a23a2-2849-4b7f-ae09-2e29651811e4-console-serving-cert\") pod \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " Sep 17 01:32:20.074550 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.074424 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-trusted-ca-bundle\") pod \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " Sep 17 01:32:20.074782 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.074596 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff0a23a2-2849-4b7f-ae09-2e29651811e4-console-oauth-config\") pod \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " Sep 17 01:32:20.074782 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.074636 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-service-ca\") pod \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " Sep 17 01:32:20.074782 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.074699 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-console-config\") pod \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\" (UID: \"ff0a23a2-2849-4b7f-ae09-2e29651811e4\") " Sep 17 01:32:20.074981 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.074956 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ff0a23a2-2849-4b7f-ae09-2e29651811e4" (UID: "ff0a23a2-2849-4b7f-ae09-2e29651811e4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:32:20.074981 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.074967 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ff0a23a2-2849-4b7f-ae09-2e29651811e4" (UID: "ff0a23a2-2849-4b7f-ae09-2e29651811e4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:32:20.075239 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.075077 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-oauth-serving-cert\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:20.075239 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.075098 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-trusted-ca-bundle\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:20.075239 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.075072 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-service-ca" (OuterVolumeSpecName: "service-ca") pod "ff0a23a2-2849-4b7f-ae09-2e29651811e4" (UID: "ff0a23a2-2849-4b7f-ae09-2e29651811e4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:32:20.075400 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.075260 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-console-config" (OuterVolumeSpecName: "console-config") pod "ff0a23a2-2849-4b7f-ae09-2e29651811e4" (UID: "ff0a23a2-2849-4b7f-ae09-2e29651811e4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:32:20.076738 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.076715 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0a23a2-2849-4b7f-ae09-2e29651811e4-kube-api-access-phdvc" (OuterVolumeSpecName: "kube-api-access-phdvc") pod "ff0a23a2-2849-4b7f-ae09-2e29651811e4" (UID: "ff0a23a2-2849-4b7f-ae09-2e29651811e4"). InnerVolumeSpecName "kube-api-access-phdvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:32:20.076839 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.076719 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff0a23a2-2849-4b7f-ae09-2e29651811e4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ff0a23a2-2849-4b7f-ae09-2e29651811e4" (UID: "ff0a23a2-2849-4b7f-ae09-2e29651811e4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:32:20.076839 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.076743 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff0a23a2-2849-4b7f-ae09-2e29651811e4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ff0a23a2-2849-4b7f-ae09-2e29651811e4" (UID: "ff0a23a2-2849-4b7f-ae09-2e29651811e4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:32:20.176179 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.176144 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff0a23a2-2849-4b7f-ae09-2e29651811e4-console-oauth-config\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:20.176179 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.176173 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-service-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:20.176179 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.176183 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff0a23a2-2849-4b7f-ae09-2e29651811e4-console-config\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:20.176410 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.176192 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phdvc\" (UniqueName: \"kubernetes.io/projected/ff0a23a2-2849-4b7f-ae09-2e29651811e4-kube-api-access-phdvc\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:20.176410 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.176200 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff0a23a2-2849-4b7f-ae09-2e29651811e4-console-serving-cert\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:20.378535 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.378501 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65"] Sep 17 01:32:20.378882 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.378869 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3c83f0-258f-4539-99c8-2ee09a09c000" containerName="extract" Sep 17 01:32:20.378926 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.378884 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3c83f0-258f-4539-99c8-2ee09a09c000" containerName="extract" Sep 17 01:32:20.378926 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.378897 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff0a23a2-2849-4b7f-ae09-2e29651811e4" containerName="console" Sep 17 01:32:20.378926 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.378903 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff0a23a2-2849-4b7f-ae09-2e29651811e4" containerName="console" Sep 17 01:32:20.378926 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.378908 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3c83f0-258f-4539-99c8-2ee09a09c000" containerName="util" Sep 17 01:32:20.378926 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.378913 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3c83f0-258f-4539-99c8-2ee09a09c000" containerName="util" Sep 17 01:32:20.378926 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.378920 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9799bbd1-9fd6-4b9c-a724-c3545b2aadb2" containerName="util" Sep 17 01:32:20.378926 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.378926 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9799bbd1-9fd6-4b9c-a724-c3545b2aadb2" containerName="util" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.378948 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62352577-a39b-49c3-b12f-079a94e4e342" containerName="util" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.378953 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="62352577-a39b-49c3-b12f-079a94e4e342" containerName="util" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.378959 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9799bbd1-9fd6-4b9c-a724-c3545b2aadb2" containerName="pull" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.378964 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9799bbd1-9fd6-4b9c-a724-c3545b2aadb2" containerName="pull" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.378970 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9799bbd1-9fd6-4b9c-a724-c3545b2aadb2" containerName="extract" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.378975 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9799bbd1-9fd6-4b9c-a724-c3545b2aadb2" containerName="extract" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.378981 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc122b0-4d70-4cc7-ad24-1ffab04cd039" containerName="pull" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.378986 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc122b0-4d70-4cc7-ad24-1ffab04cd039" containerName="pull" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.378994 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62352577-a39b-49c3-b12f-079a94e4e342" containerName="pull" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.379000 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="62352577-a39b-49c3-b12f-079a94e4e342" containerName="pull" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.379007 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62352577-a39b-49c3-b12f-079a94e4e342" containerName="extract" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.379013 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="62352577-a39b-49c3-b12f-079a94e4e342" containerName="extract" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.379023 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc122b0-4d70-4cc7-ad24-1ffab04cd039" containerName="util" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.379028 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc122b0-4d70-4cc7-ad24-1ffab04cd039" containerName="util" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.379035 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3c83f0-258f-4539-99c8-2ee09a09c000" containerName="pull" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.379040 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3c83f0-258f-4539-99c8-2ee09a09c000" containerName="pull" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.379046 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc122b0-4d70-4cc7-ad24-1ffab04cd039" containerName="extract" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.379051 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc122b0-4d70-4cc7-ad24-1ffab04cd039" containerName="extract" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.379114 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c3c83f0-258f-4539-99c8-2ee09a09c000" containerName="extract" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.379121 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff0a23a2-2849-4b7f-ae09-2e29651811e4" containerName="console" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.379127 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9799bbd1-9fd6-4b9c-a724-c3545b2aadb2" containerName="extract" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.379132 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="62352577-a39b-49c3-b12f-079a94e4e342" containerName="extract" Sep 17 01:32:20.379204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.379139 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbc122b0-4d70-4cc7-ad24-1ffab04cd039" containerName="extract" Sep 17 01:32:20.382416 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.382398 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" Sep 17 01:32:20.384651 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.384624 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 17 01:32:20.384773 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.384632 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 17 01:32:20.384773 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.384683 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kl65s\"" Sep 17 01:32:20.396082 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.396057 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65"] Sep 17 01:32:20.478858 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.478816 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxg9j\" (UniqueName: \"kubernetes.io/projected/02aa664d-d964-4452-a4d9-1b24ed4109c5-kube-api-access-dxg9j\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2jf65\" (UID: \"02aa664d-d964-4452-a4d9-1b24ed4109c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" Sep 17 01:32:20.479062 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.478951 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02aa664d-d964-4452-a4d9-1b24ed4109c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2jf65\" (UID: \"02aa664d-d964-4452-a4d9-1b24ed4109c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" Sep 17 01:32:20.580375 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.580331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02aa664d-d964-4452-a4d9-1b24ed4109c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2jf65\" (UID: \"02aa664d-d964-4452-a4d9-1b24ed4109c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" Sep 17 01:32:20.580575 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.580430 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxg9j\" (UniqueName: \"kubernetes.io/projected/02aa664d-d964-4452-a4d9-1b24ed4109c5-kube-api-access-dxg9j\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2jf65\" (UID: \"02aa664d-d964-4452-a4d9-1b24ed4109c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" Sep 17 01:32:20.580723 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.580697 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02aa664d-d964-4452-a4d9-1b24ed4109c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2jf65\" (UID: \"02aa664d-d964-4452-a4d9-1b24ed4109c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" Sep 17 01:32:20.597385 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.597358 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxg9j\" (UniqueName: \"kubernetes.io/projected/02aa664d-d964-4452-a4d9-1b24ed4109c5-kube-api-access-dxg9j\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2jf65\" (UID: \"02aa664d-d964-4452-a4d9-1b24ed4109c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" Sep 17 01:32:20.692574 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.692489 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" Sep 17 01:32:20.836187 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.836152 2577 generic.go:356] "Generic (PLEG): container finished" podID="ff0a23a2-2849-4b7f-ae09-2e29651811e4" containerID="d3dfef928563146aa892ee4967079813a5257b580409b91a78fd68197763ec1e" exitCode=0 Sep 17 01:32:20.836589 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.836201 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fdc664585-s8pnk" Sep 17 01:32:20.836589 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.836237 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fdc664585-s8pnk" event={"ID":"ff0a23a2-2849-4b7f-ae09-2e29651811e4","Type":"ContainerDied","Data":"d3dfef928563146aa892ee4967079813a5257b580409b91a78fd68197763ec1e"} Sep 17 01:32:20.836589 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.836297 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fdc664585-s8pnk" event={"ID":"ff0a23a2-2849-4b7f-ae09-2e29651811e4","Type":"ContainerDied","Data":"e6dd29fd1425ca7d481bd7b954fba9fc769272083f49b11839d621d7da20256c"} Sep 17 01:32:20.836589 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.836321 2577 scope.go:117] "RemoveContainer" containerID="d3dfef928563146aa892ee4967079813a5257b580409b91a78fd68197763ec1e" Sep 17 01:32:20.846037 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.846019 2577 scope.go:117] "RemoveContainer" containerID="d3dfef928563146aa892ee4967079813a5257b580409b91a78fd68197763ec1e" Sep 17 01:32:20.846289 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:32:20.846272 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3dfef928563146aa892ee4967079813a5257b580409b91a78fd68197763ec1e\": container with ID starting with d3dfef928563146aa892ee4967079813a5257b580409b91a78fd68197763ec1e not found: ID does not exist" containerID="d3dfef928563146aa892ee4967079813a5257b580409b91a78fd68197763ec1e" Sep 17 01:32:20.846342 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.846296 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3dfef928563146aa892ee4967079813a5257b580409b91a78fd68197763ec1e"} err="failed to get container status \"d3dfef928563146aa892ee4967079813a5257b580409b91a78fd68197763ec1e\": rpc error: code = NotFound desc = could not find container \"d3dfef928563146aa892ee4967079813a5257b580409b91a78fd68197763ec1e\": container with ID starting with d3dfef928563146aa892ee4967079813a5257b580409b91a78fd68197763ec1e not found: ID does not exist" Sep 17 01:32:20.869362 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.869334 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fdc664585-s8pnk"] Sep 17 01:32:20.871666 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.871645 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fdc664585-s8pnk"] Sep 17 01:32:20.999636 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:20.999612 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65"] Sep 17 01:32:21.001750 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:32:21.001722 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02aa664d_d964_4452_a4d9_1b24ed4109c5.slice/crio-7367cad5ddd62094b50724a038dbc657b536f220ff7b57d53ae0952da254d2d5 WatchSource:0}: Error finding container 7367cad5ddd62094b50724a038dbc657b536f220ff7b57d53ae0952da254d2d5: Status 404 returned error can't find the container with id 7367cad5ddd62094b50724a038dbc657b536f220ff7b57d53ae0952da254d2d5 Sep 17 01:32:21.212216 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:21.212183 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff0a23a2-2849-4b7f-ae09-2e29651811e4" path="/var/lib/kubelet/pods/ff0a23a2-2849-4b7f-ae09-2e29651811e4/volumes" Sep 17 01:32:21.842725 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:21.842687 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" event={"ID":"02aa664d-d964-4452-a4d9-1b24ed4109c5","Type":"ContainerStarted","Data":"7367cad5ddd62094b50724a038dbc657b536f220ff7b57d53ae0952da254d2d5"} Sep 17 01:32:26.579047 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:26.578999 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wt2k"] Sep 17 01:32:26.582919 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:26.582888 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wt2k" Sep 17 01:32:26.585178 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:26.585151 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-lp596\"" Sep 17 01:32:26.585178 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:26.585165 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 17 01:32:26.591549 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:26.591523 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wt2k"] Sep 17 01:32:26.732138 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:26.732099 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wssq9\" (UniqueName: \"kubernetes.io/projected/81042ac3-f417-42ca-b6ab-f14933e3c927-kube-api-access-wssq9\") pod \"dns-operator-controller-manager-7b84cd7c9c-8wt2k\" (UID: \"81042ac3-f417-42ca-b6ab-f14933e3c927\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wt2k" Sep 17 01:32:26.833497 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:26.833417 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wssq9\" (UniqueName: \"kubernetes.io/projected/81042ac3-f417-42ca-b6ab-f14933e3c927-kube-api-access-wssq9\") pod \"dns-operator-controller-manager-7b84cd7c9c-8wt2k\" (UID: \"81042ac3-f417-42ca-b6ab-f14933e3c927\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wt2k" Sep 17 01:32:26.845554 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:26.845525 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wssq9\" (UniqueName: \"kubernetes.io/projected/81042ac3-f417-42ca-b6ab-f14933e3c927-kube-api-access-wssq9\") pod \"dns-operator-controller-manager-7b84cd7c9c-8wt2k\" (UID: \"81042ac3-f417-42ca-b6ab-f14933e3c927\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wt2k" Sep 17 01:32:26.896945 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:26.896910 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wt2k" Sep 17 01:32:27.181148 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:27.181120 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wt2k"] Sep 17 01:32:27.182050 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:32:27.182026 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81042ac3_f417_42ca_b6ab_f14933e3c927.slice/crio-ad2e40fa04c1c545d27951a723fc41325ce707e638d1fc05f041d2e37bce97b3 WatchSource:0}: Error finding container ad2e40fa04c1c545d27951a723fc41325ce707e638d1fc05f041d2e37bce97b3: Status 404 returned error can't find the container with id ad2e40fa04c1c545d27951a723fc41325ce707e638d1fc05f041d2e37bce97b3 Sep 17 01:32:27.883430 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:27.883391 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wt2k" event={"ID":"81042ac3-f417-42ca-b6ab-f14933e3c927","Type":"ContainerStarted","Data":"ad2e40fa04c1c545d27951a723fc41325ce707e638d1fc05f041d2e37bce97b3"} Sep 17 01:32:29.895977 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:29.895949 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" event={"ID":"02aa664d-d964-4452-a4d9-1b24ed4109c5","Type":"ContainerStarted","Data":"0594863f002efe613be6caa205a45012595bd836c03a3e21289a775f757778b4"} Sep 17 01:32:29.896353 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:29.896102 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" Sep 17 01:32:29.915660 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:29.915620 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" podStartSLOduration=2.036881317 podStartE2EDuration="9.915604441s" totalImagesPullingTime="7.878723124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:32:29.912289079 +0000 UTC m=+615.147436277" watchObservedRunningTime="2026-09-17 01:32:29.915604441 +0000 UTC m=+615.150751629" Sep 17 01:32:30.903414 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:30.903376 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wt2k" event={"ID":"81042ac3-f417-42ca-b6ab-f14933e3c927","Type":"ContainerStarted","Data":"0d949bdd061ace3333c49c49c8f9dbdb6ec3fc007def528d498ecac1f8663e7e"} Sep 17 01:32:30.920242 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:30.920181 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wt2k" podStartSLOduration=2.241949505 podStartE2EDuration="4.92016079s" totalImagesPullingTime="2.678211285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:32:30.919621339 +0000 UTC m=+616.154768541" watchObservedRunningTime="2026-09-17 01:32:30.92016079 +0000 UTC m=+616.155307979" Sep 17 01:32:31.831198 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:31.831165 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bf8sz"] Sep 17 01:32:31.834828 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:31.834805 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bf8sz" Sep 17 01:32:31.836780 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:31.836751 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 17 01:32:31.836904 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:31.836757 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gkprx\"" Sep 17 01:32:31.836904 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:31.836757 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 17 01:32:31.840980 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:31.840958 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bf8sz"] Sep 17 01:32:31.908632 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:31.908592 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wt2k" Sep 17 01:32:31.982035 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:31.981998 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bf8sz\" (UID: \"6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bf8sz" Sep 17 01:32:31.982237 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:31.982069 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bf8sz\" (UID: \"6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bf8sz" Sep 17 01:32:31.982237 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:31.982149 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8ktw\" (UniqueName: \"kubernetes.io/projected/6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1-kube-api-access-q8ktw\") pod \"kuadrant-console-plugin-fbdd4f445-bf8sz\" (UID: \"6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bf8sz" Sep 17 01:32:32.083388 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:32.083306 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8ktw\" (UniqueName: \"kubernetes.io/projected/6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1-kube-api-access-q8ktw\") pod \"kuadrant-console-plugin-fbdd4f445-bf8sz\" (UID: \"6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bf8sz" Sep 17 01:32:32.083499 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:32.083436 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bf8sz\" (UID: \"6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bf8sz" Sep 17 01:32:32.083499 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:32.083479 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bf8sz\" (UID: \"6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bf8sz" Sep 17 01:32:32.084151 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:32.084127 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bf8sz\" (UID: \"6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bf8sz" Sep 17 01:32:32.085899 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:32.085874 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bf8sz\" (UID: \"6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bf8sz" Sep 17 01:32:32.094269 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:32.094241 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8ktw\" (UniqueName: \"kubernetes.io/projected/6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1-kube-api-access-q8ktw\") pod \"kuadrant-console-plugin-fbdd4f445-bf8sz\" (UID: \"6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bf8sz" Sep 17 01:32:32.144568 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:32.144542 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bf8sz" Sep 17 01:32:32.413596 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:32.413566 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bf8sz"] Sep 17 01:32:32.415164 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:32:32.415137 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ab4c687_a263_4ad4_a8be_ae0eb5ecddb1.slice/crio-6dd79c0cf514102394c29f507017c8c33cd97541f13f4ba5ed28467db85a1f54 WatchSource:0}: Error finding container 6dd79c0cf514102394c29f507017c8c33cd97541f13f4ba5ed28467db85a1f54: Status 404 returned error can't find the container with id 6dd79c0cf514102394c29f507017c8c33cd97541f13f4ba5ed28467db85a1f54 Sep 17 01:32:32.921363 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:32.921319 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bf8sz" event={"ID":"6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1","Type":"ContainerStarted","Data":"6dd79c0cf514102394c29f507017c8c33cd97541f13f4ba5ed28467db85a1f54"} Sep 17 01:32:37.957518 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:37.957434 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bf8sz" event={"ID":"6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1","Type":"ContainerStarted","Data":"3b8d83b0bd70fba9dead6114ffb24e9aa4ca5fb8ae06b686184ccd6ad551ed52"} Sep 17 01:32:37.973744 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:37.973694 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bf8sz" podStartSLOduration=2.380223832 podStartE2EDuration="6.973678283s" totalImagesPullingTime="4.593454451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:32:37.972910239 +0000 UTC m=+623.208057438" watchObservedRunningTime="2026-09-17 01:32:37.973678283 +0000 UTC m=+623.208825471" Sep 17 01:32:40.906091 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:40.906061 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" Sep 17 01:32:42.924431 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:42.924399 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wt2k" Sep 17 01:32:42.965767 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:42.965739 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65"] Sep 17 01:32:42.965993 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:42.965914 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" podUID="02aa664d-d964-4452-a4d9-1b24ed4109c5" containerName="manager" containerID="cri-o://0594863f002efe613be6caa205a45012595bd836c03a3e21289a775f757778b4" gracePeriod=2 Sep 17 01:32:42.981033 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:42.981005 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65"] Sep 17 01:32:42.990492 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:32:42.990447 2577 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"02aa664d-d964-4452-a4d9-1b24ed4109c5\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-2jf65\": pods \"kuadrant-operator-controller-manager-76f8f686bf-2jf65\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" Sep 17 01:32:42.991574 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:42.991547 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2"] Sep 17 01:32:42.992024 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:42.992006 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02aa664d-d964-4452-a4d9-1b24ed4109c5" containerName="manager" Sep 17 01:32:42.992079 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:42.992027 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="02aa664d-d964-4452-a4d9-1b24ed4109c5" containerName="manager" Sep 17 01:32:42.992172 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:42.992159 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="02aa664d-d964-4452-a4d9-1b24ed4109c5" containerName="manager" Sep 17 01:32:42.995439 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:42.995422 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2" Sep 17 01:32:42.997165 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:32:42.997137 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-2jf65\" is forbidden: User \"system:node:ip-10-0-138-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-83.ec2.internal' and this object" podUID="02aa664d-d964-4452-a4d9-1b24ed4109c5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" Sep 17 01:32:43.008414 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.008390 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2"] Sep 17 01:32:43.024530 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.024506 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7xlfz"] Sep 17 01:32:43.027983 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.027965 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7xlfz" Sep 17 01:32:43.031000 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.030983 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-xbjcm\"" Sep 17 01:32:43.042816 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.042792 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7xlfz"] Sep 17 01:32:43.056631 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:32:43.056604 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-2jf65\" is forbidden: User \"system:node:ip-10-0-138-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-83.ec2.internal' and this object" podUID="02aa664d-d964-4452-a4d9-1b24ed4109c5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" Sep 17 01:32:43.080379 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.080353 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/868fb40c-34a5-42c5-b04f-5935a3a3b165-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7wbt2\" (UID: \"868fb40c-34a5-42c5-b04f-5935a3a3b165\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2" Sep 17 01:32:43.080476 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.080430 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x4fm\" (UniqueName: \"kubernetes.io/projected/63872aa9-d5cd-4c35-8ebd-3e7c5b18d0ec-kube-api-access-9x4fm\") pod \"limitador-operator-controller-manager-74495c69f8-7xlfz\" (UID: \"63872aa9-d5cd-4c35-8ebd-3e7c5b18d0ec\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7xlfz" Sep 17 01:32:43.080476 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.080459 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9c8g\" (UniqueName: \"kubernetes.io/projected/868fb40c-34a5-42c5-b04f-5935a3a3b165-kube-api-access-s9c8g\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7wbt2\" (UID: \"868fb40c-34a5-42c5-b04f-5935a3a3b165\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2" Sep 17 01:32:43.181067 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.180992 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/868fb40c-34a5-42c5-b04f-5935a3a3b165-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7wbt2\" (UID: \"868fb40c-34a5-42c5-b04f-5935a3a3b165\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2" Sep 17 01:32:43.181232 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.181086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x4fm\" (UniqueName: \"kubernetes.io/projected/63872aa9-d5cd-4c35-8ebd-3e7c5b18d0ec-kube-api-access-9x4fm\") pod \"limitador-operator-controller-manager-74495c69f8-7xlfz\" (UID: \"63872aa9-d5cd-4c35-8ebd-3e7c5b18d0ec\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7xlfz" Sep 17 01:32:43.181232 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.181132 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9c8g\" (UniqueName: \"kubernetes.io/projected/868fb40c-34a5-42c5-b04f-5935a3a3b165-kube-api-access-s9c8g\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7wbt2\" (UID: \"868fb40c-34a5-42c5-b04f-5935a3a3b165\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2" Sep 17 01:32:43.181386 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.181363 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/868fb40c-34a5-42c5-b04f-5935a3a3b165-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7wbt2\" (UID: \"868fb40c-34a5-42c5-b04f-5935a3a3b165\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2" Sep 17 01:32:43.190024 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.190001 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x4fm\" (UniqueName: \"kubernetes.io/projected/63872aa9-d5cd-4c35-8ebd-3e7c5b18d0ec-kube-api-access-9x4fm\") pod \"limitador-operator-controller-manager-74495c69f8-7xlfz\" (UID: \"63872aa9-d5cd-4c35-8ebd-3e7c5b18d0ec\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7xlfz" Sep 17 01:32:43.190024 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.190016 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9c8g\" (UniqueName: \"kubernetes.io/projected/868fb40c-34a5-42c5-b04f-5935a3a3b165-kube-api-access-s9c8g\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7wbt2\" (UID: \"868fb40c-34a5-42c5-b04f-5935a3a3b165\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2" Sep 17 01:32:43.205832 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.205805 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" Sep 17 01:32:43.207818 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:32:43.207785 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-2jf65\" is forbidden: User \"system:node:ip-10-0-138-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-83.ec2.internal' and this object" podUID="02aa664d-d964-4452-a4d9-1b24ed4109c5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" Sep 17 01:32:43.281562 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.281535 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxg9j\" (UniqueName: \"kubernetes.io/projected/02aa664d-d964-4452-a4d9-1b24ed4109c5-kube-api-access-dxg9j\") pod \"02aa664d-d964-4452-a4d9-1b24ed4109c5\" (UID: \"02aa664d-d964-4452-a4d9-1b24ed4109c5\") " Sep 17 01:32:43.281706 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.281586 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02aa664d-d964-4452-a4d9-1b24ed4109c5-extensions-socket-volume\") pod \"02aa664d-d964-4452-a4d9-1b24ed4109c5\" (UID: \"02aa664d-d964-4452-a4d9-1b24ed4109c5\") " Sep 17 01:32:43.282134 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.282107 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02aa664d-d964-4452-a4d9-1b24ed4109c5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "02aa664d-d964-4452-a4d9-1b24ed4109c5" (UID: "02aa664d-d964-4452-a4d9-1b24ed4109c5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:32:43.283745 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.283717 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02aa664d-d964-4452-a4d9-1b24ed4109c5-kube-api-access-dxg9j" (OuterVolumeSpecName: "kube-api-access-dxg9j") pod "02aa664d-d964-4452-a4d9-1b24ed4109c5" (UID: "02aa664d-d964-4452-a4d9-1b24ed4109c5"). InnerVolumeSpecName "kube-api-access-dxg9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:32:43.359633 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.359591 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2" Sep 17 01:32:43.365701 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.365674 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7xlfz" Sep 17 01:32:43.383005 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.382981 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02aa664d-d964-4452-a4d9-1b24ed4109c5-extensions-socket-volume\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:43.383112 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.383007 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxg9j\" (UniqueName: \"kubernetes.io/projected/02aa664d-d964-4452-a4d9-1b24ed4109c5-kube-api-access-dxg9j\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:32:43.664734 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.664611 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7xlfz"] Sep 17 01:32:43.667234 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.667203 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2"] Sep 17 01:32:43.667947 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:32:43.667906 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868fb40c_34a5_42c5_b04f_5935a3a3b165.slice/crio-f4234de658d85cf0c9f1685eda23346f92c103c1bdc15fd3a1504500ae454eff WatchSource:0}: Error finding container f4234de658d85cf0c9f1685eda23346f92c103c1bdc15fd3a1504500ae454eff: Status 404 returned error can't find the container with id f4234de658d85cf0c9f1685eda23346f92c103c1bdc15fd3a1504500ae454eff Sep 17 01:32:43.668511 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:32:43.668489 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63872aa9_d5cd_4c35_8ebd_3e7c5b18d0ec.slice/crio-9377efd257fe8b745f0a16b288c5563306cd9e8a2fc22cb104d78c126d888a1e WatchSource:0}: Error finding container 9377efd257fe8b745f0a16b288c5563306cd9e8a2fc22cb104d78c126d888a1e: Status 404 returned error can't find the container with id 9377efd257fe8b745f0a16b288c5563306cd9e8a2fc22cb104d78c126d888a1e Sep 17 01:32:43.996573 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.996476 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2" event={"ID":"868fb40c-34a5-42c5-b04f-5935a3a3b165","Type":"ContainerStarted","Data":"7390830e5e199fe795c8a1d3c8638c9c5e8c78ea45de44d6f84e0a3878993fa3"} Sep 17 01:32:43.996573 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.996537 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2" event={"ID":"868fb40c-34a5-42c5-b04f-5935a3a3b165","Type":"ContainerStarted","Data":"f4234de658d85cf0c9f1685eda23346f92c103c1bdc15fd3a1504500ae454eff"} Sep 17 01:32:43.997045 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.996665 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2" Sep 17 01:32:43.997913 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.997886 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7xlfz" event={"ID":"63872aa9-d5cd-4c35-8ebd-3e7c5b18d0ec","Type":"ContainerStarted","Data":"9377efd257fe8b745f0a16b288c5563306cd9e8a2fc22cb104d78c126d888a1e"} Sep 17 01:32:43.999306 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.999286 2577 generic.go:356] "Generic (PLEG): container finished" podID="02aa664d-d964-4452-a4d9-1b24ed4109c5" containerID="0594863f002efe613be6caa205a45012595bd836c03a3e21289a775f757778b4" exitCode=0 Sep 17 01:32:43.999391 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.999322 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2jf65" Sep 17 01:32:43.999432 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:43.999325 2577 scope.go:117] "RemoveContainer" containerID="0594863f002efe613be6caa205a45012595bd836c03a3e21289a775f757778b4" Sep 17 01:32:44.009045 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:44.009029 2577 scope.go:117] "RemoveContainer" containerID="0594863f002efe613be6caa205a45012595bd836c03a3e21289a775f757778b4" Sep 17 01:32:44.009296 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:32:44.009280 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0594863f002efe613be6caa205a45012595bd836c03a3e21289a775f757778b4\": container with ID starting with 0594863f002efe613be6caa205a45012595bd836c03a3e21289a775f757778b4 not found: ID does not exist" containerID="0594863f002efe613be6caa205a45012595bd836c03a3e21289a775f757778b4" Sep 17 01:32:44.009342 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:44.009301 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0594863f002efe613be6caa205a45012595bd836c03a3e21289a775f757778b4"} err="failed to get container status \"0594863f002efe613be6caa205a45012595bd836c03a3e21289a775f757778b4\": rpc error: code = NotFound desc = could not find container \"0594863f002efe613be6caa205a45012595bd836c03a3e21289a775f757778b4\": container with ID starting with 0594863f002efe613be6caa205a45012595bd836c03a3e21289a775f757778b4 not found: ID does not exist" Sep 17 01:32:44.016833 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:44.016792 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2" podStartSLOduration=2.01677927 podStartE2EDuration="2.01677927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:32:44.01502562 +0000 UTC m=+629.250172960" watchObservedRunningTime="2026-09-17 01:32:44.01677927 +0000 UTC m=+629.251926457" Sep 17 01:32:45.211731 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:45.211693 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02aa664d-d964-4452-a4d9-1b24ed4109c5" path="/var/lib/kubelet/pods/02aa664d-d964-4452-a4d9-1b24ed4109c5/volumes" Sep 17 01:32:46.016140 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:46.016100 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7xlfz" event={"ID":"63872aa9-d5cd-4c35-8ebd-3e7c5b18d0ec","Type":"ContainerStarted","Data":"93eeb3f8c481a3ea73350cfbb20457edd101c5618d4b0967df55f5fa6d16301d"} Sep 17 01:32:46.016331 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:46.016187 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7xlfz" Sep 17 01:32:46.036786 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:46.036733 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7xlfz" podStartSLOduration=2.589894812 podStartE2EDuration="4.036716872s" totalImagesPullingTime="1.44682206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:32:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:32:46.033977651 +0000 UTC m=+631.269124839" watchObservedRunningTime="2026-09-17 01:32:46.036716872 +0000 UTC m=+631.271864142" Sep 17 01:32:55.011274 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:55.011240 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2" Sep 17 01:32:57.024980 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:32:57.024922 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7xlfz" Sep 17 01:33:01.198754 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:01.198718 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2"] Sep 17 01:33:01.199213 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:01.198960 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2" podUID="868fb40c-34a5-42c5-b04f-5935a3a3b165" containerName="manager" containerID="cri-o://7390830e5e199fe795c8a1d3c8638c9c5e8c78ea45de44d6f84e0a3878993fa3" gracePeriod=10 Sep 17 01:33:01.420066 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:01.420027 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6"] Sep 17 01:33:01.424129 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:01.424111 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" Sep 17 01:33:01.452366 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:01.452296 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6"] Sep 17 01:33:01.459488 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:01.459464 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2" Sep 17 01:33:01.549453 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:01.549421 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/868fb40c-34a5-42c5-b04f-5935a3a3b165-extensions-socket-volume\") pod \"868fb40c-34a5-42c5-b04f-5935a3a3b165\" (UID: \"868fb40c-34a5-42c5-b04f-5935a3a3b165\") " Sep 17 01:33:01.549453 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:01.549454 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9c8g\" (UniqueName: \"kubernetes.io/projected/868fb40c-34a5-42c5-b04f-5935a3a3b165-kube-api-access-s9c8g\") pod \"868fb40c-34a5-42c5-b04f-5935a3a3b165\" (UID: \"868fb40c-34a5-42c5-b04f-5935a3a3b165\") " Sep 17 01:33:01.549677 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:01.549656 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/36a411dd-11a4-4abb-9ff1-677dfc09cac7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m6sw6\" (UID: \"36a411dd-11a4-4abb-9ff1-677dfc09cac7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" Sep 17 01:33:01.549837 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:01.549808 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbsgg\" (UniqueName: \"kubernetes.io/projected/36a411dd-11a4-4abb-9ff1-677dfc09cac7-kube-api-access-dbsgg\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m6sw6\" (UID: \"36a411dd-11a4-4abb-9ff1-677dfc09cac7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" Sep 17 01:33:01.549837 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:01.549821 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868fb40c-34a5-42c5-b04f-5935a3a3b165-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "868fb40c-34a5-42c5-b04f-5935a3a3b165" (UID: "868fb40c-34a5-42c5-b04f-5935a3a3b165"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:33:01.550025 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:01.549890 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/868fb40c-34a5-42c5-b04f-5935a3a3b165-extensions-socket-volume\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:33:01.551440 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:01.551419 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/868fb40c-34a5-42c5-b04f-5935a3a3b165-kube-api-access-s9c8g" (OuterVolumeSpecName: "kube-api-access-s9c8g") pod "868fb40c-34a5-42c5-b04f-5935a3a3b165" (UID: "868fb40c-34a5-42c5-b04f-5935a3a3b165"). InnerVolumeSpecName "kube-api-access-s9c8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:33:01.651054 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:01.651015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/36a411dd-11a4-4abb-9ff1-677dfc09cac7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m6sw6\" (UID: \"36a411dd-11a4-4abb-9ff1-677dfc09cac7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" Sep 17 01:33:01.651203 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:01.651102 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbsgg\" (UniqueName: \"kubernetes.io/projected/36a411dd-11a4-4abb-9ff1-677dfc09cac7-kube-api-access-dbsgg\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m6sw6\" (UID: \"36a411dd-11a4-4abb-9ff1-677dfc09cac7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" Sep 17 01:33:01.651203 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:01.651130 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9c8g\" (UniqueName: \"kubernetes.io/projected/868fb40c-34a5-42c5-b04f-5935a3a3b165-kube-api-access-s9c8g\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:33:01.651405 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:01.651383 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/36a411dd-11a4-4abb-9ff1-677dfc09cac7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m6sw6\" (UID: \"36a411dd-11a4-4abb-9ff1-677dfc09cac7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" Sep 17 01:33:01.658182 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:01.658159 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbsgg\" (UniqueName: \"kubernetes.io/projected/36a411dd-11a4-4abb-9ff1-677dfc09cac7-kube-api-access-dbsgg\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m6sw6\" (UID: \"36a411dd-11a4-4abb-9ff1-677dfc09cac7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" Sep 17 01:33:01.734043 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:01.733961 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" Sep 17 01:33:02.007450 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:02.007421 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6"] Sep 17 01:33:02.008569 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:33:02.008543 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a411dd_11a4_4abb_9ff1_677dfc09cac7.slice/crio-58d72b79b7c70627c40103355e259179df84883a3b11ad73723df33637d3cf68 WatchSource:0}: Error finding container 58d72b79b7c70627c40103355e259179df84883a3b11ad73723df33637d3cf68: Status 404 returned error can't find the container with id 58d72b79b7c70627c40103355e259179df84883a3b11ad73723df33637d3cf68 Sep 17 01:33:02.115173 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:02.115143 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" event={"ID":"36a411dd-11a4-4abb-9ff1-677dfc09cac7","Type":"ContainerStarted","Data":"58d72b79b7c70627c40103355e259179df84883a3b11ad73723df33637d3cf68"} Sep 17 01:33:02.116632 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:02.116607 2577 generic.go:356] "Generic (PLEG): container finished" podID="868fb40c-34a5-42c5-b04f-5935a3a3b165" containerID="7390830e5e199fe795c8a1d3c8638c9c5e8c78ea45de44d6f84e0a3878993fa3" exitCode=0 Sep 17 01:33:02.116753 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:02.116653 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2" Sep 17 01:33:02.116753 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:02.116653 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2" event={"ID":"868fb40c-34a5-42c5-b04f-5935a3a3b165","Type":"ContainerDied","Data":"7390830e5e199fe795c8a1d3c8638c9c5e8c78ea45de44d6f84e0a3878993fa3"} Sep 17 01:33:02.116856 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:02.116764 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2" event={"ID":"868fb40c-34a5-42c5-b04f-5935a3a3b165","Type":"ContainerDied","Data":"f4234de658d85cf0c9f1685eda23346f92c103c1bdc15fd3a1504500ae454eff"} Sep 17 01:33:02.116856 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:02.116786 2577 scope.go:117] "RemoveContainer" containerID="7390830e5e199fe795c8a1d3c8638c9c5e8c78ea45de44d6f84e0a3878993fa3" Sep 17 01:33:02.125789 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:02.125773 2577 scope.go:117] "RemoveContainer" containerID="7390830e5e199fe795c8a1d3c8638c9c5e8c78ea45de44d6f84e0a3878993fa3" Sep 17 01:33:02.126076 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:33:02.126057 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7390830e5e199fe795c8a1d3c8638c9c5e8c78ea45de44d6f84e0a3878993fa3\": container with ID starting with 7390830e5e199fe795c8a1d3c8638c9c5e8c78ea45de44d6f84e0a3878993fa3 not found: ID does not exist" containerID="7390830e5e199fe795c8a1d3c8638c9c5e8c78ea45de44d6f84e0a3878993fa3" Sep 17 01:33:02.126148 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:02.126085 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7390830e5e199fe795c8a1d3c8638c9c5e8c78ea45de44d6f84e0a3878993fa3"} err="failed to get container status \"7390830e5e199fe795c8a1d3c8638c9c5e8c78ea45de44d6f84e0a3878993fa3\": rpc error: code = NotFound desc = could not find container \"7390830e5e199fe795c8a1d3c8638c9c5e8c78ea45de44d6f84e0a3878993fa3\": container with ID starting with 7390830e5e199fe795c8a1d3c8638c9c5e8c78ea45de44d6f84e0a3878993fa3 not found: ID does not exist" Sep 17 01:33:02.138454 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:02.138431 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2"] Sep 17 01:33:02.142942 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:02.142907 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7wbt2"] Sep 17 01:33:03.126165 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:03.126125 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" event={"ID":"36a411dd-11a4-4abb-9ff1-677dfc09cac7","Type":"ContainerStarted","Data":"c297fc2df366a5cc16cb060a7c457fde28758e19f602ca5a98349403c0bd9093"} Sep 17 01:33:03.126560 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:03.126248 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" Sep 17 01:33:03.149856 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:03.149804 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" podStartSLOduration=2.149786551 podStartE2EDuration="2.149786551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:33:03.146998645 +0000 UTC m=+648.382145836" watchObservedRunningTime="2026-09-17 01:33:03.149786551 +0000 UTC m=+648.384933738" Sep 17 01:33:03.218760 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:03.218723 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="868fb40c-34a5-42c5-b04f-5935a3a3b165" path="/var/lib/kubelet/pods/868fb40c-34a5-42c5-b04f-5935a3a3b165/volumes" Sep 17 01:33:14.134362 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:33:14.134321 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" Sep 17 01:34:37.061309 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.061261 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s"] Sep 17 01:34:37.061765 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.061720 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="868fb40c-34a5-42c5-b04f-5935a3a3b165" containerName="manager" Sep 17 01:34:37.061765 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.061732 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="868fb40c-34a5-42c5-b04f-5935a3a3b165" containerName="manager" Sep 17 01:34:37.061850 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.061825 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="868fb40c-34a5-42c5-b04f-5935a3a3b165" containerName="manager" Sep 17 01:34:37.064964 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.064926 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s" Sep 17 01:34:37.069918 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.069895 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-qfwsb\"" Sep 17 01:34:37.070216 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.070196 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 17 01:34:37.075570 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.075546 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s"] Sep 17 01:34:37.092760 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.092730 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7xstn"] Sep 17 01:34:37.096919 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.096898 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7xstn" Sep 17 01:34:37.100088 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.100065 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7xstn"] Sep 17 01:34:37.173755 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.173718 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ad3828-5fbb-46c2-8851-fa4363774e6c-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-qt49s\" (UID: \"24ad3828-5fbb-46c2-8851-fa4363774e6c\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s" Sep 17 01:34:37.173903 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.173764 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsj6q\" (UniqueName: \"kubernetes.io/projected/26da0a42-296e-4280-9529-81cd1c49e8b4-kube-api-access-nsj6q\") pod \"payload-processing-5b54b7fbb7-7xstn\" (UID: \"26da0a42-296e-4280-9529-81cd1c49e8b4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7xstn" Sep 17 01:34:37.173972 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.173897 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26da0a42-296e-4280-9529-81cd1c49e8b4-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7xstn\" (UID: \"26da0a42-296e-4280-9529-81cd1c49e8b4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7xstn" Sep 17 01:34:37.173972 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.173943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bkff\" (UniqueName: \"kubernetes.io/projected/24ad3828-5fbb-46c2-8851-fa4363774e6c-kube-api-access-9bkff\") pod \"payload-pre-processing-59fbc47bf8-qt49s\" (UID: \"24ad3828-5fbb-46c2-8851-fa4363774e6c\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s" Sep 17 01:34:37.275367 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.275333 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26da0a42-296e-4280-9529-81cd1c49e8b4-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7xstn\" (UID: \"26da0a42-296e-4280-9529-81cd1c49e8b4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7xstn" Sep 17 01:34:37.275567 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.275378 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bkff\" (UniqueName: \"kubernetes.io/projected/24ad3828-5fbb-46c2-8851-fa4363774e6c-kube-api-access-9bkff\") pod \"payload-pre-processing-59fbc47bf8-qt49s\" (UID: \"24ad3828-5fbb-46c2-8851-fa4363774e6c\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s" Sep 17 01:34:37.275567 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.275431 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ad3828-5fbb-46c2-8851-fa4363774e6c-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-qt49s\" (UID: \"24ad3828-5fbb-46c2-8851-fa4363774e6c\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s" Sep 17 01:34:37.275567 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.275456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsj6q\" (UniqueName: \"kubernetes.io/projected/26da0a42-296e-4280-9529-81cd1c49e8b4-kube-api-access-nsj6q\") pod \"payload-processing-5b54b7fbb7-7xstn\" (UID: \"26da0a42-296e-4280-9529-81cd1c49e8b4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7xstn" Sep 17 01:34:37.276134 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.276113 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26da0a42-296e-4280-9529-81cd1c49e8b4-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7xstn\" (UID: \"26da0a42-296e-4280-9529-81cd1c49e8b4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7xstn" Sep 17 01:34:37.276185 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.276140 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ad3828-5fbb-46c2-8851-fa4363774e6c-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-qt49s\" (UID: \"24ad3828-5fbb-46c2-8851-fa4363774e6c\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s" Sep 17 01:34:37.289316 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.289294 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsj6q\" (UniqueName: \"kubernetes.io/projected/26da0a42-296e-4280-9529-81cd1c49e8b4-kube-api-access-nsj6q\") pod \"payload-processing-5b54b7fbb7-7xstn\" (UID: \"26da0a42-296e-4280-9529-81cd1c49e8b4\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7xstn" Sep 17 01:34:37.289436 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.289395 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bkff\" (UniqueName: \"kubernetes.io/projected/24ad3828-5fbb-46c2-8851-fa4363774e6c-kube-api-access-9bkff\") pod \"payload-pre-processing-59fbc47bf8-qt49s\" (UID: \"24ad3828-5fbb-46c2-8851-fa4363774e6c\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s" Sep 17 01:34:37.376361 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.376333 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s" Sep 17 01:34:37.409485 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.409436 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7xstn" Sep 17 01:34:37.653326 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.653295 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s"] Sep 17 01:34:37.655165 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:34:37.655140 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ad3828_5fbb_46c2_8851_fa4363774e6c.slice/crio-56f0cc200d19e86efa6138ac98d5aa5b66b87dc191dc55f0734f3d6140a0a74a WatchSource:0}: Error finding container 56f0cc200d19e86efa6138ac98d5aa5b66b87dc191dc55f0734f3d6140a0a74a: Status 404 returned error can't find the container with id 56f0cc200d19e86efa6138ac98d5aa5b66b87dc191dc55f0734f3d6140a0a74a Sep 17 01:34:37.657144 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.657126 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 01:34:37.694729 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:37.694703 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7xstn"] Sep 17 01:34:37.696609 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:34:37.696578 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26da0a42_296e_4280_9529_81cd1c49e8b4.slice/crio-7baa70bbc86e93a440614049268712f1935033c528a69a315c05c20731571480 WatchSource:0}: Error finding container 7baa70bbc86e93a440614049268712f1935033c528a69a315c05c20731571480: Status 404 returned error can't find the container with id 7baa70bbc86e93a440614049268712f1935033c528a69a315c05c20731571480 Sep 17 01:34:38.669098 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:38.669056 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7xstn" event={"ID":"26da0a42-296e-4280-9529-81cd1c49e8b4","Type":"ContainerStarted","Data":"7baa70bbc86e93a440614049268712f1935033c528a69a315c05c20731571480"} Sep 17 01:34:38.670568 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:38.670532 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s" event={"ID":"24ad3828-5fbb-46c2-8851-fa4363774e6c","Type":"ContainerStarted","Data":"56f0cc200d19e86efa6138ac98d5aa5b66b87dc191dc55f0734f3d6140a0a74a"} Sep 17 01:34:39.678430 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:39.678396 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7xstn" event={"ID":"26da0a42-296e-4280-9529-81cd1c49e8b4","Type":"ContainerStarted","Data":"03c009478a1ef80c04733c48b72ce8725b76545e9c5fa4c3b11a869a58c6d510"} Sep 17 01:34:39.678882 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:39.678554 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-7xstn" Sep 17 01:34:39.680129 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:39.680107 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s" event={"ID":"24ad3828-5fbb-46c2-8851-fa4363774e6c","Type":"ContainerStarted","Data":"826f50e1f36dc9897f6c91b382e00d497910885919a2993d6775aad2ee1bdd20"} Sep 17 01:34:39.680249 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:39.680216 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s" Sep 17 01:34:39.694625 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:39.694582 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-7xstn" podStartSLOduration=1.429799588 podStartE2EDuration="2.6945563s" totalImagesPullingTime="1.264756712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:34:39.692212834 +0000 UTC m=+744.927360021" watchObservedRunningTime="2026-09-17 01:34:39.6945563 +0000 UTC m=+744.929703488" Sep 17 01:34:39.718391 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:39.714449 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s" podStartSLOduration=1.405762844 podStartE2EDuration="2.714416603s" totalImagesPullingTime="1.308653759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:34:39.710621224 +0000 UTC m=+744.945768409" watchObservedRunningTime="2026-09-17 01:34:39.714416603 +0000 UTC m=+744.949563791" Sep 17 01:34:50.688446 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:50.688414 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s" Sep 17 01:34:50.688904 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:34:50.688500 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-7xstn" Sep 17 01:37:15.182357 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:15.182325 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 01:37:15.184742 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:15.184715 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 01:37:33.170258 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.170219 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7xstn"] Sep 17 01:37:33.172726 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.170420 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5b54b7fbb7-7xstn" podUID="26da0a42-296e-4280-9529-81cd1c49e8b4" containerName="payload-processing" containerID="cri-o://03c009478a1ef80c04733c48b72ce8725b76545e9c5fa4c3b11a869a58c6d510" gracePeriod=30 Sep 17 01:37:33.257002 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.256968 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s"] Sep 17 01:37:33.257196 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.257169 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s" podUID="24ad3828-5fbb-46c2-8851-fa4363774e6c" containerName="payload-pre-processing" containerID="cri-o://826f50e1f36dc9897f6c91b382e00d497910885919a2993d6775aad2ee1bdd20" gracePeriod=30 Sep 17 01:37:33.501377 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.501345 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7xstn" Sep 17 01:37:33.533463 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.533436 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s" Sep 17 01:37:33.630967 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.630916 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ad3828-5fbb-46c2-8851-fa4363774e6c-plugins-config-volume\") pod \"24ad3828-5fbb-46c2-8851-fa4363774e6c\" (UID: \"24ad3828-5fbb-46c2-8851-fa4363774e6c\") " Sep 17 01:37:33.631141 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.631018 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bkff\" (UniqueName: \"kubernetes.io/projected/24ad3828-5fbb-46c2-8851-fa4363774e6c-kube-api-access-9bkff\") pod \"24ad3828-5fbb-46c2-8851-fa4363774e6c\" (UID: \"24ad3828-5fbb-46c2-8851-fa4363774e6c\") " Sep 17 01:37:33.631141 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.631058 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsj6q\" (UniqueName: \"kubernetes.io/projected/26da0a42-296e-4280-9529-81cd1c49e8b4-kube-api-access-nsj6q\") pod \"26da0a42-296e-4280-9529-81cd1c49e8b4\" (UID: \"26da0a42-296e-4280-9529-81cd1c49e8b4\") " Sep 17 01:37:33.631141 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.631083 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26da0a42-296e-4280-9529-81cd1c49e8b4-plugins-config-volume\") pod \"26da0a42-296e-4280-9529-81cd1c49e8b4\" (UID: \"26da0a42-296e-4280-9529-81cd1c49e8b4\") " Sep 17 01:37:33.631324 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.631277 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ad3828-5fbb-46c2-8851-fa4363774e6c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24ad3828-5fbb-46c2-8851-fa4363774e6c" (UID: "24ad3828-5fbb-46c2-8851-fa4363774e6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:37:33.631529 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.631498 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26da0a42-296e-4280-9529-81cd1c49e8b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26da0a42-296e-4280-9529-81cd1c49e8b4" (UID: "26da0a42-296e-4280-9529-81cd1c49e8b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:37:33.631634 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.631563 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ad3828-5fbb-46c2-8851-fa4363774e6c-plugins-config-volume\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:37:33.633174 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.633144 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ad3828-5fbb-46c2-8851-fa4363774e6c-kube-api-access-9bkff" (OuterVolumeSpecName: "kube-api-access-9bkff") pod "24ad3828-5fbb-46c2-8851-fa4363774e6c" (UID: "24ad3828-5fbb-46c2-8851-fa4363774e6c"). InnerVolumeSpecName "kube-api-access-9bkff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:37:33.633275 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.633202 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26da0a42-296e-4280-9529-81cd1c49e8b4-kube-api-access-nsj6q" (OuterVolumeSpecName: "kube-api-access-nsj6q") pod "26da0a42-296e-4280-9529-81cd1c49e8b4" (UID: "26da0a42-296e-4280-9529-81cd1c49e8b4"). InnerVolumeSpecName "kube-api-access-nsj6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:37:33.678238 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.678203 2577 generic.go:356] "Generic (PLEG): container finished" podID="26da0a42-296e-4280-9529-81cd1c49e8b4" containerID="03c009478a1ef80c04733c48b72ce8725b76545e9c5fa4c3b11a869a58c6d510" exitCode=0 Sep 17 01:37:33.678376 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.678255 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7xstn" Sep 17 01:37:33.678376 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.678292 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7xstn" event={"ID":"26da0a42-296e-4280-9529-81cd1c49e8b4","Type":"ContainerDied","Data":"03c009478a1ef80c04733c48b72ce8725b76545e9c5fa4c3b11a869a58c6d510"} Sep 17 01:37:33.678376 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.678356 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7xstn" event={"ID":"26da0a42-296e-4280-9529-81cd1c49e8b4","Type":"ContainerDied","Data":"7baa70bbc86e93a440614049268712f1935033c528a69a315c05c20731571480"} Sep 17 01:37:33.678543 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.678390 2577 scope.go:117] "RemoveContainer" containerID="03c009478a1ef80c04733c48b72ce8725b76545e9c5fa4c3b11a869a58c6d510" Sep 17 01:37:33.680197 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.680169 2577 generic.go:356] "Generic (PLEG): container finished" podID="24ad3828-5fbb-46c2-8851-fa4363774e6c" containerID="826f50e1f36dc9897f6c91b382e00d497910885919a2993d6775aad2ee1bdd20" exitCode=0 Sep 17 01:37:33.680312 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.680217 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s" Sep 17 01:37:33.680312 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.680248 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s" event={"ID":"24ad3828-5fbb-46c2-8851-fa4363774e6c","Type":"ContainerDied","Data":"826f50e1f36dc9897f6c91b382e00d497910885919a2993d6775aad2ee1bdd20"} Sep 17 01:37:33.680312 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.680291 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s" event={"ID":"24ad3828-5fbb-46c2-8851-fa4363774e6c","Type":"ContainerDied","Data":"56f0cc200d19e86efa6138ac98d5aa5b66b87dc191dc55f0734f3d6140a0a74a"} Sep 17 01:37:33.688333 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.688269 2577 scope.go:117] "RemoveContainer" containerID="03c009478a1ef80c04733c48b72ce8725b76545e9c5fa4c3b11a869a58c6d510" Sep 17 01:37:33.688656 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:37:33.688630 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c009478a1ef80c04733c48b72ce8725b76545e9c5fa4c3b11a869a58c6d510\": container with ID starting with 03c009478a1ef80c04733c48b72ce8725b76545e9c5fa4c3b11a869a58c6d510 not found: ID does not exist" containerID="03c009478a1ef80c04733c48b72ce8725b76545e9c5fa4c3b11a869a58c6d510" Sep 17 01:37:33.688749 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.688669 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c009478a1ef80c04733c48b72ce8725b76545e9c5fa4c3b11a869a58c6d510"} err="failed to get container status \"03c009478a1ef80c04733c48b72ce8725b76545e9c5fa4c3b11a869a58c6d510\": rpc error: code = NotFound desc = could not find container \"03c009478a1ef80c04733c48b72ce8725b76545e9c5fa4c3b11a869a58c6d510\": container with ID starting with 03c009478a1ef80c04733c48b72ce8725b76545e9c5fa4c3b11a869a58c6d510 not found: ID does not exist" Sep 17 01:37:33.688749 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.688692 2577 scope.go:117] "RemoveContainer" containerID="826f50e1f36dc9897f6c91b382e00d497910885919a2993d6775aad2ee1bdd20" Sep 17 01:37:33.697565 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.697543 2577 scope.go:117] "RemoveContainer" containerID="826f50e1f36dc9897f6c91b382e00d497910885919a2993d6775aad2ee1bdd20" Sep 17 01:37:33.697819 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:37:33.697799 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"826f50e1f36dc9897f6c91b382e00d497910885919a2993d6775aad2ee1bdd20\": container with ID starting with 826f50e1f36dc9897f6c91b382e00d497910885919a2993d6775aad2ee1bdd20 not found: ID does not exist" containerID="826f50e1f36dc9897f6c91b382e00d497910885919a2993d6775aad2ee1bdd20" Sep 17 01:37:33.697869 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.697846 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"826f50e1f36dc9897f6c91b382e00d497910885919a2993d6775aad2ee1bdd20"} err="failed to get container status \"826f50e1f36dc9897f6c91b382e00d497910885919a2993d6775aad2ee1bdd20\": rpc error: code = NotFound desc = could not find container \"826f50e1f36dc9897f6c91b382e00d497910885919a2993d6775aad2ee1bdd20\": container with ID starting with 826f50e1f36dc9897f6c91b382e00d497910885919a2993d6775aad2ee1bdd20 not found: ID does not exist" Sep 17 01:37:33.702725 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.702704 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7xstn"] Sep 17 01:37:33.705759 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.705734 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7xstn"] Sep 17 01:37:33.715024 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.715000 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s"] Sep 17 01:37:33.718328 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.718305 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-qt49s"] Sep 17 01:37:33.732388 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.732364 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bkff\" (UniqueName: \"kubernetes.io/projected/24ad3828-5fbb-46c2-8851-fa4363774e6c-kube-api-access-9bkff\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:37:33.732388 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.732387 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsj6q\" (UniqueName: \"kubernetes.io/projected/26da0a42-296e-4280-9529-81cd1c49e8b4-kube-api-access-nsj6q\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:37:33.732515 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:33.732398 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26da0a42-296e-4280-9529-81cd1c49e8b4-plugins-config-volume\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:37:35.219508 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:35.219461 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24ad3828-5fbb-46c2-8851-fa4363774e6c" path="/var/lib/kubelet/pods/24ad3828-5fbb-46c2-8851-fa4363774e6c/volumes" Sep 17 01:37:35.219977 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:35.219926 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26da0a42-296e-4280-9529-81cd1c49e8b4" path="/var/lib/kubelet/pods/26da0a42-296e-4280-9529-81cd1c49e8b4/volumes" Sep 17 01:37:46.716075 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:46.716029 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6b8f44464c-jrzsh"] Sep 17 01:37:46.716694 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:46.716669 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26da0a42-296e-4280-9529-81cd1c49e8b4" containerName="payload-processing" Sep 17 01:37:46.716788 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:46.716697 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="26da0a42-296e-4280-9529-81cd1c49e8b4" containerName="payload-processing" Sep 17 01:37:46.716788 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:46.716721 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ad3828-5fbb-46c2-8851-fa4363774e6c" containerName="payload-pre-processing" Sep 17 01:37:46.716788 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:46.716731 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ad3828-5fbb-46c2-8851-fa4363774e6c" containerName="payload-pre-processing" Sep 17 01:37:46.716952 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:46.716830 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="26da0a42-296e-4280-9529-81cd1c49e8b4" containerName="payload-processing" Sep 17 01:37:46.716952 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:46.716845 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="24ad3828-5fbb-46c2-8851-fa4363774e6c" containerName="payload-pre-processing" Sep 17 01:37:46.719795 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:46.719772 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6b8f44464c-jrzsh" Sep 17 01:37:46.721775 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:46.721745 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-sz4lt\"" Sep 17 01:37:46.721775 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:46.721745 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Sep 17 01:37:46.725708 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:46.725686 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6b8f44464c-jrzsh"] Sep 17 01:37:46.743660 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:46.743638 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6htfd\" (UniqueName: \"kubernetes.io/projected/40d991af-a898-4005-a1ce-21d24acce961-kube-api-access-6htfd\") pod \"ai-gateway-controller-6b8f44464c-jrzsh\" (UID: \"40d991af-a898-4005-a1ce-21d24acce961\") " pod="opendatahub/ai-gateway-controller-6b8f44464c-jrzsh" Sep 17 01:37:46.844698 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:46.844666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6htfd\" (UniqueName: \"kubernetes.io/projected/40d991af-a898-4005-a1ce-21d24acce961-kube-api-access-6htfd\") pod \"ai-gateway-controller-6b8f44464c-jrzsh\" (UID: \"40d991af-a898-4005-a1ce-21d24acce961\") " pod="opendatahub/ai-gateway-controller-6b8f44464c-jrzsh" Sep 17 01:37:46.851748 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:46.851723 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6htfd\" (UniqueName: \"kubernetes.io/projected/40d991af-a898-4005-a1ce-21d24acce961-kube-api-access-6htfd\") pod \"ai-gateway-controller-6b8f44464c-jrzsh\" (UID: \"40d991af-a898-4005-a1ce-21d24acce961\") " pod="opendatahub/ai-gateway-controller-6b8f44464c-jrzsh" Sep 17 01:37:47.031075 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:47.030996 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6b8f44464c-jrzsh" Sep 17 01:37:47.298970 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:47.298943 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6b8f44464c-jrzsh"] Sep 17 01:37:47.300813 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:37:47.300788 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d991af_a898_4005_a1ce_21d24acce961.slice/crio-31d01f3b832c9c07340eaecbc6b7a8d6776a5aab8cfcbf61453044320aa38f80 WatchSource:0}: Error finding container 31d01f3b832c9c07340eaecbc6b7a8d6776a5aab8cfcbf61453044320aa38f80: Status 404 returned error can't find the container with id 31d01f3b832c9c07340eaecbc6b7a8d6776a5aab8cfcbf61453044320aa38f80 Sep 17 01:37:47.769913 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:47.769876 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6b8f44464c-jrzsh" event={"ID":"40d991af-a898-4005-a1ce-21d24acce961","Type":"ContainerStarted","Data":"31d01f3b832c9c07340eaecbc6b7a8d6776a5aab8cfcbf61453044320aa38f80"} Sep 17 01:37:50.265102 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.265021 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-85fd65c94c-wh5fn"] Sep 17 01:37:50.268638 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.268615 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" Sep 17 01:37:50.270659 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.270636 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-qfwsb\"" Sep 17 01:37:50.270795 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.270636 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 17 01:37:50.277197 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.277173 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-85fd65c94c-wh5fn"] Sep 17 01:37:50.296582 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.296544 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-76f657cdc4-p72fm"] Sep 17 01:37:50.300949 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.300894 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" Sep 17 01:37:50.307613 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.307579 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-76f657cdc4-p72fm"] Sep 17 01:37:50.379785 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.379750 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3faa76e-121f-491e-a616-1ac261e85583-plugins-config-volume\") pod \"payload-pre-processing-76f657cdc4-p72fm\" (UID: \"d3faa76e-121f-491e-a616-1ac261e85583\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" Sep 17 01:37:50.379963 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.379801 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a26ca5a-c33d-42f7-96c2-cfa79451016c-plugins-config-volume\") pod \"payload-processing-85fd65c94c-wh5fn\" (UID: \"6a26ca5a-c33d-42f7-96c2-cfa79451016c\") " pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" Sep 17 01:37:50.379963 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.379828 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98qws\" (UniqueName: \"kubernetes.io/projected/6a26ca5a-c33d-42f7-96c2-cfa79451016c-kube-api-access-98qws\") pod \"payload-processing-85fd65c94c-wh5fn\" (UID: \"6a26ca5a-c33d-42f7-96c2-cfa79451016c\") " pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" Sep 17 01:37:50.379963 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.379909 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmnlq\" (UniqueName: \"kubernetes.io/projected/d3faa76e-121f-491e-a616-1ac261e85583-kube-api-access-qmnlq\") pod \"payload-pre-processing-76f657cdc4-p72fm\" (UID: \"d3faa76e-121f-491e-a616-1ac261e85583\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" Sep 17 01:37:50.481291 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.481256 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a26ca5a-c33d-42f7-96c2-cfa79451016c-plugins-config-volume\") pod \"payload-processing-85fd65c94c-wh5fn\" (UID: \"6a26ca5a-c33d-42f7-96c2-cfa79451016c\") " pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" Sep 17 01:37:50.481291 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.481292 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98qws\" (UniqueName: \"kubernetes.io/projected/6a26ca5a-c33d-42f7-96c2-cfa79451016c-kube-api-access-98qws\") pod \"payload-processing-85fd65c94c-wh5fn\" (UID: \"6a26ca5a-c33d-42f7-96c2-cfa79451016c\") " pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" Sep 17 01:37:50.481557 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.481323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmnlq\" (UniqueName: \"kubernetes.io/projected/d3faa76e-121f-491e-a616-1ac261e85583-kube-api-access-qmnlq\") pod \"payload-pre-processing-76f657cdc4-p72fm\" (UID: \"d3faa76e-121f-491e-a616-1ac261e85583\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" Sep 17 01:37:50.481557 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.481415 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3faa76e-121f-491e-a616-1ac261e85583-plugins-config-volume\") pod \"payload-pre-processing-76f657cdc4-p72fm\" (UID: \"d3faa76e-121f-491e-a616-1ac261e85583\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" Sep 17 01:37:50.482232 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.482205 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a26ca5a-c33d-42f7-96c2-cfa79451016c-plugins-config-volume\") pod \"payload-processing-85fd65c94c-wh5fn\" (UID: \"6a26ca5a-c33d-42f7-96c2-cfa79451016c\") " pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" Sep 17 01:37:50.482333 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.482291 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3faa76e-121f-491e-a616-1ac261e85583-plugins-config-volume\") pod \"payload-pre-processing-76f657cdc4-p72fm\" (UID: \"d3faa76e-121f-491e-a616-1ac261e85583\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" Sep 17 01:37:50.489000 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.488971 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmnlq\" (UniqueName: \"kubernetes.io/projected/d3faa76e-121f-491e-a616-1ac261e85583-kube-api-access-qmnlq\") pod \"payload-pre-processing-76f657cdc4-p72fm\" (UID: \"d3faa76e-121f-491e-a616-1ac261e85583\") " pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" Sep 17 01:37:50.489101 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.489012 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98qws\" (UniqueName: \"kubernetes.io/projected/6a26ca5a-c33d-42f7-96c2-cfa79451016c-kube-api-access-98qws\") pod \"payload-processing-85fd65c94c-wh5fn\" (UID: \"6a26ca5a-c33d-42f7-96c2-cfa79451016c\") " pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" Sep 17 01:37:50.581996 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.581922 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" Sep 17 01:37:50.614158 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.614129 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" Sep 17 01:37:50.792943 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.792904 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6b8f44464c-jrzsh" event={"ID":"40d991af-a898-4005-a1ce-21d24acce961","Type":"ContainerStarted","Data":"d7516e4662ce82427b8be05405fffb235a896bab5c1b09df2935690825cc175a"} Sep 17 01:37:50.793089 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.792986 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6b8f44464c-jrzsh" Sep 17 01:37:50.809464 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.809419 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6b8f44464c-jrzsh" podStartSLOduration=2.2495342210000002 podStartE2EDuration="4.809402395s" totalImagesPullingTime="2.559868174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:50.806575247 +0000 UTC m=+936.041722436" watchObservedRunningTime="2026-09-17 01:37:50.809402395 +0000 UTC m=+936.044549584" Sep 17 01:37:50.856416 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.856381 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-85fd65c94c-wh5fn"] Sep 17 01:37:50.857594 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:37:50.857567 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a26ca5a_c33d_42f7_96c2_cfa79451016c.slice/crio-557a16d767ee3202a94a8b599d37eda38c69f4e64cc0dfd2fa72cfa9763bff46 WatchSource:0}: Error finding container 557a16d767ee3202a94a8b599d37eda38c69f4e64cc0dfd2fa72cfa9763bff46: Status 404 returned error can't find the container with id 557a16d767ee3202a94a8b599d37eda38c69f4e64cc0dfd2fa72cfa9763bff46 Sep 17 01:37:50.898471 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:50.898444 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-76f657cdc4-p72fm"] Sep 17 01:37:50.899206 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:37:50.899176 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3faa76e_121f_491e_a616_1ac261e85583.slice/crio-61ab3b750b976ba94a524654482299aa04a826e46e2e46ec0623c4c4e9cdb4cc WatchSource:0}: Error finding container 61ab3b750b976ba94a524654482299aa04a826e46e2e46ec0623c4c4e9cdb4cc: Status 404 returned error can't find the container with id 61ab3b750b976ba94a524654482299aa04a826e46e2e46ec0623c4c4e9cdb4cc Sep 17 01:37:51.800649 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:51.800614 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" event={"ID":"6a26ca5a-c33d-42f7-96c2-cfa79451016c","Type":"ContainerStarted","Data":"557a16d767ee3202a94a8b599d37eda38c69f4e64cc0dfd2fa72cfa9763bff46"} Sep 17 01:37:51.801868 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:51.801839 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" event={"ID":"d3faa76e-121f-491e-a616-1ac261e85583","Type":"ContainerStarted","Data":"61ab3b750b976ba94a524654482299aa04a826e46e2e46ec0623c4c4e9cdb4cc"} Sep 17 01:37:52.809094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:52.809060 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" event={"ID":"d3faa76e-121f-491e-a616-1ac261e85583","Type":"ContainerStarted","Data":"d16e566d4844e94f164b690aad32d0aec552f6c2304629f0b37a31ae82fa609b"} Sep 17 01:37:52.809547 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:52.809157 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" Sep 17 01:37:52.810634 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:52.810609 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" event={"ID":"6a26ca5a-c33d-42f7-96c2-cfa79451016c","Type":"ContainerStarted","Data":"5030e38ff5a8f83d7c4f885513ea0c443f92a448c405132c191a5c6d02c6af1e"} Sep 17 01:37:52.810739 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:52.810726 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" Sep 17 01:37:52.824419 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:52.824371 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" podStartSLOduration=1.85640452 podStartE2EDuration="2.82435746s" totalImagesPullingTime="967.95294ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:52.823122262 +0000 UTC m=+938.058269461" watchObservedRunningTime="2026-09-17 01:37:52.82435746 +0000 UTC m=+938.059504648" Sep 17 01:37:52.838394 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:37:52.838346 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" podStartSLOduration=1.830844331 podStartE2EDuration="2.838333243s" totalImagesPullingTime="1.007488912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:37:52.836301259 +0000 UTC m=+938.071448448" watchObservedRunningTime="2026-09-17 01:37:52.838333243 +0000 UTC m=+938.073480430" Sep 17 01:38:01.804569 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:01.804534 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6b8f44464c-jrzsh" Sep 17 01:38:01.978323 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:01.978285 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6bbfbbc97b-krnlb"] Sep 17 01:38:01.982036 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:01.982011 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6bbfbbc97b-krnlb" Sep 17 01:38:01.990960 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:01.990910 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6bbfbbc97b-krnlb"] Sep 17 01:38:02.085679 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:02.085602 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rqwz\" (UniqueName: \"kubernetes.io/projected/e13e08b9-5f27-4c88-9dbb-b8ec40a0ff44-kube-api-access-4rqwz\") pod \"ai-gateway-controller-6bbfbbc97b-krnlb\" (UID: \"e13e08b9-5f27-4c88-9dbb-b8ec40a0ff44\") " pod="opendatahub/ai-gateway-controller-6bbfbbc97b-krnlb" Sep 17 01:38:02.186781 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:02.186744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rqwz\" (UniqueName: \"kubernetes.io/projected/e13e08b9-5f27-4c88-9dbb-b8ec40a0ff44-kube-api-access-4rqwz\") pod \"ai-gateway-controller-6bbfbbc97b-krnlb\" (UID: \"e13e08b9-5f27-4c88-9dbb-b8ec40a0ff44\") " pod="opendatahub/ai-gateway-controller-6bbfbbc97b-krnlb" Sep 17 01:38:02.194300 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:02.194265 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rqwz\" (UniqueName: \"kubernetes.io/projected/e13e08b9-5f27-4c88-9dbb-b8ec40a0ff44-kube-api-access-4rqwz\") pod \"ai-gateway-controller-6bbfbbc97b-krnlb\" (UID: \"e13e08b9-5f27-4c88-9dbb-b8ec40a0ff44\") " pod="opendatahub/ai-gateway-controller-6bbfbbc97b-krnlb" Sep 17 01:38:02.294427 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:02.294396 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6bbfbbc97b-krnlb" Sep 17 01:38:02.563489 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:02.563458 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6bbfbbc97b-krnlb"] Sep 17 01:38:02.564987 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:38:02.564959 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13e08b9_5f27_4c88_9dbb_b8ec40a0ff44.slice/crio-576ce9c02f70a087ba9cd23a73d5af38945e4f85427021032028dff5d0aa9852 WatchSource:0}: Error finding container 576ce9c02f70a087ba9cd23a73d5af38945e4f85427021032028dff5d0aa9852: Status 404 returned error can't find the container with id 576ce9c02f70a087ba9cd23a73d5af38945e4f85427021032028dff5d0aa9852 Sep 17 01:38:02.878677 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:02.878641 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6bbfbbc97b-krnlb" event={"ID":"e13e08b9-5f27-4c88-9dbb-b8ec40a0ff44","Type":"ContainerStarted","Data":"576ce9c02f70a087ba9cd23a73d5af38945e4f85427021032028dff5d0aa9852"} Sep 17 01:38:03.819399 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:03.819368 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" Sep 17 01:38:03.819691 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:03.819670 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" Sep 17 01:38:03.819753 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:03.819730 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:38:03.820017 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:03.820001 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:38:03.886540 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:03.886510 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6bbfbbc97b-krnlb" event={"ID":"e13e08b9-5f27-4c88-9dbb-b8ec40a0ff44","Type":"ContainerStarted","Data":"37e6b1dae2fdfff348df9dccf65abe12a437ffaea8c545d7809ac161f1a98a4c"} Sep 17 01:38:03.886906 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:03.886648 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6bbfbbc97b-krnlb" Sep 17 01:38:03.901559 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:03.901511 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6bbfbbc97b-krnlb" podStartSLOduration=2.531151384 podStartE2EDuration="2.901497904s" totalImagesPullingTime="370.34652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:38:03.899122029 +0000 UTC m=+949.134269216" watchObservedRunningTime="2026-09-17 01:38:03.901497904 +0000 UTC m=+949.136645091" Sep 17 01:38:14.894859 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:14.894821 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6bbfbbc97b-krnlb" Sep 17 01:38:14.932389 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:14.932345 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6b8f44464c-jrzsh"] Sep 17 01:38:14.932576 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:14.932540 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-6b8f44464c-jrzsh" podUID="40d991af-a898-4005-a1ce-21d24acce961" containerName="manager" containerID="cri-o://d7516e4662ce82427b8be05405fffb235a896bab5c1b09df2935690825cc175a" gracePeriod=30 Sep 17 01:38:15.186923 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:15.186894 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6b8f44464c-jrzsh" Sep 17 01:38:15.309808 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:15.309714 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6htfd\" (UniqueName: \"kubernetes.io/projected/40d991af-a898-4005-a1ce-21d24acce961-kube-api-access-6htfd\") pod \"40d991af-a898-4005-a1ce-21d24acce961\" (UID: \"40d991af-a898-4005-a1ce-21d24acce961\") " Sep 17 01:38:15.311815 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:15.311790 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d991af-a898-4005-a1ce-21d24acce961-kube-api-access-6htfd" (OuterVolumeSpecName: "kube-api-access-6htfd") pod "40d991af-a898-4005-a1ce-21d24acce961" (UID: "40d991af-a898-4005-a1ce-21d24acce961"). InnerVolumeSpecName "kube-api-access-6htfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:38:15.410615 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:15.410529 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6htfd\" (UniqueName: \"kubernetes.io/projected/40d991af-a898-4005-a1ce-21d24acce961-kube-api-access-6htfd\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:38:15.958526 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:15.958488 2577 generic.go:356] "Generic (PLEG): container finished" podID="40d991af-a898-4005-a1ce-21d24acce961" containerID="d7516e4662ce82427b8be05405fffb235a896bab5c1b09df2935690825cc175a" exitCode=0 Sep 17 01:38:15.958955 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:15.958537 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6b8f44464c-jrzsh" event={"ID":"40d991af-a898-4005-a1ce-21d24acce961","Type":"ContainerDied","Data":"d7516e4662ce82427b8be05405fffb235a896bab5c1b09df2935690825cc175a"} Sep 17 01:38:15.958955 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:15.958578 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6b8f44464c-jrzsh" event={"ID":"40d991af-a898-4005-a1ce-21d24acce961","Type":"ContainerDied","Data":"31d01f3b832c9c07340eaecbc6b7a8d6776a5aab8cfcbf61453044320aa38f80"} Sep 17 01:38:15.958955 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:15.958603 2577 scope.go:117] "RemoveContainer" containerID="d7516e4662ce82427b8be05405fffb235a896bab5c1b09df2935690825cc175a" Sep 17 01:38:15.958955 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:15.958538 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6b8f44464c-jrzsh" Sep 17 01:38:15.969147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:15.968510 2577 scope.go:117] "RemoveContainer" containerID="d7516e4662ce82427b8be05405fffb235a896bab5c1b09df2935690825cc175a" Sep 17 01:38:15.969354 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:38:15.969321 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7516e4662ce82427b8be05405fffb235a896bab5c1b09df2935690825cc175a\": container with ID starting with d7516e4662ce82427b8be05405fffb235a896bab5c1b09df2935690825cc175a not found: ID does not exist" containerID="d7516e4662ce82427b8be05405fffb235a896bab5c1b09df2935690825cc175a" Sep 17 01:38:15.969421 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:15.969368 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7516e4662ce82427b8be05405fffb235a896bab5c1b09df2935690825cc175a"} err="failed to get container status \"d7516e4662ce82427b8be05405fffb235a896bab5c1b09df2935690825cc175a\": rpc error: code = NotFound desc = could not find container \"d7516e4662ce82427b8be05405fffb235a896bab5c1b09df2935690825cc175a\": container with ID starting with d7516e4662ce82427b8be05405fffb235a896bab5c1b09df2935690825cc175a not found: ID does not exist" Sep 17 01:38:15.982074 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:15.982039 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6b8f44464c-jrzsh"] Sep 17 01:38:15.984256 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:15.984234 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-6b8f44464c-jrzsh"] Sep 17 01:38:17.212209 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:38:17.212175 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40d991af-a898-4005-a1ce-21d24acce961" path="/var/lib/kubelet/pods/40d991af-a898-4005-a1ce-21d24acce961/volumes" Sep 17 01:39:04.608569 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.608528 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92"] Sep 17 01:39:04.609195 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.609175 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d991af-a898-4005-a1ce-21d24acce961" containerName="manager" Sep 17 01:39:04.609195 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.609196 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d991af-a898-4005-a1ce-21d24acce961" containerName="manager" Sep 17 01:39:04.609297 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.609288 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="40d991af-a898-4005-a1ce-21d24acce961" containerName="manager" Sep 17 01:39:04.612451 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.612434 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.614846 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.614819 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-cc98e0-openshift-default-dockercfg-gbhqx\"" Sep 17 01:39:04.624013 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.623986 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92"] Sep 17 01:39:04.755310 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.755275 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-token\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.755310 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.755310 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-envoy\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.755544 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.755332 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-workload-certs\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.755544 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.755376 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-podinfo\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.755544 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.755406 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqmqn\" (UniqueName: \"kubernetes.io/projected/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-kube-api-access-jqmqn\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.755544 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.755512 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-data\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.755544 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.755534 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istiod-ca-cert\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.755788 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.755606 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-workload-socket\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.755788 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.755666 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-credential-socket\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.811437 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.811404 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c"] Sep 17 01:39:04.815012 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.814995 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:04.817225 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.817199 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d8d689-openshift-default-dockercfg-j6t5w\"" Sep 17 01:39:04.826631 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.826604 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c"] Sep 17 01:39:04.856543 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.856515 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-token\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.856662 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.856548 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-envoy\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.856662 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.856569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-workload-certs\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.856753 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.856675 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-podinfo\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.856753 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.856709 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmqn\" (UniqueName: \"kubernetes.io/projected/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-kube-api-access-jqmqn\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.856833 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.856771 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-data\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.856833 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.856808 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istiod-ca-cert\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.856958 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.856862 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-workload-socket\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.856958 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.856914 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-credential-socket\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.857074 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.856988 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-workload-certs\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.857206 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.857178 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-data\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.857324 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.857249 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-workload-socket\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.857391 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.857340 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-credential-socket\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.857544 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.857517 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istiod-ca-cert\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.858886 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.858837 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-envoy\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.859107 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.859088 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-podinfo\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.863822 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.863796 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-token\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.863967 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.863951 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmqn\" (UniqueName: \"kubernetes.io/projected/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-kube-api-access-jqmqn\") pod \"e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.925646 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.925615 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:04.958082 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.958043 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-workload-socket\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:04.958214 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.958091 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-istio-envoy\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:04.958214 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.958125 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-credential-socket\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:04.958214 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.958146 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/010061bf-7ab5-4623-826a-545d286b26e9-istiod-ca-cert\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:04.958214 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.958179 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/010061bf-7ab5-4623-826a-545d286b26e9-istio-podinfo\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:04.958434 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.958261 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010061bf-7ab5-4623-826a-545d286b26e9-istio-token\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:04.958434 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.958331 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-workload-certs\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:04.958434 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.958393 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69xr8\" (UniqueName: \"kubernetes.io/projected/010061bf-7ab5-4623-826a-545d286b26e9-kube-api-access-69xr8\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:04.958581 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:04.958438 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-istio-data\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.059087 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.059051 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-credential-socket\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.059087 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.059087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/010061bf-7ab5-4623-826a-545d286b26e9-istiod-ca-cert\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.059299 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.059109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/010061bf-7ab5-4623-826a-545d286b26e9-istio-podinfo\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.059299 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.059230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010061bf-7ab5-4623-826a-545d286b26e9-istio-token\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.059408 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.059310 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-workload-certs\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.059408 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.059383 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69xr8\" (UniqueName: \"kubernetes.io/projected/010061bf-7ab5-4623-826a-545d286b26e9-kube-api-access-69xr8\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.059516 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.059427 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-istio-data\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.059516 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.059458 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-workload-socket\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.059516 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.059506 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-istio-envoy\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.059676 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.059576 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-credential-socket\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.059676 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.059630 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-workload-certs\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.059809 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.059786 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-istio-data\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.059914 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.059894 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-workload-socket\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.060037 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.060018 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/010061bf-7ab5-4623-826a-545d286b26e9-istiod-ca-cert\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.061682 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.061663 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-istio-envoy\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.061782 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.061718 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/010061bf-7ab5-4623-826a-545d286b26e9-istio-podinfo\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.070776 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.070756 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010061bf-7ab5-4623-826a-545d286b26e9-istio-token\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.070869 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.070799 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69xr8\" (UniqueName: \"kubernetes.io/projected/010061bf-7ab5-4623-826a-545d286b26e9-kube-api-access-69xr8\") pod \"e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.128734 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.128670 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:05.212017 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:39:05.211974 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdfe78aa_ff8e_4e7b_9ca7_84d1598bfdf1.slice/crio-b8079016b4475e1468ee2f5408f4625dbdfd878e86cd2104cd4d4a7e782d266f WatchSource:0}: Error finding container b8079016b4475e1468ee2f5408f4625dbdfd878e86cd2104cd4d4a7e782d266f: Status 404 returned error can't find the container with id b8079016b4475e1468ee2f5408f4625dbdfd878e86cd2104cd4d4a7e782d266f Sep 17 01:39:05.214351 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.214317 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:39:05.214468 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.214394 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:39:05.214468 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.214435 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:39:05.215504 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.215481 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92"] Sep 17 01:39:05.261533 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.261504 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" event={"ID":"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1","Type":"ContainerStarted","Data":"b8079016b4475e1468ee2f5408f4625dbdfd878e86cd2104cd4d4a7e782d266f"} Sep 17 01:39:05.402426 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:39:05.402382 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010061bf_7ab5_4623_826a_545d286b26e9.slice/crio-0de16394d0e785e7e43f24e10327790ba5346baf19f40495da8b309803d6c95b WatchSource:0}: Error finding container 0de16394d0e785e7e43f24e10327790ba5346baf19f40495da8b309803d6c95b: Status 404 returned error can't find the container with id 0de16394d0e785e7e43f24e10327790ba5346baf19f40495da8b309803d6c95b Sep 17 01:39:05.402672 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.402646 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c"] Sep 17 01:39:05.404828 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.404795 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:39:05.404913 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.404869 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:39:05.404913 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:05.404906 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:39:06.269528 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:06.269488 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" event={"ID":"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1","Type":"ContainerStarted","Data":"5f07311e4c731682d82c04d67c9ced9ece88b36003bf62c37f0b98cd0c39f5c1"} Sep 17 01:39:06.271471 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:06.271440 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" event={"ID":"010061bf-7ab5-4623-826a-545d286b26e9","Type":"ContainerStarted","Data":"689714ffb0a4e2337dddc5a3d7545aaee97ec94055b54958534875ae58f9c27b"} Sep 17 01:39:06.271611 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:06.271493 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" event={"ID":"010061bf-7ab5-4623-826a-545d286b26e9","Type":"ContainerStarted","Data":"0de16394d0e785e7e43f24e10327790ba5346baf19f40495da8b309803d6c95b"} Sep 17 01:39:06.291057 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:06.290997 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" podStartSLOduration=2.290977278 podStartE2EDuration="2.290977278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:39:06.286147134 +0000 UTC m=+1011.521294321" watchObservedRunningTime="2026-09-17 01:39:06.290977278 +0000 UTC m=+1011.526124469" Sep 17 01:39:06.305802 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:06.305753 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" podStartSLOduration=2.305742796 podStartE2EDuration="2.305742796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:39:06.304501471 +0000 UTC m=+1011.539648660" watchObservedRunningTime="2026-09-17 01:39:06.305742796 +0000 UTC m=+1011.540889984" Sep 17 01:39:06.510105 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:06.510064 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10056-68e69d"] Sep 17 01:39:06.513959 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:06.513912 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10056-68e69d" Sep 17 01:39:06.516103 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:06.516076 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qszjr\"" Sep 17 01:39:06.517454 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:06.517431 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10056-68e69d"] Sep 17 01:39:06.674315 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:06.674280 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls492\" (UniqueName: \"kubernetes.io/projected/1e6de695-0711-43a7-8801-281bcc4bde1b-kube-api-access-ls492\") pod \"test-curl-10056-68e69d\" (UID: \"1e6de695-0711-43a7-8801-281bcc4bde1b\") " pod="opendatahub/test-curl-10056-68e69d" Sep 17 01:39:06.775669 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:06.775577 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls492\" (UniqueName: \"kubernetes.io/projected/1e6de695-0711-43a7-8801-281bcc4bde1b-kube-api-access-ls492\") pod \"test-curl-10056-68e69d\" (UID: \"1e6de695-0711-43a7-8801-281bcc4bde1b\") " pod="opendatahub/test-curl-10056-68e69d" Sep 17 01:39:06.782717 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:06.782690 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls492\" (UniqueName: \"kubernetes.io/projected/1e6de695-0711-43a7-8801-281bcc4bde1b-kube-api-access-ls492\") pod \"test-curl-10056-68e69d\" (UID: \"1e6de695-0711-43a7-8801-281bcc4bde1b\") " pod="opendatahub/test-curl-10056-68e69d" Sep 17 01:39:06.826261 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:06.826238 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10056-68e69d" Sep 17 01:39:06.926620 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:06.926591 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:06.932077 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:06.932040 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:07.095685 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:07.095656 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10056-68e69d"] Sep 17 01:39:07.097528 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:39:07.097498 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e6de695_0711_43a7_8801_281bcc4bde1b.slice/crio-8a49e861e1d63d37f9da481ea42499990e4985b0fcdd1c1ab9d14157ba496e25 WatchSource:0}: Error finding container 8a49e861e1d63d37f9da481ea42499990e4985b0fcdd1c1ab9d14157ba496e25: Status 404 returned error can't find the container with id 8a49e861e1d63d37f9da481ea42499990e4985b0fcdd1c1ab9d14157ba496e25 Sep 17 01:39:07.129380 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:07.129354 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:07.134588 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:07.134568 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:07.280266 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:07.280230 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10056-68e69d" event={"ID":"1e6de695-0711-43a7-8801-281bcc4bde1b","Type":"ContainerStarted","Data":"8a49e861e1d63d37f9da481ea42499990e4985b0fcdd1c1ab9d14157ba496e25"} Sep 17 01:39:07.280663 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:07.280434 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:07.280728 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:07.280708 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:07.281504 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:07.281482 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:07.281623 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:07.281540 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:08.287526 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:08.287487 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10056-68e69d" event={"ID":"1e6de695-0711-43a7-8801-281bcc4bde1b","Type":"ContainerStarted","Data":"43856958f25f05c4e2d710a7faf6b0a36d702b1c11abae892baaa2d7beb5e719"} Sep 17 01:39:08.300029 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:08.299986 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10056-68e69d" podStartSLOduration=1.7095890740000002 podStartE2EDuration="2.299972055s" totalImagesPullingTime="590.382981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:39:08.299738435 +0000 UTC m=+1013.534885623" watchObservedRunningTime="2026-09-17 01:39:08.299972055 +0000 UTC m=+1013.535119242" Sep 17 01:39:08.835439 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:08.835401 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10056-68e69d"] Sep 17 01:39:08.841568 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:08.841539 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10056-68e69d"] Sep 17 01:39:08.953780 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:08.953697 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10056-b90d98"] Sep 17 01:39:08.954172 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:08.954157 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e6de695-0711-43a7-8801-281bcc4bde1b" containerName="test-curl-10056-68e69d" Sep 17 01:39:08.954216 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:08.954174 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e6de695-0711-43a7-8801-281bcc4bde1b" containerName="test-curl-10056-68e69d" Sep 17 01:39:08.954254 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:08.954241 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e6de695-0711-43a7-8801-281bcc4bde1b" containerName="test-curl-10056-68e69d" Sep 17 01:39:08.957527 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:08.957510 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10056-b90d98" Sep 17 01:39:08.972650 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:08.972613 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10056-b90d98"] Sep 17 01:39:09.099431 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:09.099393 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj4rb\" (UniqueName: \"kubernetes.io/projected/b4a10093-3819-4482-8514-f1e8851ed019-kube-api-access-tj4rb\") pod \"test-curl-10056-b90d98\" (UID: \"b4a10093-3819-4482-8514-f1e8851ed019\") " pod="opendatahub/test-curl-10056-b90d98" Sep 17 01:39:09.200388 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:09.200348 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj4rb\" (UniqueName: \"kubernetes.io/projected/b4a10093-3819-4482-8514-f1e8851ed019-kube-api-access-tj4rb\") pod \"test-curl-10056-b90d98\" (UID: \"b4a10093-3819-4482-8514-f1e8851ed019\") " pod="opendatahub/test-curl-10056-b90d98" Sep 17 01:39:09.215746 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:09.215677 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj4rb\" (UniqueName: \"kubernetes.io/projected/b4a10093-3819-4482-8514-f1e8851ed019-kube-api-access-tj4rb\") pod \"test-curl-10056-b90d98\" (UID: \"b4a10093-3819-4482-8514-f1e8851ed019\") " pod="opendatahub/test-curl-10056-b90d98" Sep 17 01:39:09.268484 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:09.268449 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10056-b90d98" Sep 17 01:39:09.294589 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:39:09.294559 2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10056-68e69d" container="test-curl-10056-68e69d" Sep 17 01:39:09.538921 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:09.538892 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10056-b90d98"] Sep 17 01:39:09.540334 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:39:09.540298 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4a10093_3819_4482_8514_f1e8851ed019.slice/crio-3460d0b85160e7f6ce31fa9866c3d872a4a65504c9e6106a62edd391acb85094 WatchSource:0}: Error finding container 3460d0b85160e7f6ce31fa9866c3d872a4a65504c9e6106a62edd391acb85094: Status 404 returned error can't find the container with id 3460d0b85160e7f6ce31fa9866c3d872a4a65504c9e6106a62edd391acb85094 Sep 17 01:39:10.302099 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.302063 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10056-b90d98" event={"ID":"b4a10093-3819-4482-8514-f1e8851ed019","Type":"ContainerStarted","Data":"bd1ecacadf269d25f49d27f89a3f04a20fae26d53ab1c4f4fe839bbb010f15bd"} Sep 17 01:39:10.302515 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.302108 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10056-b90d98" event={"ID":"b4a10093-3819-4482-8514-f1e8851ed019","Type":"ContainerStarted","Data":"3460d0b85160e7f6ce31fa9866c3d872a4a65504c9e6106a62edd391acb85094"} Sep 17 01:39:10.302515 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.302175 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10056-68e69d" podUID="1e6de695-0711-43a7-8801-281bcc4bde1b" containerName="test-curl-10056-68e69d" containerID="cri-o://43856958f25f05c4e2d710a7faf6b0a36d702b1c11abae892baaa2d7beb5e719" gracePeriod=2 Sep 17 01:39:10.317569 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.317519 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10056-b90d98" podStartSLOduration=2.317503401 podStartE2EDuration="2.317503401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:39:10.316651745 +0000 UTC m=+1015.551798934" watchObservedRunningTime="2026-09-17 01:39:10.317503401 +0000 UTC m=+1015.552650631" Sep 17 01:39:10.669316 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.669284 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d"] Sep 17 01:39:10.673057 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.673040 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d" Sep 17 01:39:10.675671 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.675646 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-8be734-dockercfg-4nqzj\"" Sep 17 01:39:10.675796 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.675712 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-8be734\"" Sep 17 01:39:10.687572 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.687545 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d"] Sep 17 01:39:10.710174 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.710146 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt"] Sep 17 01:39:10.713802 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.713782 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt" Sep 17 01:39:10.731735 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.731710 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt"] Sep 17 01:39:10.815453 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.815418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dghf7\" (UniqueName: \"kubernetes.io/projected/81a31583-90a7-4f7a-90e0-4870220056b1-kube-api-access-dghf7\") pod \"payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d\" (UID: \"81a31583-90a7-4f7a-90e0-4870220056b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d" Sep 17 01:39:10.815453 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.815459 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt\" (UID: \"b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt" Sep 17 01:39:10.815689 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.815586 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a31583-90a7-4f7a-90e0-4870220056b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d\" (UID: \"81a31583-90a7-4f7a-90e0-4870220056b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d" Sep 17 01:39:10.815740 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.815686 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lglz\" (UniqueName: \"kubernetes.io/projected/b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6-kube-api-access-4lglz\") pod \"payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt\" (UID: \"b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt" Sep 17 01:39:10.917141 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.917033 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dghf7\" (UniqueName: \"kubernetes.io/projected/81a31583-90a7-4f7a-90e0-4870220056b1-kube-api-access-dghf7\") pod \"payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d\" (UID: \"81a31583-90a7-4f7a-90e0-4870220056b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d" Sep 17 01:39:10.917141 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.917093 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt\" (UID: \"b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt" Sep 17 01:39:10.917377 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.917144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a31583-90a7-4f7a-90e0-4870220056b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d\" (UID: \"81a31583-90a7-4f7a-90e0-4870220056b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d" Sep 17 01:39:10.917377 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.917255 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lglz\" (UniqueName: \"kubernetes.io/projected/b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6-kube-api-access-4lglz\") pod \"payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt\" (UID: \"b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt" Sep 17 01:39:10.918021 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.917989 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a31583-90a7-4f7a-90e0-4870220056b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d\" (UID: \"81a31583-90a7-4f7a-90e0-4870220056b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d" Sep 17 01:39:10.918321 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.918299 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt\" (UID: \"b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt" Sep 17 01:39:10.921299 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.921276 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10056-b90d98"] Sep 17 01:39:10.925499 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.925061 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10056-b90d98"] Sep 17 01:39:10.925499 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.925433 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lglz\" (UniqueName: \"kubernetes.io/projected/b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6-kube-api-access-4lglz\") pod \"payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt\" (UID: \"b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt" Sep 17 01:39:10.926157 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.926132 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dghf7\" (UniqueName: \"kubernetes.io/projected/81a31583-90a7-4f7a-90e0-4870220056b1-kube-api-access-dghf7\") pod \"payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d\" (UID: \"81a31583-90a7-4f7a-90e0-4870220056b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d" Sep 17 01:39:10.983869 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:10.983841 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d" Sep 17 01:39:11.023999 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:11.023966 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt" Sep 17 01:39:11.264496 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:11.264460 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d"] Sep 17 01:39:11.265577 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:39:11.265550 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a31583_90a7_4f7a_90e0_4870220056b1.slice/crio-09d2b115e225fb9c7beb363a270f5853ba07676784c798645406b0f2eb18dd78 WatchSource:0}: Error finding container 09d2b115e225fb9c7beb363a270f5853ba07676784c798645406b0f2eb18dd78: Status 404 returned error can't find the container with id 09d2b115e225fb9c7beb363a270f5853ba07676784c798645406b0f2eb18dd78 Sep 17 01:39:11.309885 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:11.309838 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt"] Sep 17 01:39:11.310900 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:39:11.310874 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d0bc32_5bcd_4cc0_8e45_558b9b9356b6.slice/crio-282fccdc63f0a4d273b21ec610c177b3a0245e1a33f14f3dcf1e714be4f58282 WatchSource:0}: Error finding container 282fccdc63f0a4d273b21ec610c177b3a0245e1a33f14f3dcf1e714be4f58282: Status 404 returned error can't find the container with id 282fccdc63f0a4d273b21ec610c177b3a0245e1a33f14f3dcf1e714be4f58282 Sep 17 01:39:11.311206 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:11.311171 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d" event={"ID":"81a31583-90a7-4f7a-90e0-4870220056b1","Type":"ContainerStarted","Data":"09d2b115e225fb9c7beb363a270f5853ba07676784c798645406b0f2eb18dd78"} Sep 17 01:39:11.311491 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:39:11.311472 2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10056-b90d98" container="test-curl-10056-b90d98" Sep 17 01:39:11.727459 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:11.727426 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr"] Sep 17 01:39:11.727909 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:11.727890 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a10093-3819-4482-8514-f1e8851ed019" containerName="test-curl-10056-b90d98" Sep 17 01:39:11.727909 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:11.727908 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a10093-3819-4482-8514-f1e8851ed019" containerName="test-curl-10056-b90d98" Sep 17 01:39:11.728108 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:11.728020 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4a10093-3819-4482-8514-f1e8851ed019" containerName="test-curl-10056-b90d98" Sep 17 01:39:11.731313 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:11.731295 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr" Sep 17 01:39:11.746389 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:11.734190 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-cc98e0\"" Sep 17 01:39:11.746389 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:11.744676 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-cc98e0-dockercfg-4h6nl\"" Sep 17 01:39:11.748865 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:11.748837 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr"] Sep 17 01:39:11.826764 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:11.826715 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6fbf20-070e-4f55-84e1-ff4021d0570a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr\" (UID: \"cf6fbf20-070e-4f55-84e1-ff4021d0570a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr" Sep 17 01:39:11.826955 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:11.826773 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzdsr\" (UniqueName: \"kubernetes.io/projected/cf6fbf20-070e-4f55-84e1-ff4021d0570a-kube-api-access-pzdsr\") pod \"payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr\" (UID: \"cf6fbf20-070e-4f55-84e1-ff4021d0570a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr" Sep 17 01:39:11.928178 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:11.928132 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6fbf20-070e-4f55-84e1-ff4021d0570a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr\" (UID: \"cf6fbf20-070e-4f55-84e1-ff4021d0570a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr" Sep 17 01:39:11.928178 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:11.928179 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzdsr\" (UniqueName: \"kubernetes.io/projected/cf6fbf20-070e-4f55-84e1-ff4021d0570a-kube-api-access-pzdsr\") pod \"payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr\" (UID: \"cf6fbf20-070e-4f55-84e1-ff4021d0570a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr" Sep 17 01:39:11.929109 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:11.929085 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6fbf20-070e-4f55-84e1-ff4021d0570a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr\" (UID: \"cf6fbf20-070e-4f55-84e1-ff4021d0570a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr" Sep 17 01:39:11.936429 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:11.936401 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzdsr\" (UniqueName: \"kubernetes.io/projected/cf6fbf20-070e-4f55-84e1-ff4021d0570a-kube-api-access-pzdsr\") pod \"payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr\" (UID: \"cf6fbf20-070e-4f55-84e1-ff4021d0570a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr" Sep 17 01:39:12.052079 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.051979 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr" Sep 17 01:39:12.322826 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.322794 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d" event={"ID":"81a31583-90a7-4f7a-90e0-4870220056b1","Type":"ContainerStarted","Data":"8d2d294ce7da059cdfe09f9f3bd21c6b1b86bad2ca74c54a7fc28c634bd3b848"} Sep 17 01:39:12.323225 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.322917 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d" Sep 17 01:39:12.324779 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.324751 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt" event={"ID":"b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6","Type":"ContainerStarted","Data":"c31983f21c84e7fe5412dbc9e4002d994f8036c703fc7a55a596872eb36ee7c9"} Sep 17 01:39:12.324902 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.324810 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt" event={"ID":"b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6","Type":"ContainerStarted","Data":"282fccdc63f0a4d273b21ec610c177b3a0245e1a33f14f3dcf1e714be4f58282"} Sep 17 01:39:12.324902 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.324839 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt" Sep 17 01:39:12.325058 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.324914 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10056-b90d98" podUID="b4a10093-3819-4482-8514-f1e8851ed019" containerName="test-curl-10056-b90d98" containerID="cri-o://bd1ecacadf269d25f49d27f89a3f04a20fae26d53ab1c4f4fe839bbb010f15bd" gracePeriod=2 Sep 17 01:39:12.331581 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.331555 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr"] Sep 17 01:39:12.342912 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:39:12.342883 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6fbf20_070e_4f55_84e1_ff4021d0570a.slice/crio-d15c2ca2ead1168647dbed407f696cf326c06974dcc7bc10ea47ae18afc314a6 WatchSource:0}: Error finding container d15c2ca2ead1168647dbed407f696cf326c06974dcc7bc10ea47ae18afc314a6: Status 404 returned error can't find the container with id d15c2ca2ead1168647dbed407f696cf326c06974dcc7bc10ea47ae18afc314a6 Sep 17 01:39:12.346950 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.346890 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d" podStartSLOduration=2.346875775 podStartE2EDuration="2.346875775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:39:12.344397343 +0000 UTC m=+1017.579544532" watchObservedRunningTime="2026-09-17 01:39:12.346875775 +0000 UTC m=+1017.582022963" Sep 17 01:39:12.362818 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.362760 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt" podStartSLOduration=2.362739294 podStartE2EDuration="2.362739294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:39:12.360035393 +0000 UTC m=+1017.595182592" watchObservedRunningTime="2026-09-17 01:39:12.362739294 +0000 UTC m=+1017.597886485" Sep 17 01:39:12.472497 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.472471 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10056-68e69d" Sep 17 01:39:12.534455 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.534350 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ls492\" (UniqueName: \"kubernetes.io/projected/1e6de695-0711-43a7-8801-281bcc4bde1b-kube-api-access-ls492\") pod \"1e6de695-0711-43a7-8801-281bcc4bde1b\" (UID: \"1e6de695-0711-43a7-8801-281bcc4bde1b\") " Sep 17 01:39:12.537230 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.537198 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e6de695-0711-43a7-8801-281bcc4bde1b-kube-api-access-ls492" (OuterVolumeSpecName: "kube-api-access-ls492") pod "1e6de695-0711-43a7-8801-281bcc4bde1b" (UID: "1e6de695-0711-43a7-8801-281bcc4bde1b"). InnerVolumeSpecName "kube-api-access-ls492". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:39:12.542440 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.542410 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7"] Sep 17 01:39:12.546831 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.546811 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7" Sep 17 01:39:12.548926 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.548901 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d8d689-dockercfg-5qrb4\"" Sep 17 01:39:12.549064 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.548901 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d8d689\"" Sep 17 01:39:12.559816 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.557387 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7"] Sep 17 01:39:12.635199 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.635166 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plbvk\" (UniqueName: \"kubernetes.io/projected/34504414-1aa5-4cd8-b033-276a0d189d75-kube-api-access-plbvk\") pod \"payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7\" (UID: \"34504414-1aa5-4cd8-b033-276a0d189d75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7" Sep 17 01:39:12.635378 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.635230 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34504414-1aa5-4cd8-b033-276a0d189d75-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7\" (UID: \"34504414-1aa5-4cd8-b033-276a0d189d75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7" Sep 17 01:39:12.635378 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.635323 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls492\" (UniqueName: \"kubernetes.io/projected/1e6de695-0711-43a7-8801-281bcc4bde1b-kube-api-access-ls492\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:39:12.736077 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.736027 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plbvk\" (UniqueName: \"kubernetes.io/projected/34504414-1aa5-4cd8-b033-276a0d189d75-kube-api-access-plbvk\") pod \"payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7\" (UID: \"34504414-1aa5-4cd8-b033-276a0d189d75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7" Sep 17 01:39:12.736258 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.736127 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34504414-1aa5-4cd8-b033-276a0d189d75-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7\" (UID: \"34504414-1aa5-4cd8-b033-276a0d189d75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7" Sep 17 01:39:12.737027 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.737004 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34504414-1aa5-4cd8-b033-276a0d189d75-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7\" (UID: \"34504414-1aa5-4cd8-b033-276a0d189d75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7" Sep 17 01:39:12.743752 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.743727 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plbvk\" (UniqueName: \"kubernetes.io/projected/34504414-1aa5-4cd8-b033-276a0d189d75-kube-api-access-plbvk\") pod \"payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7\" (UID: \"34504414-1aa5-4cd8-b033-276a0d189d75\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7" Sep 17 01:39:12.862980 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:12.862953 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7" Sep 17 01:39:13.139330 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:13.139295 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7"] Sep 17 01:39:13.139878 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:39:13.139849 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34504414_1aa5_4cd8_b033_276a0d189d75.slice/crio-b6bf1e41d7caa53950db45f4fe11493213ef9c8bee9f78b54f4a0863d995aa8d WatchSource:0}: Error finding container b6bf1e41d7caa53950db45f4fe11493213ef9c8bee9f78b54f4a0863d995aa8d: Status 404 returned error can't find the container with id b6bf1e41d7caa53950db45f4fe11493213ef9c8bee9f78b54f4a0863d995aa8d Sep 17 01:39:13.213319 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:13.213287 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e6de695-0711-43a7-8801-281bcc4bde1b" path="/var/lib/kubelet/pods/1e6de695-0711-43a7-8801-281bcc4bde1b/volumes" Sep 17 01:39:13.334330 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:13.334284 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7" event={"ID":"34504414-1aa5-4cd8-b033-276a0d189d75","Type":"ContainerStarted","Data":"24c3f164b5789c385995ebea93f35f8033a8e708b4281a98d34b0c51be40273d"} Sep 17 01:39:13.334790 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:13.334368 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7" event={"ID":"34504414-1aa5-4cd8-b033-276a0d189d75","Type":"ContainerStarted","Data":"b6bf1e41d7caa53950db45f4fe11493213ef9c8bee9f78b54f4a0863d995aa8d"} Sep 17 01:39:13.334790 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:13.334402 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7" Sep 17 01:39:13.336184 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:13.336146 2577 generic.go:356] "Generic (PLEG): container finished" podID="1e6de695-0711-43a7-8801-281bcc4bde1b" containerID="43856958f25f05c4e2d710a7faf6b0a36d702b1c11abae892baaa2d7beb5e719" exitCode=137 Sep 17 01:39:13.336338 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:13.336186 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10056-68e69d" Sep 17 01:39:13.336338 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:13.336198 2577 scope.go:117] "RemoveContainer" containerID="43856958f25f05c4e2d710a7faf6b0a36d702b1c11abae892baaa2d7beb5e719" Sep 17 01:39:13.339675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:13.339651 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr" event={"ID":"cf6fbf20-070e-4f55-84e1-ff4021d0570a","Type":"ContainerStarted","Data":"5699bfaa435846010a73997d67b0dc08b0004905a71a7dc35ccd352f497b881e"} Sep 17 01:39:13.339835 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:13.339817 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr" event={"ID":"cf6fbf20-070e-4f55-84e1-ff4021d0570a","Type":"ContainerStarted","Data":"d15c2ca2ead1168647dbed407f696cf326c06974dcc7bc10ea47ae18afc314a6"} Sep 17 01:39:13.340064 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:13.340039 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr" Sep 17 01:39:13.347554 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:13.347527 2577 scope.go:117] "RemoveContainer" containerID="43856958f25f05c4e2d710a7faf6b0a36d702b1c11abae892baaa2d7beb5e719" Sep 17 01:39:13.347820 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:39:13.347802 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43856958f25f05c4e2d710a7faf6b0a36d702b1c11abae892baaa2d7beb5e719\": container with ID starting with 43856958f25f05c4e2d710a7faf6b0a36d702b1c11abae892baaa2d7beb5e719 not found: ID does not exist" containerID="43856958f25f05c4e2d710a7faf6b0a36d702b1c11abae892baaa2d7beb5e719" Sep 17 01:39:13.347900 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:13.347832 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43856958f25f05c4e2d710a7faf6b0a36d702b1c11abae892baaa2d7beb5e719"} err="failed to get container status \"43856958f25f05c4e2d710a7faf6b0a36d702b1c11abae892baaa2d7beb5e719\": rpc error: code = NotFound desc = could not find container \"43856958f25f05c4e2d710a7faf6b0a36d702b1c11abae892baaa2d7beb5e719\": container with ID starting with 43856958f25f05c4e2d710a7faf6b0a36d702b1c11abae892baaa2d7beb5e719 not found: ID does not exist" Sep 17 01:39:13.351462 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:13.351419 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7" podStartSLOduration=1.351407088 podStartE2EDuration="1.351407088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:39:13.350202685 +0000 UTC m=+1018.585349876" watchObservedRunningTime="2026-09-17 01:39:13.351407088 +0000 UTC m=+1018.586554275" Sep 17 01:39:13.366708 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:13.366665 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr" podStartSLOduration=2.366652018 podStartE2EDuration="2.366652018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:39:13.365168839 +0000 UTC m=+1018.600316035" watchObservedRunningTime="2026-09-17 01:39:13.366652018 +0000 UTC m=+1018.601799481" Sep 17 01:39:14.510275 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:14.510253 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10056-b90d98" Sep 17 01:39:14.555576 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:14.555535 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj4rb\" (UniqueName: \"kubernetes.io/projected/b4a10093-3819-4482-8514-f1e8851ed019-kube-api-access-tj4rb\") pod \"b4a10093-3819-4482-8514-f1e8851ed019\" (UID: \"b4a10093-3819-4482-8514-f1e8851ed019\") " Sep 17 01:39:14.558531 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:14.558455 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a10093-3819-4482-8514-f1e8851ed019-kube-api-access-tj4rb" (OuterVolumeSpecName: "kube-api-access-tj4rb") pod "b4a10093-3819-4482-8514-f1e8851ed019" (UID: "b4a10093-3819-4482-8514-f1e8851ed019"). InnerVolumeSpecName "kube-api-access-tj4rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:39:14.657327 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:14.657298 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj4rb\" (UniqueName: \"kubernetes.io/projected/b4a10093-3819-4482-8514-f1e8851ed019-kube-api-access-tj4rb\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:39:15.213251 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:15.213218 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4a10093-3819-4482-8514-f1e8851ed019" path="/var/lib/kubelet/pods/b4a10093-3819-4482-8514-f1e8851ed019/volumes" Sep 17 01:39:15.355235 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:15.355184 2577 generic.go:356] "Generic (PLEG): container finished" podID="b4a10093-3819-4482-8514-f1e8851ed019" containerID="bd1ecacadf269d25f49d27f89a3f04a20fae26d53ab1c4f4fe839bbb010f15bd" exitCode=137 Sep 17 01:39:15.355413 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:15.355250 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10056-b90d98" Sep 17 01:39:15.355413 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:15.355261 2577 scope.go:117] "RemoveContainer" containerID="bd1ecacadf269d25f49d27f89a3f04a20fae26d53ab1c4f4fe839bbb010f15bd" Sep 17 01:39:15.364723 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:15.364704 2577 scope.go:117] "RemoveContainer" containerID="bd1ecacadf269d25f49d27f89a3f04a20fae26d53ab1c4f4fe839bbb010f15bd" Sep 17 01:39:15.365035 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:39:15.365004 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd1ecacadf269d25f49d27f89a3f04a20fae26d53ab1c4f4fe839bbb010f15bd\": container with ID starting with bd1ecacadf269d25f49d27f89a3f04a20fae26d53ab1c4f4fe839bbb010f15bd not found: ID does not exist" containerID="bd1ecacadf269d25f49d27f89a3f04a20fae26d53ab1c4f4fe839bbb010f15bd" Sep 17 01:39:15.365114 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:15.365045 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd1ecacadf269d25f49d27f89a3f04a20fae26d53ab1c4f4fe839bbb010f15bd"} err="failed to get container status \"bd1ecacadf269d25f49d27f89a3f04a20fae26d53ab1c4f4fe839bbb010f15bd\": rpc error: code = NotFound desc = could not find container \"bd1ecacadf269d25f49d27f89a3f04a20fae26d53ab1c4f4fe839bbb010f15bd\": container with ID starting with bd1ecacadf269d25f49d27f89a3f04a20fae26d53ab1c4f4fe839bbb010f15bd not found: ID does not exist" Sep 17 01:39:17.205665 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:17.205627 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:39:18.854601 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:18.854539 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm"] Sep 17 01:39:18.860897 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:18.860863 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" Sep 17 01:39:18.863477 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:18.863297 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s7lw7\"" Sep 17 01:39:18.867002 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:18.864302 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4889635f-serving-cert\"" Sep 17 01:39:18.867002 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:18.864515 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 01:39:18.867002 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:18.864701 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 01:39:18.867002 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:18.864952 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 01:39:18.872746 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:18.872721 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm"] Sep 17 01:39:18.899390 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:18.899357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a75d03db-d1e3-4c82-93cc-fc35e617f765-metrics-certs\") pod \"maas-api-e2e-ait-4889635f-68d796647-wtxgm\" (UID: \"a75d03db-d1e3-4c82-93cc-fc35e617f765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" Sep 17 01:39:18.899542 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:18.899399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm8ff\" (UniqueName: \"kubernetes.io/projected/a75d03db-d1e3-4c82-93cc-fc35e617f765-kube-api-access-qm8ff\") pod \"maas-api-e2e-ait-4889635f-68d796647-wtxgm\" (UID: \"a75d03db-d1e3-4c82-93cc-fc35e617f765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" Sep 17 01:39:18.899542 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:18.899448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75d03db-d1e3-4c82-93cc-fc35e617f765-maas-api-tls\") pod \"maas-api-e2e-ait-4889635f-68d796647-wtxgm\" (UID: \"a75d03db-d1e3-4c82-93cc-fc35e617f765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" Sep 17 01:39:18.899663 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:18.899539 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75d03db-d1e3-4c82-93cc-fc35e617f765-openshift-service-ca\") pod \"maas-api-e2e-ait-4889635f-68d796647-wtxgm\" (UID: \"a75d03db-d1e3-4c82-93cc-fc35e617f765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" Sep 17 01:39:19.000421 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.000379 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a75d03db-d1e3-4c82-93cc-fc35e617f765-metrics-certs\") pod \"maas-api-e2e-ait-4889635f-68d796647-wtxgm\" (UID: \"a75d03db-d1e3-4c82-93cc-fc35e617f765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" Sep 17 01:39:19.000421 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.000424 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm8ff\" (UniqueName: \"kubernetes.io/projected/a75d03db-d1e3-4c82-93cc-fc35e617f765-kube-api-access-qm8ff\") pod \"maas-api-e2e-ait-4889635f-68d796647-wtxgm\" (UID: \"a75d03db-d1e3-4c82-93cc-fc35e617f765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" Sep 17 01:39:19.000675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.000461 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75d03db-d1e3-4c82-93cc-fc35e617f765-maas-api-tls\") pod \"maas-api-e2e-ait-4889635f-68d796647-wtxgm\" (UID: \"a75d03db-d1e3-4c82-93cc-fc35e617f765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" Sep 17 01:39:19.000675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.000483 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75d03db-d1e3-4c82-93cc-fc35e617f765-openshift-service-ca\") pod \"maas-api-e2e-ait-4889635f-68d796647-wtxgm\" (UID: \"a75d03db-d1e3-4c82-93cc-fc35e617f765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" Sep 17 01:39:19.000790 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:39:19.000762 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-serving-cert: secret "maas-api-e2e-ait-4889635f-serving-cert" not found Sep 17 01:39:19.000847 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:39:19.000836 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a75d03db-d1e3-4c82-93cc-fc35e617f765-maas-api-tls podName:a75d03db-d1e3-4c82-93cc-fc35e617f765 nodeName:}" failed. No retries permitted until 2026-09-17 01:39:19.500822883 +0000 UTC m=+1024.735970049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a75d03db-d1e3-4c82-93cc-fc35e617f765-maas-api-tls") pod "maas-api-e2e-ait-4889635f-68d796647-wtxgm" (UID: "a75d03db-d1e3-4c82-93cc-fc35e617f765") : secret "maas-api-e2e-ait-4889635f-serving-cert" not found Sep 17 01:39:19.001137 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.001116 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75d03db-d1e3-4c82-93cc-fc35e617f765-openshift-service-ca\") pod \"maas-api-e2e-ait-4889635f-68d796647-wtxgm\" (UID: \"a75d03db-d1e3-4c82-93cc-fc35e617f765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" Sep 17 01:39:19.002853 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.002829 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a75d03db-d1e3-4c82-93cc-fc35e617f765-metrics-certs\") pod \"maas-api-e2e-ait-4889635f-68d796647-wtxgm\" (UID: \"a75d03db-d1e3-4c82-93cc-fc35e617f765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" Sep 17 01:39:19.009110 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.009089 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm8ff\" (UniqueName: \"kubernetes.io/projected/a75d03db-d1e3-4c82-93cc-fc35e617f765-kube-api-access-qm8ff\") pod \"maas-api-e2e-ait-4889635f-68d796647-wtxgm\" (UID: \"a75d03db-d1e3-4c82-93cc-fc35e617f765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" Sep 17 01:39:19.504704 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.504667 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75d03db-d1e3-4c82-93cc-fc35e617f765-maas-api-tls\") pod \"maas-api-e2e-ait-4889635f-68d796647-wtxgm\" (UID: \"a75d03db-d1e3-4c82-93cc-fc35e617f765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" Sep 17 01:39:19.507991 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.507959 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75d03db-d1e3-4c82-93cc-fc35e617f765-maas-api-tls\") pod \"maas-api-e2e-ait-4889635f-68d796647-wtxgm\" (UID: \"a75d03db-d1e3-4c82-93cc-fc35e617f765\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" Sep 17 01:39:19.567331 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.564985 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb"] Sep 17 01:39:19.577120 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.576214 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb" Sep 17 01:39:19.580252 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.580228 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4889635f-dockercfg-7xbht\"" Sep 17 01:39:19.580746 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.580728 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4889635f\"" Sep 17 01:39:19.581898 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.581872 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb"] Sep 17 01:39:19.605780 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.605750 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfbvr\" (UniqueName: \"kubernetes.io/projected/26ca8b47-900b-45c9-846c-dc794bcb707b-kube-api-access-sfbvr\") pod \"payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb\" (UID: \"26ca8b47-900b-45c9-846c-dc794bcb707b\") " pod="openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb" Sep 17 01:39:19.605903 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.605831 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ca8b47-900b-45c9-846c-dc794bcb707b-plugins-config-volume\") pod \"payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb\" (UID: \"26ca8b47-900b-45c9-846c-dc794bcb707b\") " pod="openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb" Sep 17 01:39:19.706605 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.706567 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ca8b47-900b-45c9-846c-dc794bcb707b-plugins-config-volume\") pod \"payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb\" (UID: \"26ca8b47-900b-45c9-846c-dc794bcb707b\") " pod="openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb" Sep 17 01:39:19.706776 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.706697 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfbvr\" (UniqueName: \"kubernetes.io/projected/26ca8b47-900b-45c9-846c-dc794bcb707b-kube-api-access-sfbvr\") pod \"payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb\" (UID: \"26ca8b47-900b-45c9-846c-dc794bcb707b\") " pod="openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb" Sep 17 01:39:19.707410 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.707337 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ca8b47-900b-45c9-846c-dc794bcb707b-plugins-config-volume\") pod \"payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb\" (UID: \"26ca8b47-900b-45c9-846c-dc794bcb707b\") " pod="openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb" Sep 17 01:39:19.713897 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.713874 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfbvr\" (UniqueName: \"kubernetes.io/projected/26ca8b47-900b-45c9-846c-dc794bcb707b-kube-api-access-sfbvr\") pod \"payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb\" (UID: \"26ca8b47-900b-45c9-846c-dc794bcb707b\") " pod="openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb" Sep 17 01:39:19.784166 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.784060 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" Sep 17 01:39:19.891594 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:19.891563 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb" Sep 17 01:39:20.109105 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:20.109080 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm"] Sep 17 01:39:20.110658 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:39:20.110633 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75d03db_d1e3_4c82_93cc_fc35e617f765.slice/crio-f00005a75f5d8b0e6a6ba950cdef0fe1de1e9039b566dcd812ab22d09696a28c WatchSource:0}: Error finding container f00005a75f5d8b0e6a6ba950cdef0fe1de1e9039b566dcd812ab22d09696a28c: Status 404 returned error can't find the container with id f00005a75f5d8b0e6a6ba950cdef0fe1de1e9039b566dcd812ab22d09696a28c Sep 17 01:39:20.233111 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:20.233052 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb"] Sep 17 01:39:20.400354 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:20.400319 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" event={"ID":"a75d03db-d1e3-4c82-93cc-fc35e617f765","Type":"ContainerStarted","Data":"f00005a75f5d8b0e6a6ba950cdef0fe1de1e9039b566dcd812ab22d09696a28c"} Sep 17 01:39:20.402019 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:20.401991 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb" event={"ID":"26ca8b47-900b-45c9-846c-dc794bcb707b","Type":"ContainerStarted","Data":"190d38f32278f93c6e0150612e118d195c742a5dd208fa6ac08e19367c613497"} Sep 17 01:39:20.402129 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:20.402046 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb" event={"ID":"26ca8b47-900b-45c9-846c-dc794bcb707b","Type":"ContainerStarted","Data":"3a4f2874d9e034787b281ee5f6cd1716ae3123299b19c4f6d89c060de1ad2269"} Sep 17 01:39:20.402194 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:20.402167 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb" Sep 17 01:39:23.322187 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:23.322087 2577 patch_prober.go:28] interesting pod/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 17 01:39:23.322187 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:23.322161 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" podUID="fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 17 01:39:23.342067 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:23.342035 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d" Sep 17 01:39:23.342261 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:23.342239 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt" Sep 17 01:39:23.357121 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:23.357074 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb" podStartSLOduration=4.357057909 podStartE2EDuration="4.357057909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:39:20.424999476 +0000 UTC m=+1025.660146673" watchObservedRunningTime="2026-09-17 01:39:23.357057909 +0000 UTC m=+1028.592205099" Sep 17 01:39:23.429681 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:23.429638 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" event={"ID":"a75d03db-d1e3-4c82-93cc-fc35e617f765","Type":"ContainerStarted","Data":"bba6d5144405b76d14a41addd9208fa0120d371de4c15b6c6267a8a50dd7f13e"} Sep 17 01:39:23.429893 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:23.429715 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" Sep 17 01:39:23.447586 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:23.447530 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" podStartSLOduration=3.172541602 podStartE2EDuration="5.447512505s" totalImagesPullingTime="2.274970903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:39:23.445184871 +0000 UTC m=+1028.680332060" watchObservedRunningTime="2026-09-17 01:39:23.447512505 +0000 UTC m=+1028.682659700" Sep 17 01:39:24.349764 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:24.349731 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7" Sep 17 01:39:25.211616 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:25.211578 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:39:25.358329 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:25.358295 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr" Sep 17 01:39:25.447347 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:25.447259 2577 generic.go:356] "Generic (PLEG): container finished" podID="fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" containerID="5f07311e4c731682d82c04d67c9ced9ece88b36003bf62c37f0b98cd0c39f5c1" exitCode=0 Sep 17 01:39:25.447347 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:25.447301 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" event={"ID":"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1","Type":"ContainerDied","Data":"5f07311e4c731682d82c04d67c9ced9ece88b36003bf62c37f0b98cd0c39f5c1"} Sep 17 01:39:25.447794 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:25.447775 2577 scope.go:117] "RemoveContainer" containerID="5f07311e4c731682d82c04d67c9ced9ece88b36003bf62c37f0b98cd0c39f5c1" Sep 17 01:39:25.449159 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:25.449130 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:39:25.449253 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:25.449209 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:39:25.449253 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:25.449242 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:39:26.456562 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:26.456521 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" event={"ID":"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1","Type":"ContainerStarted","Data":"6804637bdbe70e3b30e2b2494789334eb256a729d459044523794cfddcd10167"} Sep 17 01:39:26.458106 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:26.458082 2577 generic.go:356] "Generic (PLEG): container finished" podID="010061bf-7ab5-4623-826a-545d286b26e9" containerID="689714ffb0a4e2337dddc5a3d7545aaee97ec94055b54958534875ae58f9c27b" exitCode=0 Sep 17 01:39:26.458222 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:26.458126 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" event={"ID":"010061bf-7ab5-4623-826a-545d286b26e9","Type":"ContainerDied","Data":"689714ffb0a4e2337dddc5a3d7545aaee97ec94055b54958534875ae58f9c27b"} Sep 17 01:39:26.458503 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:26.458484 2577 scope.go:117] "RemoveContainer" containerID="689714ffb0a4e2337dddc5a3d7545aaee97ec94055b54958534875ae58f9c27b" Sep 17 01:39:26.459581 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:26.459544 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:39:26.459672 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:26.459633 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:39:26.459742 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:26.459687 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:39:26.926105 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:26.926070 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:26.926540 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:26.926478 2577 patch_prober.go:28] interesting pod/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 17 01:39:26.926672 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:26.926558 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" podUID="fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 17 01:39:27.466812 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:27.466769 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" event={"ID":"010061bf-7ab5-4623-826a-545d286b26e9","Type":"ContainerStarted","Data":"3b10de500dc96d811b5637f6dddb1e2183d1c584ccc0055b373bac74a8cf06b5"} Sep 17 01:39:27.926916 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:27.926866 2577 patch_prober.go:28] interesting pod/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 17 01:39:27.927148 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:27.926960 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" podUID="fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 17 01:39:28.129387 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:28.129339 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:28.129387 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:28.129362 2577 patch_prober.go:28] interesting pod/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Sep 17 01:39:28.129660 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:28.129420 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" podUID="010061bf-7ab5-4623-826a-545d286b26e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Sep 17 01:39:28.927091 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:28.927051 2577 patch_prober.go:28] interesting pod/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 17 01:39:28.927680 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:28.927645 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" podUID="fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 17 01:39:29.129169 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:29.129129 2577 patch_prober.go:28] interesting pod/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Sep 17 01:39:29.129464 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:29.129434 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" podUID="010061bf-7ab5-4623-826a-545d286b26e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Sep 17 01:39:29.454393 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:29.454352 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" Sep 17 01:39:29.926351 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:29.926308 2577 patch_prober.go:28] interesting pod/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 17 01:39:29.926558 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:29.926375 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" podUID="fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 17 01:39:30.129768 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:30.129727 2577 patch_prober.go:28] interesting pod/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Sep 17 01:39:30.130416 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:30.130380 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" podUID="010061bf-7ab5-4623-826a-545d286b26e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Sep 17 01:39:30.926765 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:30.926724 2577 patch_prober.go:28] interesting pod/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 17 01:39:30.927116 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:30.927079 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" podUID="fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 17 01:39:31.129806 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:31.129744 2577 patch_prober.go:28] interesting pod/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Sep 17 01:39:31.130452 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:31.130394 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" podUID="010061bf-7ab5-4623-826a-545d286b26e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Sep 17 01:39:31.411582 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:31.411553 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb" Sep 17 01:39:31.942371 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:31.942314 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:31.943588 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:31.943527 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:31.948893 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:31.948857 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:39:32.130953 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:32.129894 2577 patch_prober.go:28] interesting pod/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Sep 17 01:39:32.130953 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:32.129985 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" podUID="010061bf-7ab5-4623-826a-545d286b26e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Sep 17 01:39:33.129420 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:33.129383 2577 patch_prober.go:28] interesting pod/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Sep 17 01:39:33.130352 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:33.130317 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" podUID="010061bf-7ab5-4623-826a-545d286b26e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Sep 17 01:39:34.133842 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:34.133810 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:34.134242 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:34.134218 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:34.134812 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:34.134795 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:39:58.487693 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.487654 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm"] Sep 17 01:39:58.488361 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.487906 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" podUID="a75d03db-d1e3-4c82-93cc-fc35e617f765" containerName="maas-api" containerID="cri-o://bba6d5144405b76d14a41addd9208fa0120d371de4c15b6c6267a8a50dd7f13e" gracePeriod=30 Sep 17 01:39:58.541128 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.541083 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb"] Sep 17 01:39:58.541358 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.541331 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb" podUID="26ca8b47-900b-45c9-846c-dc794bcb707b" containerName="payload-processing" containerID="cri-o://190d38f32278f93c6e0150612e118d195c742a5dd208fa6ac08e19367c613497" gracePeriod=30 Sep 17 01:39:58.673788 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.673741 2577 generic.go:356] "Generic (PLEG): container finished" podID="26ca8b47-900b-45c9-846c-dc794bcb707b" containerID="190d38f32278f93c6e0150612e118d195c742a5dd208fa6ac08e19367c613497" exitCode=0 Sep 17 01:39:58.673978 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.673829 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb" event={"ID":"26ca8b47-900b-45c9-846c-dc794bcb707b","Type":"ContainerDied","Data":"190d38f32278f93c6e0150612e118d195c742a5dd208fa6ac08e19367c613497"} Sep 17 01:39:58.675847 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.675807 2577 generic.go:356] "Generic (PLEG): container finished" podID="a75d03db-d1e3-4c82-93cc-fc35e617f765" containerID="bba6d5144405b76d14a41addd9208fa0120d371de4c15b6c6267a8a50dd7f13e" exitCode=0 Sep 17 01:39:58.675997 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.675916 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" event={"ID":"a75d03db-d1e3-4c82-93cc-fc35e617f765","Type":"ContainerDied","Data":"bba6d5144405b76d14a41addd9208fa0120d371de4c15b6c6267a8a50dd7f13e"} Sep 17 01:39:58.803514 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.803485 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" Sep 17 01:39:58.844603 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.844574 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb" Sep 17 01:39:58.930027 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.929996 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfbvr\" (UniqueName: \"kubernetes.io/projected/26ca8b47-900b-45c9-846c-dc794bcb707b-kube-api-access-sfbvr\") pod \"26ca8b47-900b-45c9-846c-dc794bcb707b\" (UID: \"26ca8b47-900b-45c9-846c-dc794bcb707b\") " Sep 17 01:39:58.930218 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.930073 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75d03db-d1e3-4c82-93cc-fc35e617f765-maas-api-tls\") pod \"a75d03db-d1e3-4c82-93cc-fc35e617f765\" (UID: \"a75d03db-d1e3-4c82-93cc-fc35e617f765\") " Sep 17 01:39:58.930218 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.930120 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm8ff\" (UniqueName: \"kubernetes.io/projected/a75d03db-d1e3-4c82-93cc-fc35e617f765-kube-api-access-qm8ff\") pod \"a75d03db-d1e3-4c82-93cc-fc35e617f765\" (UID: \"a75d03db-d1e3-4c82-93cc-fc35e617f765\") " Sep 17 01:39:58.930218 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.930154 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a75d03db-d1e3-4c82-93cc-fc35e617f765-metrics-certs\") pod \"a75d03db-d1e3-4c82-93cc-fc35e617f765\" (UID: \"a75d03db-d1e3-4c82-93cc-fc35e617f765\") " Sep 17 01:39:58.930218 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.930193 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ca8b47-900b-45c9-846c-dc794bcb707b-plugins-config-volume\") pod \"26ca8b47-900b-45c9-846c-dc794bcb707b\" (UID: \"26ca8b47-900b-45c9-846c-dc794bcb707b\") " Sep 17 01:39:58.930218 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.930217 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75d03db-d1e3-4c82-93cc-fc35e617f765-openshift-service-ca\") pod \"a75d03db-d1e3-4c82-93cc-fc35e617f765\" (UID: \"a75d03db-d1e3-4c82-93cc-fc35e617f765\") " Sep 17 01:39:58.930614 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.930578 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ca8b47-900b-45c9-846c-dc794bcb707b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26ca8b47-900b-45c9-846c-dc794bcb707b" (UID: "26ca8b47-900b-45c9-846c-dc794bcb707b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:39:58.930889 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.930858 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75d03db-d1e3-4c82-93cc-fc35e617f765-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a75d03db-d1e3-4c82-93cc-fc35e617f765" (UID: "a75d03db-d1e3-4c82-93cc-fc35e617f765"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:39:58.932390 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.932362 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75d03db-d1e3-4c82-93cc-fc35e617f765-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a75d03db-d1e3-4c82-93cc-fc35e617f765" (UID: "a75d03db-d1e3-4c82-93cc-fc35e617f765"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:39:58.932674 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.932649 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75d03db-d1e3-4c82-93cc-fc35e617f765-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a75d03db-d1e3-4c82-93cc-fc35e617f765" (UID: "a75d03db-d1e3-4c82-93cc-fc35e617f765"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:39:58.932755 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.932669 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75d03db-d1e3-4c82-93cc-fc35e617f765-kube-api-access-qm8ff" (OuterVolumeSpecName: "kube-api-access-qm8ff") pod "a75d03db-d1e3-4c82-93cc-fc35e617f765" (UID: "a75d03db-d1e3-4c82-93cc-fc35e617f765"). InnerVolumeSpecName "kube-api-access-qm8ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:39:58.932755 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:58.932686 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ca8b47-900b-45c9-846c-dc794bcb707b-kube-api-access-sfbvr" (OuterVolumeSpecName: "kube-api-access-sfbvr") pod "26ca8b47-900b-45c9-846c-dc794bcb707b" (UID: "26ca8b47-900b-45c9-846c-dc794bcb707b"). InnerVolumeSpecName "kube-api-access-sfbvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:39:59.031889 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.031784 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26ca8b47-900b-45c9-846c-dc794bcb707b-plugins-config-volume\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:39:59.031889 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.031816 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75d03db-d1e3-4c82-93cc-fc35e617f765-openshift-service-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:39:59.031889 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.031826 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfbvr\" (UniqueName: \"kubernetes.io/projected/26ca8b47-900b-45c9-846c-dc794bcb707b-kube-api-access-sfbvr\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:39:59.031889 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.031835 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75d03db-d1e3-4c82-93cc-fc35e617f765-maas-api-tls\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:39:59.031889 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.031844 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm8ff\" (UniqueName: \"kubernetes.io/projected/a75d03db-d1e3-4c82-93cc-fc35e617f765-kube-api-access-qm8ff\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:39:59.031889 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.031855 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a75d03db-d1e3-4c82-93cc-fc35e617f765-metrics-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:39:59.683787 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.683758 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb" Sep 17 01:39:59.684285 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.683806 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb" event={"ID":"26ca8b47-900b-45c9-846c-dc794bcb707b","Type":"ContainerDied","Data":"3a4f2874d9e034787b281ee5f6cd1716ae3123299b19c4f6d89c060de1ad2269"} Sep 17 01:39:59.684285 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.683869 2577 scope.go:117] "RemoveContainer" containerID="190d38f32278f93c6e0150612e118d195c742a5dd208fa6ac08e19367c613497" Sep 17 01:39:59.685394 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.685375 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" Sep 17 01:39:59.685486 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.685374 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm" event={"ID":"a75d03db-d1e3-4c82-93cc-fc35e617f765","Type":"ContainerDied","Data":"f00005a75f5d8b0e6a6ba950cdef0fe1de1e9039b566dcd812ab22d09696a28c"} Sep 17 01:39:59.694127 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.694101 2577 scope.go:117] "RemoveContainer" containerID="bba6d5144405b76d14a41addd9208fa0120d371de4c15b6c6267a8a50dd7f13e" Sep 17 01:39:59.705574 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.705549 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb"] Sep 17 01:39:59.709908 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.709887 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4889635f-6f8978984f-v2ngb"] Sep 17 01:39:59.725075 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.725043 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm"] Sep 17 01:39:59.736907 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.736881 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4889635f-68d796647-wtxgm"] Sep 17 01:39:59.786254 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.786181 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h"] Sep 17 01:39:59.786696 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.786681 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75d03db-d1e3-4c82-93cc-fc35e617f765" containerName="maas-api" Sep 17 01:39:59.786747 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.786700 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75d03db-d1e3-4c82-93cc-fc35e617f765" containerName="maas-api" Sep 17 01:39:59.786747 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.786729 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ca8b47-900b-45c9-846c-dc794bcb707b" containerName="payload-processing" Sep 17 01:39:59.786747 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.786736 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ca8b47-900b-45c9-846c-dc794bcb707b" containerName="payload-processing" Sep 17 01:39:59.786841 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.786811 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a75d03db-d1e3-4c82-93cc-fc35e617f765" containerName="maas-api" Sep 17 01:39:59.786841 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.786824 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ca8b47-900b-45c9-846c-dc794bcb707b" containerName="payload-processing" Sep 17 01:39:59.791549 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.791528 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.793496 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.793475 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-c7f22faa-gw-openshift-default-dockercfg-49m26\"" Sep 17 01:39:59.803171 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.803146 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h"] Sep 17 01:39:59.840094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.840061 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-podinfo\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.840251 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.840140 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-data\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.840251 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.840203 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrtcx\" (UniqueName: \"kubernetes.io/projected/1e077cfa-09a7-4607-846a-4235eb1fb8c8-kube-api-access-mrtcx\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.840251 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.840230 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-token\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.840423 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.840258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-envoy\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.840423 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.840293 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-credential-socket\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.840423 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.840315 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istiod-ca-cert\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.840423 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.840382 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-workload-certs\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.840620 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.840423 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-workload-socket\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.941812 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.941772 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-data\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.942050 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.941865 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrtcx\" (UniqueName: \"kubernetes.io/projected/1e077cfa-09a7-4607-846a-4235eb1fb8c8-kube-api-access-mrtcx\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.942050 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.941906 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-token\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.942050 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.941967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-envoy\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.942050 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.941992 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-credential-socket\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.942050 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.942014 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istiod-ca-cert\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.942333 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.942068 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-workload-certs\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.942333 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.942119 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-workload-socket\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.942333 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.942144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-podinfo\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.942490 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.942394 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-credential-socket\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.942567 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.942486 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-workload-certs\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.942567 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.942508 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-workload-socket\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.942784 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.942754 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istiod-ca-cert\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.942918 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.942856 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-data\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.944319 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.944295 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-envoy\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.944681 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.944660 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-podinfo\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.949544 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.949519 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-token\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:39:59.950135 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:39:59.950117 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrtcx\" (UniqueName: \"kubernetes.io/projected/1e077cfa-09a7-4607-846a-4235eb1fb8c8-kube-api-access-mrtcx\") pod \"e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:40:00.103703 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:00.103674 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:40:00.384853 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:00.384821 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h"] Sep 17 01:40:00.385405 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:40:00.385381 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e077cfa_09a7_4607_846a_4235eb1fb8c8.slice/crio-a8a4308707bedc6f3f07bfbdbaa5adf9df9f07d27a4a4fe4962160800b5409e5 WatchSource:0}: Error finding container a8a4308707bedc6f3f07bfbdbaa5adf9df9f07d27a4a4fe4962160800b5409e5: Status 404 returned error can't find the container with id a8a4308707bedc6f3f07bfbdbaa5adf9df9f07d27a4a4fe4962160800b5409e5 Sep 17 01:40:00.387692 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:00.387663 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:40:00.387766 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:00.387722 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:40:00.387766 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:00.387749 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:40:00.693290 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:00.693180 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" event={"ID":"1e077cfa-09a7-4607-846a-4235eb1fb8c8","Type":"ContainerStarted","Data":"1f0753f004922ec185dddcf60ea2b850a694cef8c9bb5ece1499083949a55311"} Sep 17 01:40:00.693290 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:00.693249 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" event={"ID":"1e077cfa-09a7-4607-846a-4235eb1fb8c8","Type":"ContainerStarted","Data":"a8a4308707bedc6f3f07bfbdbaa5adf9df9f07d27a4a4fe4962160800b5409e5"} Sep 17 01:40:00.712980 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:00.712908 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" podStartSLOduration=1.712885747 podStartE2EDuration="1.712885747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:40:00.709140253 +0000 UTC m=+1065.944287442" watchObservedRunningTime="2026-09-17 01:40:00.712885747 +0000 UTC m=+1065.948032934" Sep 17 01:40:01.104410 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:01.104378 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:40:01.109595 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:01.109571 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:40:01.212406 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:01.212368 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26ca8b47-900b-45c9-846c-dc794bcb707b" path="/var/lib/kubelet/pods/26ca8b47-900b-45c9-846c-dc794bcb707b/volumes" Sep 17 01:40:01.212789 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:01.212774 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a75d03db-d1e3-4c82-93cc-fc35e617f765" path="/var/lib/kubelet/pods/a75d03db-d1e3-4c82-93cc-fc35e617f765/volumes" Sep 17 01:40:01.702299 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:01.702262 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:40:01.703728 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:01.703704 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:40:03.397232 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:03.397193 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl"] Sep 17 01:40:03.402217 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:03.402198 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" Sep 17 01:40:03.404253 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:03.404226 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-c7f22faa-serving-cert\"" Sep 17 01:40:03.404798 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:03.404772 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 01:40:03.405570 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:03.405384 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 01:40:03.405570 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:03.405469 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s7lw7\"" Sep 17 01:40:03.405570 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:03.405535 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 01:40:03.408222 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:03.408197 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl"] Sep 17 01:40:03.479603 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:03.479568 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-metrics-certs\") pod \"maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl\" (UID: \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" Sep 17 01:40:03.479769 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:03.479607 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-maas-api-tls\") pod \"maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl\" (UID: \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" Sep 17 01:40:03.479769 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:03.479711 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-openshift-service-ca\") pod \"maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl\" (UID: \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" Sep 17 01:40:03.479769 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:03.479752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkqhl\" (UniqueName: \"kubernetes.io/projected/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-kube-api-access-jkqhl\") pod \"maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl\" (UID: \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" Sep 17 01:40:03.581006 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:03.580965 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-metrics-certs\") pod \"maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl\" (UID: \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" Sep 17 01:40:03.581184 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:03.581020 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-maas-api-tls\") pod \"maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl\" (UID: \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" Sep 17 01:40:03.581184 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:03.581076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-openshift-service-ca\") pod \"maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl\" (UID: \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" Sep 17 01:40:03.581184 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:03.581111 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkqhl\" (UniqueName: \"kubernetes.io/projected/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-kube-api-access-jkqhl\") pod \"maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl\" (UID: \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" Sep 17 01:40:03.581184 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:40:03.581150 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-serving-cert: secret "maas-api-e2e-migrate-c7f22faa-serving-cert" not found Sep 17 01:40:03.581412 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:40:03.581231 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-maas-api-tls podName:c2617bbc-6a65-4eb2-b54c-6c8a8d749c08 nodeName:}" failed. No retries permitted until 2026-09-17 01:40:04.081209433 +0000 UTC m=+1069.316356604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-maas-api-tls") pod "maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" (UID: "c2617bbc-6a65-4eb2-b54c-6c8a8d749c08") : secret "maas-api-e2e-migrate-c7f22faa-serving-cert" not found Sep 17 01:40:03.581775 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:03.581756 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-openshift-service-ca\") pod \"maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl\" (UID: \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" Sep 17 01:40:03.583416 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:03.583394 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-metrics-certs\") pod \"maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl\" (UID: \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" Sep 17 01:40:03.588764 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:03.588739 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkqhl\" (UniqueName: \"kubernetes.io/projected/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-kube-api-access-jkqhl\") pod \"maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl\" (UID: \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" Sep 17 01:40:04.086210 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:04.086168 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-maas-api-tls\") pod \"maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl\" (UID: \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" Sep 17 01:40:04.088625 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:04.088599 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-maas-api-tls\") pod \"maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl\" (UID: \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" Sep 17 01:40:04.314548 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:04.314502 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" Sep 17 01:40:04.622430 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:04.622402 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl"] Sep 17 01:40:04.623810 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:40:04.623781 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2617bbc_6a65_4eb2_b54c_6c8a8d749c08.slice/crio-1d418cf8032a1bad6e0391ca7133581c51ec2478290a0f43fa432e9190a02103 WatchSource:0}: Error finding container 1d418cf8032a1bad6e0391ca7133581c51ec2478290a0f43fa432e9190a02103: Status 404 returned error can't find the container with id 1d418cf8032a1bad6e0391ca7133581c51ec2478290a0f43fa432e9190a02103 Sep 17 01:40:04.625488 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:04.625467 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 01:40:04.721824 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:04.721784 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" event={"ID":"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08","Type":"ContainerStarted","Data":"1d418cf8032a1bad6e0391ca7133581c51ec2478290a0f43fa432e9190a02103"} Sep 17 01:40:05.730456 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:05.730408 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" event={"ID":"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08","Type":"ContainerStarted","Data":"c1ca45865ffa1248349daa4a67616f364455fa6bd84399bb84cc4874ca53310d"} Sep 17 01:40:05.730988 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:05.730533 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" Sep 17 01:40:05.751363 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:05.751293 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" podStartSLOduration=2.439532838 podStartE2EDuration="2.751269999s" totalImagesPullingTime="311.737161ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:40:05.748652568 +0000 UTC m=+1070.983799756" watchObservedRunningTime="2026-09-17 01:40:05.751269999 +0000 UTC m=+1070.986417189" Sep 17 01:40:11.741647 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:11.741554 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" Sep 17 01:40:22.841290 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:22.841253 2577 generic.go:356] "Generic (PLEG): container finished" podID="1e077cfa-09a7-4607-846a-4235eb1fb8c8" containerID="1f0753f004922ec185dddcf60ea2b850a694cef8c9bb5ece1499083949a55311" exitCode=0 Sep 17 01:40:22.841753 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:22.841333 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" event={"ID":"1e077cfa-09a7-4607-846a-4235eb1fb8c8","Type":"ContainerDied","Data":"1f0753f004922ec185dddcf60ea2b850a694cef8c9bb5ece1499083949a55311"} Sep 17 01:40:22.841824 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:22.841770 2577 scope.go:117] "RemoveContainer" containerID="1f0753f004922ec185dddcf60ea2b850a694cef8c9bb5ece1499083949a55311" Sep 17 01:40:22.842915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:22.842886 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:40:22.843004 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:22.842983 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:40:22.843049 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:22.843023 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:40:23.855553 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:23.855501 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" event={"ID":"1e077cfa-09a7-4607-846a-4235eb1fb8c8","Type":"ContainerStarted","Data":"b5614ec154deabe53b59e056c59e6f95ab4d5a0a68476df69301a3b101475612"} Sep 17 01:40:24.104033 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:24.103996 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:40:24.104231 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:24.104206 2577 patch_prober.go:28] interesting pod/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 17 01:40:24.104287 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:24.104268 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" podUID="1e077cfa-09a7-4607-846a-4235eb1fb8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 17 01:40:25.104385 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:25.104343 2577 patch_prober.go:28] interesting pod/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 17 01:40:25.104841 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:25.104415 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" podUID="1e077cfa-09a7-4607-846a-4235eb1fb8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 17 01:40:26.104203 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:26.104162 2577 patch_prober.go:28] interesting pod/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 17 01:40:26.104411 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:26.104227 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" podUID="1e077cfa-09a7-4607-846a-4235eb1fb8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 17 01:40:27.104754 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:27.104717 2577 patch_prober.go:28] interesting pod/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 17 01:40:27.105232 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:27.104829 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" podUID="1e077cfa-09a7-4607-846a-4235eb1fb8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 17 01:40:28.104376 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:28.104323 2577 patch_prober.go:28] interesting pod/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 17 01:40:28.104560 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:28.104389 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" podUID="1e077cfa-09a7-4607-846a-4235eb1fb8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 17 01:40:29.213499 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:29.213467 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:40:29.214132 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:29.213873 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:40:29.214848 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:29.214826 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:40:34.088015 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.087976 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl"] Sep 17 01:40:34.088595 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.088205 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" podUID="c2617bbc-6a65-4eb2-b54c-6c8a8d749c08" containerName="maas-api" containerID="cri-o://c1ca45865ffa1248349daa4a67616f364455fa6bd84399bb84cc4874ca53310d" gracePeriod=30 Sep 17 01:40:34.207670 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.207635 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:40:34.352123 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.352098 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" Sep 17 01:40:34.404753 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.398034 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-maas-api-tls\") pod \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\" (UID: \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\") " Sep 17 01:40:34.404753 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.398137 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-metrics-certs\") pod \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\" (UID: \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\") " Sep 17 01:40:34.404753 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.398198 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkqhl\" (UniqueName: \"kubernetes.io/projected/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-kube-api-access-jkqhl\") pod \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\" (UID: \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\") " Sep 17 01:40:34.404753 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.398232 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-openshift-service-ca\") pod \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\" (UID: \"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08\") " Sep 17 01:40:34.404753 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.400953 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2617bbc-6a65-4eb2-b54c-6c8a8d749c08" (UID: "c2617bbc-6a65-4eb2-b54c-6c8a8d749c08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:40:34.408919 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.408869 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-kube-api-access-jkqhl" (OuterVolumeSpecName: "kube-api-access-jkqhl") pod "c2617bbc-6a65-4eb2-b54c-6c8a8d749c08" (UID: "c2617bbc-6a65-4eb2-b54c-6c8a8d749c08"). InnerVolumeSpecName "kube-api-access-jkqhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:40:34.409238 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.409205 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c2617bbc-6a65-4eb2-b54c-6c8a8d749c08" (UID: "c2617bbc-6a65-4eb2-b54c-6c8a8d749c08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:40:34.411971 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.411873 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2617bbc-6a65-4eb2-b54c-6c8a8d749c08" (UID: "c2617bbc-6a65-4eb2-b54c-6c8a8d749c08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:40:34.427461 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.427424 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h"] Sep 17 01:40:34.427690 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.427659 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" podUID="1e077cfa-09a7-4607-846a-4235eb1fb8c8" containerName="istio-proxy" containerID="cri-o://b5614ec154deabe53b59e056c59e6f95ab4d5a0a68476df69301a3b101475612" gracePeriod=30 Sep 17 01:40:34.499437 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.499406 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-metrics-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:40:34.499437 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.499435 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkqhl\" (UniqueName: \"kubernetes.io/projected/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-kube-api-access-jkqhl\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:40:34.499437 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.499444 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-openshift-service-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:40:34.499662 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.499453 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08-maas-api-tls\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:40:34.925780 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.925740 2577 generic.go:356] "Generic (PLEG): container finished" podID="c2617bbc-6a65-4eb2-b54c-6c8a8d749c08" containerID="c1ca45865ffa1248349daa4a67616f364455fa6bd84399bb84cc4874ca53310d" exitCode=0 Sep 17 01:40:34.926010 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.925807 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" Sep 17 01:40:34.926010 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.925819 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" event={"ID":"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08","Type":"ContainerDied","Data":"c1ca45865ffa1248349daa4a67616f364455fa6bd84399bb84cc4874ca53310d"} Sep 17 01:40:34.926010 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.925899 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl" event={"ID":"c2617bbc-6a65-4eb2-b54c-6c8a8d749c08","Type":"ContainerDied","Data":"1d418cf8032a1bad6e0391ca7133581c51ec2478290a0f43fa432e9190a02103"} Sep 17 01:40:34.926010 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.925921 2577 scope.go:117] "RemoveContainer" containerID="c1ca45865ffa1248349daa4a67616f364455fa6bd84399bb84cc4874ca53310d" Sep 17 01:40:34.936495 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.936479 2577 scope.go:117] "RemoveContainer" containerID="c1ca45865ffa1248349daa4a67616f364455fa6bd84399bb84cc4874ca53310d" Sep 17 01:40:34.936757 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:40:34.936736 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ca45865ffa1248349daa4a67616f364455fa6bd84399bb84cc4874ca53310d\": container with ID starting with c1ca45865ffa1248349daa4a67616f364455fa6bd84399bb84cc4874ca53310d not found: ID does not exist" containerID="c1ca45865ffa1248349daa4a67616f364455fa6bd84399bb84cc4874ca53310d" Sep 17 01:40:34.936832 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.936767 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ca45865ffa1248349daa4a67616f364455fa6bd84399bb84cc4874ca53310d"} err="failed to get container status \"c1ca45865ffa1248349daa4a67616f364455fa6bd84399bb84cc4874ca53310d\": rpc error: code = NotFound desc = could not find container \"c1ca45865ffa1248349daa4a67616f364455fa6bd84399bb84cc4874ca53310d\": container with ID starting with c1ca45865ffa1248349daa4a67616f364455fa6bd84399bb84cc4874ca53310d not found: ID does not exist" Sep 17 01:40:34.947369 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.947339 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl"] Sep 17 01:40:34.950048 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:34.950023 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c7f22faa-54cfd86ff5-t2dfl"] Sep 17 01:40:35.221004 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:35.218291 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2617bbc-6a65-4eb2-b54c-6c8a8d749c08" path="/var/lib/kubelet/pods/c2617bbc-6a65-4eb2-b54c-6c8a8d749c08/volumes" Sep 17 01:40:39.706198 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.706133 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:40:39.854906 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.854870 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-podinfo\") pod \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " Sep 17 01:40:39.855093 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.854953 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-workload-socket\") pod \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " Sep 17 01:40:39.855093 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.854999 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-envoy\") pod \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " Sep 17 01:40:39.855093 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.855027 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-token\") pod \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " Sep 17 01:40:39.855093 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.855075 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-credential-socket\") pod \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " Sep 17 01:40:39.855324 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.855101 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istiod-ca-cert\") pod \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " Sep 17 01:40:39.855324 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.855131 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrtcx\" (UniqueName: \"kubernetes.io/projected/1e077cfa-09a7-4607-846a-4235eb1fb8c8-kube-api-access-mrtcx\") pod \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " Sep 17 01:40:39.855324 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.855175 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-workload-certs\") pod \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " Sep 17 01:40:39.855324 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.855209 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-data\") pod \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\" (UID: \"1e077cfa-09a7-4607-846a-4235eb1fb8c8\") " Sep 17 01:40:39.855324 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.855254 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1e077cfa-09a7-4607-846a-4235eb1fb8c8" (UID: "1e077cfa-09a7-4607-846a-4235eb1fb8c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:40:39.855586 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.855477 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1e077cfa-09a7-4607-846a-4235eb1fb8c8" (UID: "1e077cfa-09a7-4607-846a-4235eb1fb8c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:40:39.855586 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.855492 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1e077cfa-09a7-4607-846a-4235eb1fb8c8" (UID: "1e077cfa-09a7-4607-846a-4235eb1fb8c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:40:39.855721 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.855701 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-credential-socket\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:40:39.855780 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.855728 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-workload-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:40:39.855780 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.855743 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-workload-socket\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:40:39.855780 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.855705 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-data" (OuterVolumeSpecName: "istio-data") pod "1e077cfa-09a7-4607-846a-4235eb1fb8c8" (UID: "1e077cfa-09a7-4607-846a-4235eb1fb8c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:40:39.855966 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.855892 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1e077cfa-09a7-4607-846a-4235eb1fb8c8" (UID: "1e077cfa-09a7-4607-846a-4235eb1fb8c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:40:39.857304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.857274 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-token" (OuterVolumeSpecName: "istio-token") pod "1e077cfa-09a7-4607-846a-4235eb1fb8c8" (UID: "1e077cfa-09a7-4607-846a-4235eb1fb8c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:40:39.857441 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.857306 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1e077cfa-09a7-4607-846a-4235eb1fb8c8" (UID: "1e077cfa-09a7-4607-846a-4235eb1fb8c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:40:39.857505 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.857493 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1e077cfa-09a7-4607-846a-4235eb1fb8c8" (UID: "1e077cfa-09a7-4607-846a-4235eb1fb8c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 01:40:39.857645 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.857623 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e077cfa-09a7-4607-846a-4235eb1fb8c8-kube-api-access-mrtcx" (OuterVolumeSpecName: "kube-api-access-mrtcx") pod "1e077cfa-09a7-4607-846a-4235eb1fb8c8" (UID: "1e077cfa-09a7-4607-846a-4235eb1fb8c8"). InnerVolumeSpecName "kube-api-access-mrtcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:40:39.956630 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.956548 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istiod-ca-cert\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:40:39.956630 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.956578 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrtcx\" (UniqueName: \"kubernetes.io/projected/1e077cfa-09a7-4607-846a-4235eb1fb8c8-kube-api-access-mrtcx\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:40:39.956630 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.956588 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-data\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:40:39.956630 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.956597 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-podinfo\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:40:39.956630 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.956607 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-envoy\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:40:39.956630 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.956615 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e077cfa-09a7-4607-846a-4235eb1fb8c8-istio-token\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:40:39.962738 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.962711 2577 generic.go:356] "Generic (PLEG): container finished" podID="1e077cfa-09a7-4607-846a-4235eb1fb8c8" containerID="b5614ec154deabe53b59e056c59e6f95ab4d5a0a68476df69301a3b101475612" exitCode=0 Sep 17 01:40:39.962866 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.962763 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" Sep 17 01:40:39.962866 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.962813 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" event={"ID":"1e077cfa-09a7-4607-846a-4235eb1fb8c8","Type":"ContainerDied","Data":"b5614ec154deabe53b59e056c59e6f95ab4d5a0a68476df69301a3b101475612"} Sep 17 01:40:39.962971 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.962882 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h" event={"ID":"1e077cfa-09a7-4607-846a-4235eb1fb8c8","Type":"ContainerDied","Data":"a8a4308707bedc6f3f07bfbdbaa5adf9df9f07d27a4a4fe4962160800b5409e5"} Sep 17 01:40:39.962971 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.962916 2577 scope.go:117] "RemoveContainer" containerID="b5614ec154deabe53b59e056c59e6f95ab4d5a0a68476df69301a3b101475612" Sep 17 01:40:39.980764 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.980734 2577 scope.go:117] "RemoveContainer" containerID="1f0753f004922ec185dddcf60ea2b850a694cef8c9bb5ece1499083949a55311" Sep 17 01:40:39.989785 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.989763 2577 scope.go:117] "RemoveContainer" containerID="b5614ec154deabe53b59e056c59e6f95ab4d5a0a68476df69301a3b101475612" Sep 17 01:40:39.989911 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.989886 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h"] Sep 17 01:40:39.990569 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:40:39.990109 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5614ec154deabe53b59e056c59e6f95ab4d5a0a68476df69301a3b101475612\": container with ID starting with b5614ec154deabe53b59e056c59e6f95ab4d5a0a68476df69301a3b101475612 not found: ID does not exist" containerID="b5614ec154deabe53b59e056c59e6f95ab4d5a0a68476df69301a3b101475612" Sep 17 01:40:39.990569 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.990145 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5614ec154deabe53b59e056c59e6f95ab4d5a0a68476df69301a3b101475612"} err="failed to get container status \"b5614ec154deabe53b59e056c59e6f95ab4d5a0a68476df69301a3b101475612\": rpc error: code = NotFound desc = could not find container \"b5614ec154deabe53b59e056c59e6f95ab4d5a0a68476df69301a3b101475612\": container with ID starting with b5614ec154deabe53b59e056c59e6f95ab4d5a0a68476df69301a3b101475612 not found: ID does not exist" Sep 17 01:40:39.990569 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.990170 2577 scope.go:117] "RemoveContainer" containerID="1f0753f004922ec185dddcf60ea2b850a694cef8c9bb5ece1499083949a55311" Sep 17 01:40:39.990569 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:40:39.990531 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f0753f004922ec185dddcf60ea2b850a694cef8c9bb5ece1499083949a55311\": container with ID starting with 1f0753f004922ec185dddcf60ea2b850a694cef8c9bb5ece1499083949a55311 not found: ID does not exist" containerID="1f0753f004922ec185dddcf60ea2b850a694cef8c9bb5ece1499083949a55311" Sep 17 01:40:39.990569 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.990560 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f0753f004922ec185dddcf60ea2b850a694cef8c9bb5ece1499083949a55311"} err="failed to get container status \"1f0753f004922ec185dddcf60ea2b850a694cef8c9bb5ece1499083949a55311\": rpc error: code = NotFound desc = could not find container \"1f0753f004922ec185dddcf60ea2b850a694cef8c9bb5ece1499083949a55311\": container with ID starting with 1f0753f004922ec185dddcf60ea2b850a694cef8c9bb5ece1499083949a55311 not found: ID does not exist" Sep 17 01:40:39.993238 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:39.993220 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-c7f22faa-gw-openshift-default-6d756f9b96-5855h"] Sep 17 01:40:40.956543 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:40.956507 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd"] Sep 17 01:40:40.957008 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:40.956979 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2617bbc-6a65-4eb2-b54c-6c8a8d749c08" containerName="maas-api" Sep 17 01:40:40.957008 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:40.956992 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2617bbc-6a65-4eb2-b54c-6c8a8d749c08" containerName="maas-api" Sep 17 01:40:40.957008 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:40.957001 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e077cfa-09a7-4607-846a-4235eb1fb8c8" containerName="istio-proxy" Sep 17 01:40:40.957008 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:40.957007 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e077cfa-09a7-4607-846a-4235eb1fb8c8" containerName="istio-proxy" Sep 17 01:40:40.957252 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:40.957094 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e077cfa-09a7-4607-846a-4235eb1fb8c8" containerName="istio-proxy" Sep 17 01:40:40.957252 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:40.957103 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2617bbc-6a65-4eb2-b54c-6c8a8d749c08" containerName="maas-api" Sep 17 01:40:40.961693 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:40.961671 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" Sep 17 01:40:40.964208 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:40.964186 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 01:40:40.965240 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:40.964903 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c9cf09ef-serving-cert\"" Sep 17 01:40:40.965240 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:40.965003 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 01:40:40.965240 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:40.965017 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 01:40:40.965482 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:40.965317 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s7lw7\"" Sep 17 01:40:40.972440 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:40.972415 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd"] Sep 17 01:40:41.066415 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.066374 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-metrics-certs\") pod \"maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd\" (UID: \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" Sep 17 01:40:41.066617 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.066441 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n4dg\" (UniqueName: \"kubernetes.io/projected/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-kube-api-access-8n4dg\") pod \"maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd\" (UID: \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" Sep 17 01:40:41.066617 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.066500 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-openshift-service-ca\") pod \"maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd\" (UID: \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" Sep 17 01:40:41.066617 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.066587 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-maas-api-tls\") pod \"maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd\" (UID: \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" Sep 17 01:40:41.167840 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.167813 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-metrics-certs\") pod \"maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd\" (UID: \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" Sep 17 01:40:41.168089 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.167850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n4dg\" (UniqueName: \"kubernetes.io/projected/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-kube-api-access-8n4dg\") pod \"maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd\" (UID: \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" Sep 17 01:40:41.168089 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.167871 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-openshift-service-ca\") pod \"maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd\" (UID: \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" Sep 17 01:40:41.168089 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.167917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-maas-api-tls\") pod \"maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd\" (UID: \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" Sep 17 01:40:41.168089 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:40:41.168051 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-serving-cert: secret "maas-api-e2e-ait-c9cf09ef-serving-cert" not found Sep 17 01:40:41.168333 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:40:41.168122 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-maas-api-tls podName:6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4 nodeName:}" failed. No retries permitted until 2026-09-17 01:40:41.66810316 +0000 UTC m=+1106.903250326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-maas-api-tls") pod "maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" (UID: "6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4") : secret "maas-api-e2e-ait-c9cf09ef-serving-cert" not found Sep 17 01:40:41.168648 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.168624 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-openshift-service-ca\") pod \"maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd\" (UID: \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" Sep 17 01:40:41.170263 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.170239 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-metrics-certs\") pod \"maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd\" (UID: \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" Sep 17 01:40:41.176239 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.176215 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n4dg\" (UniqueName: \"kubernetes.io/projected/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-kube-api-access-8n4dg\") pod \"maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd\" (UID: \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" Sep 17 01:40:41.212510 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.212443 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e077cfa-09a7-4607-846a-4235eb1fb8c8" path="/var/lib/kubelet/pods/1e077cfa-09a7-4607-846a-4235eb1fb8c8/volumes" Sep 17 01:40:41.520874 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.520788 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt"] Sep 17 01:40:41.521573 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.521551 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e077cfa-09a7-4607-846a-4235eb1fb8c8" containerName="istio-proxy" Sep 17 01:40:41.521626 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.521577 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e077cfa-09a7-4607-846a-4235eb1fb8c8" containerName="istio-proxy" Sep 17 01:40:41.521696 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.521685 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e077cfa-09a7-4607-846a-4235eb1fb8c8" containerName="istio-proxy" Sep 17 01:40:41.526541 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.526520 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt" Sep 17 01:40:41.528571 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.528549 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c9cf09ef\"" Sep 17 01:40:41.528681 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.528606 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c9cf09ef-dockercfg-24qjg\"" Sep 17 01:40:41.532559 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.532531 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt"] Sep 17 01:40:41.565427 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.565398 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd"] Sep 17 01:40:41.569139 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.569120 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd" Sep 17 01:40:41.578972 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.578945 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd"] Sep 17 01:40:41.672676 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.672640 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb-plugins-config-volume\") pod \"payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd\" (UID: \"b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb\") " pod="openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd" Sep 17 01:40:41.672850 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.672743 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4494d845-e90d-435b-8106-9d1a8c66d2e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt\" (UID: \"4494d845-e90d-435b-8106-9d1a8c66d2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt" Sep 17 01:40:41.672850 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.672812 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-maas-api-tls\") pod \"maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd\" (UID: \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" Sep 17 01:40:41.672850 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.672834 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx44d\" (UniqueName: \"kubernetes.io/projected/4494d845-e90d-435b-8106-9d1a8c66d2e6-kube-api-access-wx44d\") pod \"payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt\" (UID: \"4494d845-e90d-435b-8106-9d1a8c66d2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt" Sep 17 01:40:41.673032 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.672857 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz9fs\" (UniqueName: \"kubernetes.io/projected/b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb-kube-api-access-jz9fs\") pod \"payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd\" (UID: \"b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb\") " pod="openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd" Sep 17 01:40:41.675227 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.675205 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-maas-api-tls\") pod \"maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd\" (UID: \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" Sep 17 01:40:41.774064 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.773974 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx44d\" (UniqueName: \"kubernetes.io/projected/4494d845-e90d-435b-8106-9d1a8c66d2e6-kube-api-access-wx44d\") pod \"payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt\" (UID: \"4494d845-e90d-435b-8106-9d1a8c66d2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt" Sep 17 01:40:41.774064 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.774020 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz9fs\" (UniqueName: \"kubernetes.io/projected/b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb-kube-api-access-jz9fs\") pod \"payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd\" (UID: \"b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb\") " pod="openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd" Sep 17 01:40:41.774273 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.774113 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb-plugins-config-volume\") pod \"payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd\" (UID: \"b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb\") " pod="openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd" Sep 17 01:40:41.774273 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.774202 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4494d845-e90d-435b-8106-9d1a8c66d2e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt\" (UID: \"4494d845-e90d-435b-8106-9d1a8c66d2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt" Sep 17 01:40:41.774877 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.774853 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb-plugins-config-volume\") pod \"payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd\" (UID: \"b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb\") " pod="openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd" Sep 17 01:40:41.775040 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.775017 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4494d845-e90d-435b-8106-9d1a8c66d2e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt\" (UID: \"4494d845-e90d-435b-8106-9d1a8c66d2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt" Sep 17 01:40:41.781398 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.781371 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz9fs\" (UniqueName: \"kubernetes.io/projected/b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb-kube-api-access-jz9fs\") pod \"payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd\" (UID: \"b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb\") " pod="openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd" Sep 17 01:40:41.781584 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.781562 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx44d\" (UniqueName: \"kubernetes.io/projected/4494d845-e90d-435b-8106-9d1a8c66d2e6-kube-api-access-wx44d\") pod \"payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt\" (UID: \"4494d845-e90d-435b-8106-9d1a8c66d2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt" Sep 17 01:40:41.840830 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.840801 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt" Sep 17 01:40:41.876068 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.876027 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" Sep 17 01:40:41.879276 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:41.879145 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd" Sep 17 01:40:42.112422 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:42.112393 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt"] Sep 17 01:40:42.113580 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:40:42.113551 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4494d845_e90d_435b_8106_9d1a8c66d2e6.slice/crio-8932911b584beea0ff2945781d932759b195d928a8657b14d5635ff5b087bf79 WatchSource:0}: Error finding container 8932911b584beea0ff2945781d932759b195d928a8657b14d5635ff5b087bf79: Status 404 returned error can't find the container with id 8932911b584beea0ff2945781d932759b195d928a8657b14d5635ff5b087bf79 Sep 17 01:40:42.176000 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:42.175463 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd"] Sep 17 01:40:42.177019 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:40:42.176990 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0eb7dfe_6efb_49f1_a0dd_4447d59a44eb.slice/crio-0fd07664f36f7a99780e7c3478eea2577cfe23ff946624ed4821bd7d02be3264 WatchSource:0}: Error finding container 0fd07664f36f7a99780e7c3478eea2577cfe23ff946624ed4821bd7d02be3264: Status 404 returned error can't find the container with id 0fd07664f36f7a99780e7c3478eea2577cfe23ff946624ed4821bd7d02be3264 Sep 17 01:40:42.180197 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:42.180168 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd"] Sep 17 01:40:42.180304 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:40:42.180285 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4f35c6_20bb_43d1_80ea_78c4efd6d4d4.slice/crio-5b32f7eea9fe5ec81911fbe4194496d2f3546c51a73070dddc45d6bffac9c3a5 WatchSource:0}: Error finding container 5b32f7eea9fe5ec81911fbe4194496d2f3546c51a73070dddc45d6bffac9c3a5: Status 404 returned error can't find the container with id 5b32f7eea9fe5ec81911fbe4194496d2f3546c51a73070dddc45d6bffac9c3a5 Sep 17 01:40:42.206511 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:42.206491 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:40:42.991715 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:42.991669 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" event={"ID":"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4","Type":"ContainerStarted","Data":"ada97cc2b3c589619c14b54afb8369b068a009f1f833fa61d475b02b236b1c0a"} Sep 17 01:40:42.991879 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:42.991744 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" event={"ID":"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4","Type":"ContainerStarted","Data":"5b32f7eea9fe5ec81911fbe4194496d2f3546c51a73070dddc45d6bffac9c3a5"} Sep 17 01:40:42.991879 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:42.991777 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" Sep 17 01:40:42.993444 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:42.993418 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd" event={"ID":"b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb","Type":"ContainerStarted","Data":"50d88f44ee220681027dc7fcbdb45cd7ac2cbd685873405dda41bc9ab526f9f1"} Sep 17 01:40:42.993550 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:42.993467 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd" Sep 17 01:40:42.993550 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:42.993492 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd" event={"ID":"b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb","Type":"ContainerStarted","Data":"0fd07664f36f7a99780e7c3478eea2577cfe23ff946624ed4821bd7d02be3264"} Sep 17 01:40:42.995146 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:42.995118 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt" event={"ID":"4494d845-e90d-435b-8106-9d1a8c66d2e6","Type":"ContainerStarted","Data":"125a1721c197df21f23db5ffd3f71e5814875627eb078e334f3d85a4fed5a2b2"} Sep 17 01:40:42.995256 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:42.995173 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt" Sep 17 01:40:42.995256 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:42.995198 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt" event={"ID":"4494d845-e90d-435b-8106-9d1a8c66d2e6","Type":"ContainerStarted","Data":"8932911b584beea0ff2945781d932759b195d928a8657b14d5635ff5b087bf79"} Sep 17 01:40:43.008386 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:43.008345 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" podStartSLOduration=2.627259017 podStartE2EDuration="3.008331076s" totalImagesPullingTime="381.072059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:40:43.007554816 +0000 UTC m=+1108.242702033" watchObservedRunningTime="2026-09-17 01:40:43.008331076 +0000 UTC m=+1108.243478264" Sep 17 01:40:43.022382 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:43.022340 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt" podStartSLOduration=2.022327961 podStartE2EDuration="2.022327961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:40:43.020913203 +0000 UTC m=+1108.256060392" watchObservedRunningTime="2026-09-17 01:40:43.022327961 +0000 UTC m=+1108.257475149" Sep 17 01:40:43.035038 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:43.035001 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd" podStartSLOduration=2.034987454 podStartE2EDuration="2.034987454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:40:43.033604941 +0000 UTC m=+1108.268752143" watchObservedRunningTime="2026-09-17 01:40:43.034987454 +0000 UTC m=+1108.270134645" Sep 17 01:40:49.008943 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:49.008900 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" Sep 17 01:40:54.005347 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:54.005302 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt" Sep 17 01:40:54.005747 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:40:54.005611 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd" Sep 17 01:41:09.654056 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:09.654019 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd"] Sep 17 01:41:09.654692 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:09.654276 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" podUID="6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4" containerName="maas-api" containerID="cri-o://ada97cc2b3c589619c14b54afb8369b068a009f1f833fa61d475b02b236b1c0a" gracePeriod=30 Sep 17 01:41:09.704619 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:09.704585 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd"] Sep 17 01:41:09.704828 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:09.704785 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd" podUID="b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb" containerName="payload-processing" containerID="cri-o://50d88f44ee220681027dc7fcbdb45cd7ac2cbd685873405dda41bc9ab526f9f1" gracePeriod=30 Sep 17 01:41:09.711553 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:09.711523 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt"] Sep 17 01:41:09.711763 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:09.711735 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt" podUID="4494d845-e90d-435b-8106-9d1a8c66d2e6" containerName="payload-pre-processing" containerID="cri-o://125a1721c197df21f23db5ffd3f71e5814875627eb078e334f3d85a4fed5a2b2" gracePeriod=30 Sep 17 01:41:10.001027 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.001000 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" Sep 17 01:41:10.004038 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.004014 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd" Sep 17 01:41:10.033452 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.033415 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-openshift-service-ca\") pod \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\" (UID: \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\") " Sep 17 01:41:10.033595 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.033491 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-metrics-certs\") pod \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\" (UID: \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\") " Sep 17 01:41:10.033595 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.033576 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz9fs\" (UniqueName: \"kubernetes.io/projected/b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb-kube-api-access-jz9fs\") pod \"b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb\" (UID: \"b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb\") " Sep 17 01:41:10.033708 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.033611 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb-plugins-config-volume\") pod \"b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb\" (UID: \"b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb\") " Sep 17 01:41:10.033708 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.033643 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-maas-api-tls\") pod \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\" (UID: \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\") " Sep 17 01:41:10.033815 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.033739 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n4dg\" (UniqueName: \"kubernetes.io/projected/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-kube-api-access-8n4dg\") pod \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\" (UID: \"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4\") " Sep 17 01:41:10.034499 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.034442 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb" (UID: "b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:41:10.037411 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.037099 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-kube-api-access-8n4dg" (OuterVolumeSpecName: "kube-api-access-8n4dg") pod "6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4" (UID: "6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4"). InnerVolumeSpecName "kube-api-access-8n4dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:41:10.037411 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.037127 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4" (UID: "6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:41:10.037411 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.037361 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4" (UID: "6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:41:10.038472 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.038328 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4" (UID: "6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:41:10.038472 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.038424 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb-kube-api-access-jz9fs" (OuterVolumeSpecName: "kube-api-access-jz9fs") pod "b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb" (UID: "b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb"). InnerVolumeSpecName "kube-api-access-jz9fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:41:10.040851 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.040832 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt" Sep 17 01:41:10.135133 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.135102 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wx44d\" (UniqueName: \"kubernetes.io/projected/4494d845-e90d-435b-8106-9d1a8c66d2e6-kube-api-access-wx44d\") pod \"4494d845-e90d-435b-8106-9d1a8c66d2e6\" (UID: \"4494d845-e90d-435b-8106-9d1a8c66d2e6\") " Sep 17 01:41:10.135296 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.135243 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4494d845-e90d-435b-8106-9d1a8c66d2e6-plugins-config-volume\") pod \"4494d845-e90d-435b-8106-9d1a8c66d2e6\" (UID: \"4494d845-e90d-435b-8106-9d1a8c66d2e6\") " Sep 17 01:41:10.135429 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.135416 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz9fs\" (UniqueName: \"kubernetes.io/projected/b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb-kube-api-access-jz9fs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:10.135470 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.135430 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb-plugins-config-volume\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:10.135470 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.135440 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-maas-api-tls\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:10.135470 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.135450 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n4dg\" (UniqueName: \"kubernetes.io/projected/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-kube-api-access-8n4dg\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:10.135470 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.135458 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-openshift-service-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:10.135470 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.135468 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4-metrics-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:10.135631 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.135573 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4494d845-e90d-435b-8106-9d1a8c66d2e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4494d845-e90d-435b-8106-9d1a8c66d2e6" (UID: "4494d845-e90d-435b-8106-9d1a8c66d2e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:41:10.137186 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.137162 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4494d845-e90d-435b-8106-9d1a8c66d2e6-kube-api-access-wx44d" (OuterVolumeSpecName: "kube-api-access-wx44d") pod "4494d845-e90d-435b-8106-9d1a8c66d2e6" (UID: "4494d845-e90d-435b-8106-9d1a8c66d2e6"). InnerVolumeSpecName "kube-api-access-wx44d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:41:10.183185 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.183095 2577 generic.go:356] "Generic (PLEG): container finished" podID="6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4" containerID="ada97cc2b3c589619c14b54afb8369b068a009f1f833fa61d475b02b236b1c0a" exitCode=0 Sep 17 01:41:10.183185 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.183151 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" Sep 17 01:41:10.183396 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.183175 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" event={"ID":"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4","Type":"ContainerDied","Data":"ada97cc2b3c589619c14b54afb8369b068a009f1f833fa61d475b02b236b1c0a"} Sep 17 01:41:10.183396 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.183230 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd" event={"ID":"6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4","Type":"ContainerDied","Data":"5b32f7eea9fe5ec81911fbe4194496d2f3546c51a73070dddc45d6bffac9c3a5"} Sep 17 01:41:10.183396 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.183253 2577 scope.go:117] "RemoveContainer" containerID="ada97cc2b3c589619c14b54afb8369b068a009f1f833fa61d475b02b236b1c0a" Sep 17 01:41:10.184654 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.184632 2577 generic.go:356] "Generic (PLEG): container finished" podID="b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb" containerID="50d88f44ee220681027dc7fcbdb45cd7ac2cbd685873405dda41bc9ab526f9f1" exitCode=0 Sep 17 01:41:10.184782 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.184675 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd" Sep 17 01:41:10.184782 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.184726 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd" event={"ID":"b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb","Type":"ContainerDied","Data":"50d88f44ee220681027dc7fcbdb45cd7ac2cbd685873405dda41bc9ab526f9f1"} Sep 17 01:41:10.184899 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.184784 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd" event={"ID":"b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb","Type":"ContainerDied","Data":"0fd07664f36f7a99780e7c3478eea2577cfe23ff946624ed4821bd7d02be3264"} Sep 17 01:41:10.186301 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.186274 2577 generic.go:356] "Generic (PLEG): container finished" podID="4494d845-e90d-435b-8106-9d1a8c66d2e6" containerID="125a1721c197df21f23db5ffd3f71e5814875627eb078e334f3d85a4fed5a2b2" exitCode=0 Sep 17 01:41:10.186413 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.186317 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt" Sep 17 01:41:10.186413 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.186363 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt" event={"ID":"4494d845-e90d-435b-8106-9d1a8c66d2e6","Type":"ContainerDied","Data":"125a1721c197df21f23db5ffd3f71e5814875627eb078e334f3d85a4fed5a2b2"} Sep 17 01:41:10.186538 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.186417 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt" event={"ID":"4494d845-e90d-435b-8106-9d1a8c66d2e6","Type":"ContainerDied","Data":"8932911b584beea0ff2945781d932759b195d928a8657b14d5635ff5b087bf79"} Sep 17 01:41:10.195561 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.195544 2577 scope.go:117] "RemoveContainer" containerID="ada97cc2b3c589619c14b54afb8369b068a009f1f833fa61d475b02b236b1c0a" Sep 17 01:41:10.195892 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:41:10.195852 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada97cc2b3c589619c14b54afb8369b068a009f1f833fa61d475b02b236b1c0a\": container with ID starting with ada97cc2b3c589619c14b54afb8369b068a009f1f833fa61d475b02b236b1c0a not found: ID does not exist" containerID="ada97cc2b3c589619c14b54afb8369b068a009f1f833fa61d475b02b236b1c0a" Sep 17 01:41:10.196015 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.195889 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada97cc2b3c589619c14b54afb8369b068a009f1f833fa61d475b02b236b1c0a"} err="failed to get container status \"ada97cc2b3c589619c14b54afb8369b068a009f1f833fa61d475b02b236b1c0a\": rpc error: code = NotFound desc = could not find container \"ada97cc2b3c589619c14b54afb8369b068a009f1f833fa61d475b02b236b1c0a\": container with ID starting with ada97cc2b3c589619c14b54afb8369b068a009f1f833fa61d475b02b236b1c0a not found: ID does not exist" Sep 17 01:41:10.196015 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.195912 2577 scope.go:117] "RemoveContainer" containerID="50d88f44ee220681027dc7fcbdb45cd7ac2cbd685873405dda41bc9ab526f9f1" Sep 17 01:41:10.206434 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.206416 2577 scope.go:117] "RemoveContainer" containerID="50d88f44ee220681027dc7fcbdb45cd7ac2cbd685873405dda41bc9ab526f9f1" Sep 17 01:41:10.206684 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:41:10.206667 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50d88f44ee220681027dc7fcbdb45cd7ac2cbd685873405dda41bc9ab526f9f1\": container with ID starting with 50d88f44ee220681027dc7fcbdb45cd7ac2cbd685873405dda41bc9ab526f9f1 not found: ID does not exist" containerID="50d88f44ee220681027dc7fcbdb45cd7ac2cbd685873405dda41bc9ab526f9f1" Sep 17 01:41:10.206763 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.206693 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d88f44ee220681027dc7fcbdb45cd7ac2cbd685873405dda41bc9ab526f9f1"} err="failed to get container status \"50d88f44ee220681027dc7fcbdb45cd7ac2cbd685873405dda41bc9ab526f9f1\": rpc error: code = NotFound desc = could not find container \"50d88f44ee220681027dc7fcbdb45cd7ac2cbd685873405dda41bc9ab526f9f1\": container with ID starting with 50d88f44ee220681027dc7fcbdb45cd7ac2cbd685873405dda41bc9ab526f9f1 not found: ID does not exist" Sep 17 01:41:10.206763 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.206713 2577 scope.go:117] "RemoveContainer" containerID="125a1721c197df21f23db5ffd3f71e5814875627eb078e334f3d85a4fed5a2b2" Sep 17 01:41:10.207175 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.207151 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd"] Sep 17 01:41:10.213739 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.213713 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9cf09ef-5585cf4869-fbtgd"] Sep 17 01:41:10.219621 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.219599 2577 scope.go:117] "RemoveContainer" containerID="125a1721c197df21f23db5ffd3f71e5814875627eb078e334f3d85a4fed5a2b2" Sep 17 01:41:10.223315 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:41:10.223269 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"125a1721c197df21f23db5ffd3f71e5814875627eb078e334f3d85a4fed5a2b2\": container with ID starting with 125a1721c197df21f23db5ffd3f71e5814875627eb078e334f3d85a4fed5a2b2 not found: ID does not exist" containerID="125a1721c197df21f23db5ffd3f71e5814875627eb078e334f3d85a4fed5a2b2" Sep 17 01:41:10.223474 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.223309 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125a1721c197df21f23db5ffd3f71e5814875627eb078e334f3d85a4fed5a2b2"} err="failed to get container status \"125a1721c197df21f23db5ffd3f71e5814875627eb078e334f3d85a4fed5a2b2\": rpc error: code = NotFound desc = could not find container \"125a1721c197df21f23db5ffd3f71e5814875627eb078e334f3d85a4fed5a2b2\": container with ID starting with 125a1721c197df21f23db5ffd3f71e5814875627eb078e334f3d85a4fed5a2b2 not found: ID does not exist" Sep 17 01:41:10.225688 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.225661 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd"] Sep 17 01:41:10.228848 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.228815 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9cf09ef-645c5654df-frgdd"] Sep 17 01:41:10.235403 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.235380 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt"] Sep 17 01:41:10.236178 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.236160 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx44d\" (UniqueName: \"kubernetes.io/projected/4494d845-e90d-435b-8106-9d1a8c66d2e6-kube-api-access-wx44d\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:10.236230 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.236181 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4494d845-e90d-435b-8106-9d1a8c66d2e6-plugins-config-volume\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:10.237716 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:10.237697 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9cf09ef-67fdbcb5bb-nnqdt"] Sep 17 01:41:11.212000 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.211958 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4494d845-e90d-435b-8106-9d1a8c66d2e6" path="/var/lib/kubelet/pods/4494d845-e90d-435b-8106-9d1a8c66d2e6/volumes" Sep 17 01:41:11.212401 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.212385 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4" path="/var/lib/kubelet/pods/6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4/volumes" Sep 17 01:41:11.212710 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.212698 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb" path="/var/lib/kubelet/pods/b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb/volumes" Sep 17 01:41:11.580303 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.580208 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm"] Sep 17 01:41:11.580716 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.580699 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4494d845-e90d-435b-8106-9d1a8c66d2e6" containerName="payload-pre-processing" Sep 17 01:41:11.580716 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.580715 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4494d845-e90d-435b-8106-9d1a8c66d2e6" containerName="payload-pre-processing" Sep 17 01:41:11.580871 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.580732 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4" containerName="maas-api" Sep 17 01:41:11.580871 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.580738 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4" containerName="maas-api" Sep 17 01:41:11.580871 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.580754 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb" containerName="payload-processing" Sep 17 01:41:11.580871 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.580759 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb" containerName="payload-processing" Sep 17 01:41:11.580871 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.580838 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0eb7dfe-6efb-49f1-a0dd-4447d59a44eb" containerName="payload-processing" Sep 17 01:41:11.580871 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.580851 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4494d845-e90d-435b-8106-9d1a8c66d2e6" containerName="payload-pre-processing" Sep 17 01:41:11.580871 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.580859 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f4f35c6-20bb-43d1-80ea-78c4efd6d4d4" containerName="maas-api" Sep 17 01:41:11.585479 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.585458 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.587650 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.587626 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5f24c314-openshift-default-dockercfg-knpm6\"" Sep 17 01:41:11.597950 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.597893 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm"] Sep 17 01:41:11.648891 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.648857 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c791e7-bc3f-457b-8013-854a41f08da2-istio-podinfo\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.649088 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.648905 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c791e7-bc3f-457b-8013-854a41f08da2-istio-token\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.649088 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.648955 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-workload-certs\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.649088 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.648973 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c791e7-bc3f-457b-8013-854a41f08da2-istiod-ca-cert\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.649088 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.648997 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-credential-socket\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.649088 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.649074 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58bj7\" (UniqueName: \"kubernetes.io/projected/44c791e7-bc3f-457b-8013-854a41f08da2-kube-api-access-58bj7\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.649273 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.649127 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-istio-data\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.649273 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.649154 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-workload-socket\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.649273 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.649169 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-istio-envoy\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.750641 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.750602 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58bj7\" (UniqueName: \"kubernetes.io/projected/44c791e7-bc3f-457b-8013-854a41f08da2-kube-api-access-58bj7\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.750819 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.750658 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-istio-data\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.750873 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.750839 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-workload-socket\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.750915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.750887 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-istio-envoy\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.751015 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.750997 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c791e7-bc3f-457b-8013-854a41f08da2-istio-podinfo\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.751094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.751052 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-istio-data\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.751094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.751085 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c791e7-bc3f-457b-8013-854a41f08da2-istio-token\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.751197 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.751114 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-workload-certs\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.751197 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.751147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c791e7-bc3f-457b-8013-854a41f08da2-istiod-ca-cert\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.751197 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.751180 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-workload-socket\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.751197 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.751194 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-credential-socket\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.751427 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.751383 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-workload-certs\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.751552 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.751531 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-credential-socket\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.751762 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.751742 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c791e7-bc3f-457b-8013-854a41f08da2-istiod-ca-cert\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.753282 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.753253 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-istio-envoy\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.753495 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.753476 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c791e7-bc3f-457b-8013-854a41f08da2-istio-podinfo\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.760198 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.760176 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c791e7-bc3f-457b-8013-854a41f08da2-istio-token\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.766516 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.766486 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58bj7\" (UniqueName: \"kubernetes.io/projected/44c791e7-bc3f-457b-8013-854a41f08da2-kube-api-access-58bj7\") pod \"e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:11.898220 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:11.898186 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:12.176533 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:12.176496 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm"] Sep 17 01:41:12.177050 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:41:12.177023 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c791e7_bc3f_457b_8013_854a41f08da2.slice/crio-088dd026f0260f0680cd56e27be516bc5285ef5771de347a065fc7294044533e WatchSource:0}: Error finding container 088dd026f0260f0680cd56e27be516bc5285ef5771de347a065fc7294044533e: Status 404 returned error can't find the container with id 088dd026f0260f0680cd56e27be516bc5285ef5771de347a065fc7294044533e Sep 17 01:41:12.179156 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:12.179123 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:41:12.179260 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:12.179193 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:41:12.179260 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:12.179227 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:41:12.204124 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:12.204094 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" event={"ID":"44c791e7-bc3f-457b-8013-854a41f08da2","Type":"ContainerStarted","Data":"088dd026f0260f0680cd56e27be516bc5285ef5771de347a065fc7294044533e"} Sep 17 01:41:13.213154 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:13.213119 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" event={"ID":"44c791e7-bc3f-457b-8013-854a41f08da2","Type":"ContainerStarted","Data":"9cf6654c8deb1d20ad1f175a2cef5b48d0472f2f7e059f094e0a0551c1500ea1"} Sep 17 01:41:13.232646 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:13.232587 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" podStartSLOduration=2.232565967 podStartE2EDuration="2.232565967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:41:13.230036613 +0000 UTC m=+1138.465183801" watchObservedRunningTime="2026-09-17 01:41:13.232565967 +0000 UTC m=+1138.467713156" Sep 17 01:41:13.898569 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:13.898526 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:13.904052 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:13.904025 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:14.221164 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:14.221079 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:14.222145 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:14.222125 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:17.456387 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.456353 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977"] Sep 17 01:41:17.461610 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.461586 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" Sep 17 01:41:17.464199 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.464170 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 01:41:17.464439 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.464423 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 01:41:17.464895 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.464873 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s7lw7\"" Sep 17 01:41:17.465050 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.464926 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 01:41:17.465050 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.464871 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-5f24c314-serving-cert\"" Sep 17 01:41:17.472075 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.472042 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977"] Sep 17 01:41:17.508603 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.508577 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0883d7ed-0c50-43b7-ae04-dbce40543a62-maas-api-tls\") pod \"maas-api-e2e-derive-5f24c314-66c5c85779-l2977\" (UID: \"0883d7ed-0c50-43b7-ae04-dbce40543a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" Sep 17 01:41:17.508750 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.508623 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0883d7ed-0c50-43b7-ae04-dbce40543a62-openshift-service-ca\") pod \"maas-api-e2e-derive-5f24c314-66c5c85779-l2977\" (UID: \"0883d7ed-0c50-43b7-ae04-dbce40543a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" Sep 17 01:41:17.508750 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.508655 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0883d7ed-0c50-43b7-ae04-dbce40543a62-metrics-certs\") pod \"maas-api-e2e-derive-5f24c314-66c5c85779-l2977\" (UID: \"0883d7ed-0c50-43b7-ae04-dbce40543a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" Sep 17 01:41:17.508750 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.508736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctnv6\" (UniqueName: \"kubernetes.io/projected/0883d7ed-0c50-43b7-ae04-dbce40543a62-kube-api-access-ctnv6\") pod \"maas-api-e2e-derive-5f24c314-66c5c85779-l2977\" (UID: \"0883d7ed-0c50-43b7-ae04-dbce40543a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" Sep 17 01:41:17.609338 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.609276 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctnv6\" (UniqueName: \"kubernetes.io/projected/0883d7ed-0c50-43b7-ae04-dbce40543a62-kube-api-access-ctnv6\") pod \"maas-api-e2e-derive-5f24c314-66c5c85779-l2977\" (UID: \"0883d7ed-0c50-43b7-ae04-dbce40543a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" Sep 17 01:41:17.609512 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.609449 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0883d7ed-0c50-43b7-ae04-dbce40543a62-maas-api-tls\") pod \"maas-api-e2e-derive-5f24c314-66c5c85779-l2977\" (UID: \"0883d7ed-0c50-43b7-ae04-dbce40543a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" Sep 17 01:41:17.609512 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.609508 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0883d7ed-0c50-43b7-ae04-dbce40543a62-openshift-service-ca\") pod \"maas-api-e2e-derive-5f24c314-66c5c85779-l2977\" (UID: \"0883d7ed-0c50-43b7-ae04-dbce40543a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" Sep 17 01:41:17.609594 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.609544 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0883d7ed-0c50-43b7-ae04-dbce40543a62-metrics-certs\") pod \"maas-api-e2e-derive-5f24c314-66c5c85779-l2977\" (UID: \"0883d7ed-0c50-43b7-ae04-dbce40543a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" Sep 17 01:41:17.610232 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.610207 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0883d7ed-0c50-43b7-ae04-dbce40543a62-openshift-service-ca\") pod \"maas-api-e2e-derive-5f24c314-66c5c85779-l2977\" (UID: \"0883d7ed-0c50-43b7-ae04-dbce40543a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" Sep 17 01:41:17.611945 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.611910 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0883d7ed-0c50-43b7-ae04-dbce40543a62-maas-api-tls\") pod \"maas-api-e2e-derive-5f24c314-66c5c85779-l2977\" (UID: \"0883d7ed-0c50-43b7-ae04-dbce40543a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" Sep 17 01:41:17.612046 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.612028 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0883d7ed-0c50-43b7-ae04-dbce40543a62-metrics-certs\") pod \"maas-api-e2e-derive-5f24c314-66c5c85779-l2977\" (UID: \"0883d7ed-0c50-43b7-ae04-dbce40543a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" Sep 17 01:41:17.617556 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.617533 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctnv6\" (UniqueName: \"kubernetes.io/projected/0883d7ed-0c50-43b7-ae04-dbce40543a62-kube-api-access-ctnv6\") pod \"maas-api-e2e-derive-5f24c314-66c5c85779-l2977\" (UID: \"0883d7ed-0c50-43b7-ae04-dbce40543a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" Sep 17 01:41:17.775514 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:17.775420 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" Sep 17 01:41:18.056724 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:18.056685 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977"] Sep 17 01:41:18.061561 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:41:18.061508 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0883d7ed_0c50_43b7_ae04_dbce40543a62.slice/crio-a62136b5d8f543a79557ea1f55150415336278ff5157402c914fc650bf5a2f88 WatchSource:0}: Error finding container a62136b5d8f543a79557ea1f55150415336278ff5157402c914fc650bf5a2f88: Status 404 returned error can't find the container with id a62136b5d8f543a79557ea1f55150415336278ff5157402c914fc650bf5a2f88 Sep 17 01:41:18.248615 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:18.248572 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" event={"ID":"0883d7ed-0c50-43b7-ae04-dbce40543a62","Type":"ContainerStarted","Data":"a62136b5d8f543a79557ea1f55150415336278ff5157402c914fc650bf5a2f88"} Sep 17 01:41:19.256314 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:19.256263 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" event={"ID":"0883d7ed-0c50-43b7-ae04-dbce40543a62","Type":"ContainerStarted","Data":"2fc0daf3f4881261c891646216e78e114af451f4bc4c8ca5618fe418e23e6a2f"} Sep 17 01:41:19.256754 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:19.256342 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" Sep 17 01:41:19.272390 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:19.272331 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" podStartSLOduration=1.75837913 podStartE2EDuration="2.272314531s" totalImagesPullingTime="513.935401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:41:19.270511842 +0000 UTC m=+1144.505659029" watchObservedRunningTime="2026-09-17 01:41:19.272314531 +0000 UTC m=+1144.507461718" Sep 17 01:41:25.267742 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:25.267711 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" Sep 17 01:41:39.395233 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:39.395199 2577 generic.go:356] "Generic (PLEG): container finished" podID="44c791e7-bc3f-457b-8013-854a41f08da2" containerID="9cf6654c8deb1d20ad1f175a2cef5b48d0472f2f7e059f094e0a0551c1500ea1" exitCode=0 Sep 17 01:41:39.395635 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:39.395269 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" event={"ID":"44c791e7-bc3f-457b-8013-854a41f08da2","Type":"ContainerDied","Data":"9cf6654c8deb1d20ad1f175a2cef5b48d0472f2f7e059f094e0a0551c1500ea1"} Sep 17 01:41:39.395690 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:39.395676 2577 scope.go:117] "RemoveContainer" containerID="9cf6654c8deb1d20ad1f175a2cef5b48d0472f2f7e059f094e0a0551c1500ea1" Sep 17 01:41:39.396941 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:39.396908 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:41:39.397007 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:39.396992 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:41:39.397043 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:39.397028 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:41:40.406122 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:40.406085 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" event={"ID":"44c791e7-bc3f-457b-8013-854a41f08da2","Type":"ContainerStarted","Data":"b1aea8cf3f67682efc028bc471cd45ee9d6fa282c3c78f14b7f94defe090fde7"} Sep 17 01:41:40.898533 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:40.898498 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:40.899802 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:40.899766 2577 patch_prober.go:28] interesting pod/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 17 01:41:40.899950 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:40.899817 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" podUID="44c791e7-bc3f-457b-8013-854a41f08da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 17 01:41:41.899388 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:41.899347 2577 patch_prober.go:28] interesting pod/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 17 01:41:41.899852 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:41.899414 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" podUID="44c791e7-bc3f-457b-8013-854a41f08da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 17 01:41:42.899190 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:42.899148 2577 patch_prober.go:28] interesting pod/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 17 01:41:42.899401 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:42.899213 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" podUID="44c791e7-bc3f-457b-8013-854a41f08da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 17 01:41:43.206559 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:43.206474 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:41:43.899603 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:43.899565 2577 patch_prober.go:28] interesting pod/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 17 01:41:43.900009 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:43.899625 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" podUID="44c791e7-bc3f-457b-8013-854a41f08da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 17 01:41:44.228159 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:44.228044 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:44.899152 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:44.899109 2577 patch_prober.go:28] interesting pod/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 17 01:41:44.899336 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:44.899178 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" podUID="44c791e7-bc3f-457b-8013-854a41f08da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 17 01:41:45.902781 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:45.902742 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:45.903981 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:45.903926 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:46.351568 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:46.351532 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977"] Sep 17 01:41:46.351799 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:46.351773 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" podUID="0883d7ed-0c50-43b7-ae04-dbce40543a62" containerName="maas-api" containerID="cri-o://2fc0daf3f4881261c891646216e78e114af451f4bc4c8ca5618fe418e23e6a2f" gracePeriod=30 Sep 17 01:41:46.611372 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:46.611293 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" Sep 17 01:41:46.668652 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:46.668616 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm"] Sep 17 01:41:46.704780 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:46.704744 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0883d7ed-0c50-43b7-ae04-dbce40543a62-metrics-certs\") pod \"0883d7ed-0c50-43b7-ae04-dbce40543a62\" (UID: \"0883d7ed-0c50-43b7-ae04-dbce40543a62\") " Sep 17 01:41:46.704991 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:46.704806 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0883d7ed-0c50-43b7-ae04-dbce40543a62-maas-api-tls\") pod \"0883d7ed-0c50-43b7-ae04-dbce40543a62\" (UID: \"0883d7ed-0c50-43b7-ae04-dbce40543a62\") " Sep 17 01:41:46.704991 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:46.704890 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctnv6\" (UniqueName: \"kubernetes.io/projected/0883d7ed-0c50-43b7-ae04-dbce40543a62-kube-api-access-ctnv6\") pod \"0883d7ed-0c50-43b7-ae04-dbce40543a62\" (UID: \"0883d7ed-0c50-43b7-ae04-dbce40543a62\") " Sep 17 01:41:46.705128 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:46.705028 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0883d7ed-0c50-43b7-ae04-dbce40543a62-openshift-service-ca\") pod \"0883d7ed-0c50-43b7-ae04-dbce40543a62\" (UID: \"0883d7ed-0c50-43b7-ae04-dbce40543a62\") " Sep 17 01:41:46.705468 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:46.705434 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0883d7ed-0c50-43b7-ae04-dbce40543a62-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0883d7ed-0c50-43b7-ae04-dbce40543a62" (UID: "0883d7ed-0c50-43b7-ae04-dbce40543a62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:41:46.707167 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:46.707142 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0883d7ed-0c50-43b7-ae04-dbce40543a62-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0883d7ed-0c50-43b7-ae04-dbce40543a62" (UID: "0883d7ed-0c50-43b7-ae04-dbce40543a62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:41:46.707273 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:46.707165 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0883d7ed-0c50-43b7-ae04-dbce40543a62-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0883d7ed-0c50-43b7-ae04-dbce40543a62" (UID: "0883d7ed-0c50-43b7-ae04-dbce40543a62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:41:46.707273 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:46.707188 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0883d7ed-0c50-43b7-ae04-dbce40543a62-kube-api-access-ctnv6" (OuterVolumeSpecName: "kube-api-access-ctnv6") pod "0883d7ed-0c50-43b7-ae04-dbce40543a62" (UID: "0883d7ed-0c50-43b7-ae04-dbce40543a62"). InnerVolumeSpecName "kube-api-access-ctnv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:41:46.806121 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:46.806068 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0883d7ed-0c50-43b7-ae04-dbce40543a62-openshift-service-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:46.806121 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:46.806124 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0883d7ed-0c50-43b7-ae04-dbce40543a62-metrics-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:46.806331 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:46.806141 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0883d7ed-0c50-43b7-ae04-dbce40543a62-maas-api-tls\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:46.806331 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:46.806157 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctnv6\" (UniqueName: \"kubernetes.io/projected/0883d7ed-0c50-43b7-ae04-dbce40543a62-kube-api-access-ctnv6\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:47.467242 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:47.467206 2577 generic.go:356] "Generic (PLEG): container finished" podID="0883d7ed-0c50-43b7-ae04-dbce40543a62" containerID="2fc0daf3f4881261c891646216e78e114af451f4bc4c8ca5618fe418e23e6a2f" exitCode=0 Sep 17 01:41:47.467727 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:47.467251 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" Sep 17 01:41:47.467727 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:47.467284 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" event={"ID":"0883d7ed-0c50-43b7-ae04-dbce40543a62","Type":"ContainerDied","Data":"2fc0daf3f4881261c891646216e78e114af451f4bc4c8ca5618fe418e23e6a2f"} Sep 17 01:41:47.467727 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:47.467343 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977" event={"ID":"0883d7ed-0c50-43b7-ae04-dbce40543a62","Type":"ContainerDied","Data":"a62136b5d8f543a79557ea1f55150415336278ff5157402c914fc650bf5a2f88"} Sep 17 01:41:47.467727 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:47.467370 2577 scope.go:117] "RemoveContainer" containerID="2fc0daf3f4881261c891646216e78e114af451f4bc4c8ca5618fe418e23e6a2f" Sep 17 01:41:47.467727 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:47.467582 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" podUID="44c791e7-bc3f-457b-8013-854a41f08da2" containerName="istio-proxy" containerID="cri-o://b1aea8cf3f67682efc028bc471cd45ee9d6fa282c3c78f14b7f94defe090fde7" gracePeriod=30 Sep 17 01:41:47.477655 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:47.477636 2577 scope.go:117] "RemoveContainer" containerID="2fc0daf3f4881261c891646216e78e114af451f4bc4c8ca5618fe418e23e6a2f" Sep 17 01:41:47.478004 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:41:47.477964 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc0daf3f4881261c891646216e78e114af451f4bc4c8ca5618fe418e23e6a2f\": container with ID starting with 2fc0daf3f4881261c891646216e78e114af451f4bc4c8ca5618fe418e23e6a2f not found: ID does not exist" containerID="2fc0daf3f4881261c891646216e78e114af451f4bc4c8ca5618fe418e23e6a2f" Sep 17 01:41:47.478062 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:47.477998 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc0daf3f4881261c891646216e78e114af451f4bc4c8ca5618fe418e23e6a2f"} err="failed to get container status \"2fc0daf3f4881261c891646216e78e114af451f4bc4c8ca5618fe418e23e6a2f\": rpc error: code = NotFound desc = could not find container \"2fc0daf3f4881261c891646216e78e114af451f4bc4c8ca5618fe418e23e6a2f\": container with ID starting with 2fc0daf3f4881261c891646216e78e114af451f4bc4c8ca5618fe418e23e6a2f not found: ID does not exist" Sep 17 01:41:47.483314 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:47.483287 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977"] Sep 17 01:41:47.488960 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:47.488917 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5f24c314-66c5c85779-l2977"] Sep 17 01:41:48.205690 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:48.205653 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:41:49.213159 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:49.213125 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0883d7ed-0c50-43b7-ae04-dbce40543a62" path="/var/lib/kubelet/pods/0883d7ed-0c50-43b7-ae04-dbce40543a62/volumes" Sep 17 01:41:52.734849 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.734776 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:52.765537 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.765510 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-istio-envoy\") pod \"44c791e7-bc3f-457b-8013-854a41f08da2\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " Sep 17 01:41:52.765537 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.765541 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-workload-socket\") pod \"44c791e7-bc3f-457b-8013-854a41f08da2\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " Sep 17 01:41:52.765770 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.765567 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c791e7-bc3f-457b-8013-854a41f08da2-istio-podinfo\") pod \"44c791e7-bc3f-457b-8013-854a41f08da2\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " Sep 17 01:41:52.765770 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.765609 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c791e7-bc3f-457b-8013-854a41f08da2-istiod-ca-cert\") pod \"44c791e7-bc3f-457b-8013-854a41f08da2\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " Sep 17 01:41:52.765770 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.765628 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-credential-socket\") pod \"44c791e7-bc3f-457b-8013-854a41f08da2\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " Sep 17 01:41:52.765770 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.765663 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58bj7\" (UniqueName: \"kubernetes.io/projected/44c791e7-bc3f-457b-8013-854a41f08da2-kube-api-access-58bj7\") pod \"44c791e7-bc3f-457b-8013-854a41f08da2\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " Sep 17 01:41:52.765770 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.765700 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-istio-data\") pod \"44c791e7-bc3f-457b-8013-854a41f08da2\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " Sep 17 01:41:52.765770 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.765757 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-workload-certs\") pod \"44c791e7-bc3f-457b-8013-854a41f08da2\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " Sep 17 01:41:52.766114 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.765799 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c791e7-bc3f-457b-8013-854a41f08da2-istio-token\") pod \"44c791e7-bc3f-457b-8013-854a41f08da2\" (UID: \"44c791e7-bc3f-457b-8013-854a41f08da2\") " Sep 17 01:41:52.766114 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.765823 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "44c791e7-bc3f-457b-8013-854a41f08da2" (UID: "44c791e7-bc3f-457b-8013-854a41f08da2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:41:52.766114 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.765998 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "44c791e7-bc3f-457b-8013-854a41f08da2" (UID: "44c791e7-bc3f-457b-8013-854a41f08da2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:41:52.766114 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.766071 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-istio-data" (OuterVolumeSpecName: "istio-data") pod "44c791e7-bc3f-457b-8013-854a41f08da2" (UID: "44c791e7-bc3f-457b-8013-854a41f08da2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:41:52.766114 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.766081 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c791e7-bc3f-457b-8013-854a41f08da2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "44c791e7-bc3f-457b-8013-854a41f08da2" (UID: "44c791e7-bc3f-457b-8013-854a41f08da2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:41:52.766372 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.766224 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-workload-socket\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:52.766372 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.766242 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c791e7-bc3f-457b-8013-854a41f08da2-istiod-ca-cert\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:52.766372 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.766246 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "44c791e7-bc3f-457b-8013-854a41f08da2" (UID: "44c791e7-bc3f-457b-8013-854a41f08da2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:41:52.766372 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.766257 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-credential-socket\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:52.766372 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.766273 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-istio-data\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:52.768026 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.767999 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44c791e7-bc3f-457b-8013-854a41f08da2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "44c791e7-bc3f-457b-8013-854a41f08da2" (UID: "44c791e7-bc3f-457b-8013-854a41f08da2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 01:41:52.768143 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.768003 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c791e7-bc3f-457b-8013-854a41f08da2-kube-api-access-58bj7" (OuterVolumeSpecName: "kube-api-access-58bj7") pod "44c791e7-bc3f-457b-8013-854a41f08da2" (UID: "44c791e7-bc3f-457b-8013-854a41f08da2"). InnerVolumeSpecName "kube-api-access-58bj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:41:52.768379 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.768364 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "44c791e7-bc3f-457b-8013-854a41f08da2" (UID: "44c791e7-bc3f-457b-8013-854a41f08da2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:41:52.768424 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.768408 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c791e7-bc3f-457b-8013-854a41f08da2-istio-token" (OuterVolumeSpecName: "istio-token") pod "44c791e7-bc3f-457b-8013-854a41f08da2" (UID: "44c791e7-bc3f-457b-8013-854a41f08da2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:41:52.867652 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.867622 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-workload-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:52.867652 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.867649 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c791e7-bc3f-457b-8013-854a41f08da2-istio-token\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:52.867652 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.867659 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c791e7-bc3f-457b-8013-854a41f08da2-istio-envoy\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:52.867877 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.867667 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c791e7-bc3f-457b-8013-854a41f08da2-istio-podinfo\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:52.867877 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:52.867676 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58bj7\" (UniqueName: \"kubernetes.io/projected/44c791e7-bc3f-457b-8013-854a41f08da2-kube-api-access-58bj7\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:41:53.508667 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:53.508577 2577 generic.go:356] "Generic (PLEG): container finished" podID="44c791e7-bc3f-457b-8013-854a41f08da2" containerID="b1aea8cf3f67682efc028bc471cd45ee9d6fa282c3c78f14b7f94defe090fde7" exitCode=0 Sep 17 01:41:53.508667 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:53.508627 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" Sep 17 01:41:53.508850 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:53.508660 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" event={"ID":"44c791e7-bc3f-457b-8013-854a41f08da2","Type":"ContainerDied","Data":"b1aea8cf3f67682efc028bc471cd45ee9d6fa282c3c78f14b7f94defe090fde7"} Sep 17 01:41:53.508850 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:53.508720 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm" event={"ID":"44c791e7-bc3f-457b-8013-854a41f08da2","Type":"ContainerDied","Data":"088dd026f0260f0680cd56e27be516bc5285ef5771de347a065fc7294044533e"} Sep 17 01:41:53.508850 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:53.508750 2577 scope.go:117] "RemoveContainer" containerID="b1aea8cf3f67682efc028bc471cd45ee9d6fa282c3c78f14b7f94defe090fde7" Sep 17 01:41:53.519597 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:53.519572 2577 scope.go:117] "RemoveContainer" containerID="9cf6654c8deb1d20ad1f175a2cef5b48d0472f2f7e059f094e0a0551c1500ea1" Sep 17 01:41:53.526707 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:53.526671 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm"] Sep 17 01:41:53.532110 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:53.532082 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5f24c314-openshift-default-f68dc9444-tp4fm"] Sep 17 01:41:53.534536 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:53.534517 2577 scope.go:117] "RemoveContainer" containerID="b1aea8cf3f67682efc028bc471cd45ee9d6fa282c3c78f14b7f94defe090fde7" Sep 17 01:41:53.534942 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:41:53.534905 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1aea8cf3f67682efc028bc471cd45ee9d6fa282c3c78f14b7f94defe090fde7\": container with ID starting with b1aea8cf3f67682efc028bc471cd45ee9d6fa282c3c78f14b7f94defe090fde7 not found: ID does not exist" containerID="b1aea8cf3f67682efc028bc471cd45ee9d6fa282c3c78f14b7f94defe090fde7" Sep 17 01:41:53.535014 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:53.534952 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1aea8cf3f67682efc028bc471cd45ee9d6fa282c3c78f14b7f94defe090fde7"} err="failed to get container status \"b1aea8cf3f67682efc028bc471cd45ee9d6fa282c3c78f14b7f94defe090fde7\": rpc error: code = NotFound desc = could not find container \"b1aea8cf3f67682efc028bc471cd45ee9d6fa282c3c78f14b7f94defe090fde7\": container with ID starting with b1aea8cf3f67682efc028bc471cd45ee9d6fa282c3c78f14b7f94defe090fde7 not found: ID does not exist" Sep 17 01:41:53.535014 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:53.534976 2577 scope.go:117] "RemoveContainer" containerID="9cf6654c8deb1d20ad1f175a2cef5b48d0472f2f7e059f094e0a0551c1500ea1" Sep 17 01:41:53.535266 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:41:53.535247 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cf6654c8deb1d20ad1f175a2cef5b48d0472f2f7e059f094e0a0551c1500ea1\": container with ID starting with 9cf6654c8deb1d20ad1f175a2cef5b48d0472f2f7e059f094e0a0551c1500ea1 not found: ID does not exist" containerID="9cf6654c8deb1d20ad1f175a2cef5b48d0472f2f7e059f094e0a0551c1500ea1" Sep 17 01:41:53.535314 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:53.535273 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cf6654c8deb1d20ad1f175a2cef5b48d0472f2f7e059f094e0a0551c1500ea1"} err="failed to get container status \"9cf6654c8deb1d20ad1f175a2cef5b48d0472f2f7e059f094e0a0551c1500ea1\": rpc error: code = NotFound desc = could not find container \"9cf6654c8deb1d20ad1f175a2cef5b48d0472f2f7e059f094e0a0551c1500ea1\": container with ID starting with 9cf6654c8deb1d20ad1f175a2cef5b48d0472f2f7e059f094e0a0551c1500ea1 not found: ID does not exist" Sep 17 01:41:55.213341 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:41:55.213307 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44c791e7-bc3f-457b-8013-854a41f08da2" path="/var/lib/kubelet/pods/44c791e7-bc3f-457b-8013-854a41f08da2/volumes" Sep 17 01:42:01.963029 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:01.962989 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz"] Sep 17 01:42:01.963542 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:01.963495 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44c791e7-bc3f-457b-8013-854a41f08da2" containerName="istio-proxy" Sep 17 01:42:01.963767 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:01.963537 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="44c791e7-bc3f-457b-8013-854a41f08da2" containerName="istio-proxy" Sep 17 01:42:01.963767 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:01.963567 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0883d7ed-0c50-43b7-ae04-dbce40543a62" containerName="maas-api" Sep 17 01:42:01.963767 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:01.963575 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0883d7ed-0c50-43b7-ae04-dbce40543a62" containerName="maas-api" Sep 17 01:42:01.963767 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:01.963607 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44c791e7-bc3f-457b-8013-854a41f08da2" containerName="istio-proxy" Sep 17 01:42:01.963767 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:01.963614 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="44c791e7-bc3f-457b-8013-854a41f08da2" containerName="istio-proxy" Sep 17 01:42:01.963767 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:01.963761 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="44c791e7-bc3f-457b-8013-854a41f08da2" containerName="istio-proxy" Sep 17 01:42:01.964105 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:01.963776 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="44c791e7-bc3f-457b-8013-854a41f08da2" containerName="istio-proxy" Sep 17 01:42:01.964105 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:01.963784 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0883d7ed-0c50-43b7-ae04-dbce40543a62" containerName="maas-api" Sep 17 01:42:01.967288 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:01.967265 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:01.970399 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:01.970375 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cc98e0\"/\"kube-root-ca.crt\"" Sep 17 01:42:01.970545 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:01.970371 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cc98e0\"/\"premium-w3-cc98e0-kserve-self-signed-certs\"" Sep 17 01:42:01.970691 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:01.970432 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cc98e0\"/\"default-dockercfg-tv6kc\"" Sep 17 01:42:01.970797 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:01.970485 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cc98e0\"/\"openshift-service-ca.crt\"" Sep 17 01:42:01.974758 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:01.974733 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz"] Sep 17 01:42:02.051002 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.050961 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm7mq\" (UniqueName: \"kubernetes.io/projected/e8f10f54-d924-4598-aff2-020ddeb3d487-kube-api-access-tm7mq\") pod \"premium-w3-cc98e0-kserve-65c74f5987-szdwz\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:02.051002 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.051001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-model-cache\") pod \"premium-w3-cc98e0-kserve-65c74f5987-szdwz\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:02.051209 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.051122 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f10f54-d924-4598-aff2-020ddeb3d487-tls-certs\") pod \"premium-w3-cc98e0-kserve-65c74f5987-szdwz\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:02.051209 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.051170 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-dshm\") pod \"premium-w3-cc98e0-kserve-65c74f5987-szdwz\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:02.051283 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.051208 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-home\") pod \"premium-w3-cc98e0-kserve-65c74f5987-szdwz\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:02.051283 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.051229 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-tmp-dir\") pod \"premium-w3-cc98e0-kserve-65c74f5987-szdwz\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:02.152035 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.151992 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f10f54-d924-4598-aff2-020ddeb3d487-tls-certs\") pod \"premium-w3-cc98e0-kserve-65c74f5987-szdwz\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:02.152224 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.152052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-dshm\") pod \"premium-w3-cc98e0-kserve-65c74f5987-szdwz\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:02.152224 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.152095 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-home\") pod \"premium-w3-cc98e0-kserve-65c74f5987-szdwz\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:02.152224 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.152125 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-tmp-dir\") pod \"premium-w3-cc98e0-kserve-65c74f5987-szdwz\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:02.152373 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.152230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm7mq\" (UniqueName: \"kubernetes.io/projected/e8f10f54-d924-4598-aff2-020ddeb3d487-kube-api-access-tm7mq\") pod \"premium-w3-cc98e0-kserve-65c74f5987-szdwz\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:02.152373 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.152254 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-model-cache\") pod \"premium-w3-cc98e0-kserve-65c74f5987-szdwz\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:02.152646 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.152614 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-tmp-dir\") pod \"premium-w3-cc98e0-kserve-65c74f5987-szdwz\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:02.152646 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.152624 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-home\") pod \"premium-w3-cc98e0-kserve-65c74f5987-szdwz\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:02.152815 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.152653 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-model-cache\") pod \"premium-w3-cc98e0-kserve-65c74f5987-szdwz\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:02.154426 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.154396 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-dshm\") pod \"premium-w3-cc98e0-kserve-65c74f5987-szdwz\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:02.154619 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.154603 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f10f54-d924-4598-aff2-020ddeb3d487-tls-certs\") pod \"premium-w3-cc98e0-kserve-65c74f5987-szdwz\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:02.159512 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.159487 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm7mq\" (UniqueName: \"kubernetes.io/projected/e8f10f54-d924-4598-aff2-020ddeb3d487-kube-api-access-tm7mq\") pod \"premium-w3-cc98e0-kserve-65c74f5987-szdwz\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:02.279705 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.279626 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:02.551894 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.551864 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz"] Sep 17 01:42:02.552821 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:42:02.552794 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8f10f54_d924_4598_aff2_020ddeb3d487.slice/crio-4e48227a4243fc826e9e4a8c7c261b3b41073920c548aed7722f33dc0b209f5c WatchSource:0}: Error finding container 4e48227a4243fc826e9e4a8c7c261b3b41073920c548aed7722f33dc0b209f5c: Status 404 returned error can't find the container with id 4e48227a4243fc826e9e4a8c7c261b3b41073920c548aed7722f33dc0b209f5c Sep 17 01:42:02.568218 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:02.568188 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" event={"ID":"e8f10f54-d924-4598-aff2-020ddeb3d487","Type":"ContainerStarted","Data":"4e48227a4243fc826e9e4a8c7c261b3b41073920c548aed7722f33dc0b209f5c"} Sep 17 01:42:04.584147 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:04.584106 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" event={"ID":"e8f10f54-d924-4598-aff2-020ddeb3d487","Type":"ContainerStarted","Data":"f1827c0788d34018a9aa579e132d7de176455c9ecb2a9ddb98ca3f1239081a57"} Sep 17 01:42:04.600239 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:04.600185 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" podStartSLOduration=2.4816640469999998 podStartE2EDuration="3.600168588s" totalImagesPullingTime="1.118504541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:42:04.598658004 +0000 UTC m=+1189.833805197" watchObservedRunningTime="2026-09-17 01:42:04.600168588 +0000 UTC m=+1189.835315776" Sep 17 01:42:12.280609 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:12.280570 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:12.281136 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:12.280625 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:12.293242 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:12.293214 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:12.645031 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:12.645000 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:42:15.231888 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:15.231855 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 01:42:15.233815 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:15.233791 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 01:42:55.209461 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:55.209426 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:42:57.206736 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:42:57.206702 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:43:20.502830 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.502798 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh"] Sep 17 01:43:20.506711 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.506686 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.508893 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.508870 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-07942c61-openshift-default-dockercfg-p654q\"" Sep 17 01:43:20.517901 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.517879 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh"] Sep 17 01:43:20.627548 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.627511 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-podinfo\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.627548 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.627551 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-workload-socket\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.627807 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.627577 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-data\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.627807 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.627719 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-credential-socket\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.627807 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.627758 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-token\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.627965 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.627838 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-envoy\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.627965 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.627879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istiod-ca-cert\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.627965 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.627941 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xjwp\" (UniqueName: \"kubernetes.io/projected/82709b60-32e9-4521-b9e6-f12aa4beb1f8-kube-api-access-4xjwp\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.628083 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.627988 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-workload-certs\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.729355 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.729321 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-envoy\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.729532 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.729361 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istiod-ca-cert\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.729532 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.729414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xjwp\" (UniqueName: \"kubernetes.io/projected/82709b60-32e9-4521-b9e6-f12aa4beb1f8-kube-api-access-4xjwp\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.729532 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.729450 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-workload-certs\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.729532 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.729499 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-podinfo\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.729720 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.729572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-workload-socket\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.729720 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.729629 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-data\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.729830 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.729716 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-credential-socket\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.729830 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.729745 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-token\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.730078 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.730051 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-credential-socket\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.730161 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.730083 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-data\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.730161 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.730116 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-workload-certs\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.730285 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.730266 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-workload-socket\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.730493 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.730468 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istiod-ca-cert\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.732130 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.732093 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-envoy\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.732886 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.732864 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-podinfo\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.736425 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.736397 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xjwp\" (UniqueName: \"kubernetes.io/projected/82709b60-32e9-4521-b9e6-f12aa4beb1f8-kube-api-access-4xjwp\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.736527 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.736453 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-token\") pod \"e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:20.819258 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:20.819148 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:21.098765 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:21.098739 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh"] Sep 17 01:43:21.100011 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:43:21.099988 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82709b60_32e9_4521_b9e6_f12aa4beb1f8.slice/crio-d75085c80018b2ad7492607b23a9651cf8f01dc2121cedb73b340c6c8b4c58ff WatchSource:0}: Error finding container d75085c80018b2ad7492607b23a9651cf8f01dc2121cedb73b340c6c8b4c58ff: Status 404 returned error can't find the container with id d75085c80018b2ad7492607b23a9651cf8f01dc2121cedb73b340c6c8b4c58ff Sep 17 01:43:21.102787 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:21.102747 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:43:21.102877 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:21.102816 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:43:21.102877 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:21.102843 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:43:22.076207 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:22.076173 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" event={"ID":"82709b60-32e9-4521-b9e6-f12aa4beb1f8","Type":"ContainerStarted","Data":"cf276112326602f261060db7f11da91fcbc99284d253f6075d1b47b1a0d4be2e"} Sep 17 01:43:22.076618 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:22.076228 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" event={"ID":"82709b60-32e9-4521-b9e6-f12aa4beb1f8","Type":"ContainerStarted","Data":"d75085c80018b2ad7492607b23a9651cf8f01dc2121cedb73b340c6c8b4c58ff"} Sep 17 01:43:22.099257 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:22.099194 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" podStartSLOduration=2.099169388 podStartE2EDuration="2.099169388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:43:22.092922199 +0000 UTC m=+1267.328069388" watchObservedRunningTime="2026-09-17 01:43:22.099169388 +0000 UTC m=+1267.334316578" Sep 17 01:43:22.820122 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:22.820076 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:22.826193 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:22.826162 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:23.082946 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:23.082843 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:23.083816 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:23.083797 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:26.761626 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:26.761589 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6"] Sep 17 01:43:26.765645 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:26.765624 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" Sep 17 01:43:26.767623 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:26.767602 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-07942c61-serving-cert\"" Sep 17 01:43:26.768267 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:26.768243 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 01:43:26.768442 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:26.768419 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 01:43:26.768748 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:26.768567 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 01:43:26.768748 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:26.768702 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s7lw7\"" Sep 17 01:43:26.776168 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:26.776036 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6"] Sep 17 01:43:26.893199 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:26.893160 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-openshift-service-ca\") pod \"maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6\" (UID: \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" Sep 17 01:43:26.893370 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:26.893222 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2xpf\" (UniqueName: \"kubernetes.io/projected/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-kube-api-access-h2xpf\") pod \"maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6\" (UID: \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" Sep 17 01:43:26.893370 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:26.893310 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-maas-api-tls\") pod \"maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6\" (UID: \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" Sep 17 01:43:26.893370 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:26.893363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-metrics-certs\") pod \"maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6\" (UID: \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" Sep 17 01:43:26.994186 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:26.994145 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-maas-api-tls\") pod \"maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6\" (UID: \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" Sep 17 01:43:26.994369 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:26.994202 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-metrics-certs\") pod \"maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6\" (UID: \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" Sep 17 01:43:26.994369 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:26.994264 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-openshift-service-ca\") pod \"maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6\" (UID: \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" Sep 17 01:43:26.994369 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:26.994301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2xpf\" (UniqueName: \"kubernetes.io/projected/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-kube-api-access-h2xpf\") pod \"maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6\" (UID: \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" Sep 17 01:43:26.995029 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:26.995005 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-openshift-service-ca\") pod \"maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6\" (UID: \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" Sep 17 01:43:26.996638 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:26.996605 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-metrics-certs\") pod \"maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6\" (UID: \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" Sep 17 01:43:26.996759 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:26.996704 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-maas-api-tls\") pod \"maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6\" (UID: \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" Sep 17 01:43:27.010103 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:27.010077 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2xpf\" (UniqueName: \"kubernetes.io/projected/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-kube-api-access-h2xpf\") pod \"maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6\" (UID: \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" Sep 17 01:43:27.080455 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:27.080369 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" Sep 17 01:43:27.362129 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:27.362098 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6"] Sep 17 01:43:27.364230 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:43:27.364198 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc4ed03_686b_4e65_a4c6_f7cb9973dc49.slice/crio-b5ce9da31a137ed66d5468a32df2ebffdbe870a0fb4b313996f7dae8e5ea27f8 WatchSource:0}: Error finding container b5ce9da31a137ed66d5468a32df2ebffdbe870a0fb4b313996f7dae8e5ea27f8: Status 404 returned error can't find the container with id b5ce9da31a137ed66d5468a32df2ebffdbe870a0fb4b313996f7dae8e5ea27f8 Sep 17 01:43:27.377982 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:27.377954 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48"] Sep 17 01:43:27.382918 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:27.382900 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48" Sep 17 01:43:27.384979 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:27.384957 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-07942c61-dockercfg-qf98g\"" Sep 17 01:43:27.385089 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:27.384977 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-07942c61\"" Sep 17 01:43:27.390980 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:27.390949 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48"] Sep 17 01:43:27.499678 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:27.499640 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8c6055-c8b4-4d05-976c-089ca6cbbfed-plugins-config-volume\") pod \"payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48\" (UID: \"4f8c6055-c8b4-4d05-976c-089ca6cbbfed\") " pod="openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48" Sep 17 01:43:27.499859 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:27.499704 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4r8d\" (UniqueName: \"kubernetes.io/projected/4f8c6055-c8b4-4d05-976c-089ca6cbbfed-kube-api-access-t4r8d\") pod \"payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48\" (UID: \"4f8c6055-c8b4-4d05-976c-089ca6cbbfed\") " pod="openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48" Sep 17 01:43:27.600462 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:27.600421 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8c6055-c8b4-4d05-976c-089ca6cbbfed-plugins-config-volume\") pod \"payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48\" (UID: \"4f8c6055-c8b4-4d05-976c-089ca6cbbfed\") " pod="openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48" Sep 17 01:43:27.600637 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:27.600500 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4r8d\" (UniqueName: \"kubernetes.io/projected/4f8c6055-c8b4-4d05-976c-089ca6cbbfed-kube-api-access-t4r8d\") pod \"payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48\" (UID: \"4f8c6055-c8b4-4d05-976c-089ca6cbbfed\") " pod="openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48" Sep 17 01:43:27.601174 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:27.601155 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8c6055-c8b4-4d05-976c-089ca6cbbfed-plugins-config-volume\") pod \"payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48\" (UID: \"4f8c6055-c8b4-4d05-976c-089ca6cbbfed\") " pod="openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48" Sep 17 01:43:27.608034 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:27.608010 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4r8d\" (UniqueName: \"kubernetes.io/projected/4f8c6055-c8b4-4d05-976c-089ca6cbbfed-kube-api-access-t4r8d\") pod \"payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48\" (UID: \"4f8c6055-c8b4-4d05-976c-089ca6cbbfed\") " pod="openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48" Sep 17 01:43:27.694808 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:27.694782 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48" Sep 17 01:43:27.972094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:27.972056 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48"] Sep 17 01:43:27.973515 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:43:27.973479 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8c6055_c8b4_4d05_976c_089ca6cbbfed.slice/crio-e931bc7ccc6defd784fff758f1175243dd29b2e632b1232fbb8222139beba1ca WatchSource:0}: Error finding container e931bc7ccc6defd784fff758f1175243dd29b2e632b1232fbb8222139beba1ca: Status 404 returned error can't find the container with id e931bc7ccc6defd784fff758f1175243dd29b2e632b1232fbb8222139beba1ca Sep 17 01:43:28.118308 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:28.118266 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" event={"ID":"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49","Type":"ContainerStarted","Data":"47bd2cccaa20f143b7433ac4de3dc14eebc522091f1017c2131ca67989c976d5"} Sep 17 01:43:28.118486 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:28.118339 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" Sep 17 01:43:28.118486 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:28.118366 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" event={"ID":"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49","Type":"ContainerStarted","Data":"b5ce9da31a137ed66d5468a32df2ebffdbe870a0fb4b313996f7dae8e5ea27f8"} Sep 17 01:43:28.120573 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:28.120418 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48" event={"ID":"4f8c6055-c8b4-4d05-976c-089ca6cbbfed","Type":"ContainerStarted","Data":"457d65823efb865b2d396aeb65b8c528c2ac319630c5145fd2b922afac6d87ca"} Sep 17 01:43:28.120573 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:28.120467 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48" event={"ID":"4f8c6055-c8b4-4d05-976c-089ca6cbbfed","Type":"ContainerStarted","Data":"e931bc7ccc6defd784fff758f1175243dd29b2e632b1232fbb8222139beba1ca"} Sep 17 01:43:28.120573 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:28.120515 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48" Sep 17 01:43:28.134903 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:28.134848 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" podStartSLOduration=1.8258204089999999 podStartE2EDuration="2.134833851s" totalImagesPullingTime="309.013442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:43:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:43:28.13285986 +0000 UTC m=+1273.368007049" watchObservedRunningTime="2026-09-17 01:43:28.134833851 +0000 UTC m=+1273.369981038" Sep 17 01:43:28.147965 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:28.147895 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48" podStartSLOduration=1.147878721 podStartE2EDuration="1.147878721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:43:28.147033585 +0000 UTC m=+1273.382180775" watchObservedRunningTime="2026-09-17 01:43:28.147878721 +0000 UTC m=+1273.383025908" Sep 17 01:43:34.135211 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:34.135178 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" Sep 17 01:43:39.131566 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:39.131532 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48" Sep 17 01:43:51.294300 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:51.294261 2577 generic.go:356] "Generic (PLEG): container finished" podID="82709b60-32e9-4521-b9e6-f12aa4beb1f8" containerID="cf276112326602f261060db7f11da91fcbc99284d253f6075d1b47b1a0d4be2e" exitCode=0 Sep 17 01:43:51.294885 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:51.294309 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" event={"ID":"82709b60-32e9-4521-b9e6-f12aa4beb1f8","Type":"ContainerDied","Data":"cf276112326602f261060db7f11da91fcbc99284d253f6075d1b47b1a0d4be2e"} Sep 17 01:43:51.294885 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:51.294876 2577 scope.go:117] "RemoveContainer" containerID="cf276112326602f261060db7f11da91fcbc99284d253f6075d1b47b1a0d4be2e" Sep 17 01:43:51.296263 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:51.296224 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:43:51.296351 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:51.296319 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:43:51.296394 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:51.296363 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:43:52.302869 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:52.302825 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" event={"ID":"82709b60-32e9-4521-b9e6-f12aa4beb1f8","Type":"ContainerStarted","Data":"34c1f71d1896f8d38776204a4ed9e7c2a115222c61838ffc4556da7a4941eb25"} Sep 17 01:43:52.821451 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:52.821415 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:52.821856 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:52.821828 2577 patch_prober.go:28] interesting pod/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 17 01:43:52.822080 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:52.822053 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" podUID="82709b60-32e9-4521-b9e6-f12aa4beb1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 17 01:43:53.083849 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:53.083754 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:53.819659 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:53.819619 2577 patch_prober.go:28] interesting pod/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 17 01:43:53.820178 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:53.819689 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" podUID="82709b60-32e9-4521-b9e6-f12aa4beb1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 17 01:43:54.820925 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:54.820432 2577 patch_prober.go:28] interesting pod/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 17 01:43:54.820925 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:54.820510 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" podUID="82709b60-32e9-4521-b9e6-f12aa4beb1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 17 01:43:55.820713 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:55.820661 2577 patch_prober.go:28] interesting pod/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 17 01:43:55.820943 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:55.820740 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" podUID="82709b60-32e9-4521-b9e6-f12aa4beb1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 17 01:43:56.819697 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:56.819654 2577 patch_prober.go:28] interesting pod/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 17 01:43:56.820340 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:56.820302 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" podUID="82709b60-32e9-4521-b9e6-f12aa4beb1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 17 01:43:57.824342 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:57.824311 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:57.825549 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:57.825525 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:43:59.206620 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:43:59.206588 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:44:04.206009 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:04.205975 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:44:06.282210 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.282174 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6"] Sep 17 01:44:06.282678 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.282434 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" podUID="7cc4ed03-686b-4e65-a4c6-f7cb9973dc49" containerName="maas-api" containerID="cri-o://47bd2cccaa20f143b7433ac4de3dc14eebc522091f1017c2131ca67989c976d5" gracePeriod=30 Sep 17 01:44:06.326489 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.326446 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48"] Sep 17 01:44:06.326734 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.326700 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48" podUID="4f8c6055-c8b4-4d05-976c-089ca6cbbfed" containerName="payload-processing" containerID="cri-o://457d65823efb865b2d396aeb65b8c528c2ac319630c5145fd2b922afac6d87ca" gracePeriod=30 Sep 17 01:44:06.406674 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.406638 2577 generic.go:356] "Generic (PLEG): container finished" podID="7cc4ed03-686b-4e65-a4c6-f7cb9973dc49" containerID="47bd2cccaa20f143b7433ac4de3dc14eebc522091f1017c2131ca67989c976d5" exitCode=0 Sep 17 01:44:06.406810 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.406711 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" event={"ID":"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49","Type":"ContainerDied","Data":"47bd2cccaa20f143b7433ac4de3dc14eebc522091f1017c2131ca67989c976d5"} Sep 17 01:44:06.618891 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.618850 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" Sep 17 01:44:06.642074 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.642041 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48" Sep 17 01:44:06.800491 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.800404 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8c6055-c8b4-4d05-976c-089ca6cbbfed-plugins-config-volume\") pod \"4f8c6055-c8b4-4d05-976c-089ca6cbbfed\" (UID: \"4f8c6055-c8b4-4d05-976c-089ca6cbbfed\") " Sep 17 01:44:06.800491 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.800477 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-metrics-certs\") pod \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\" (UID: \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\") " Sep 17 01:44:06.800723 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.800520 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4r8d\" (UniqueName: \"kubernetes.io/projected/4f8c6055-c8b4-4d05-976c-089ca6cbbfed-kube-api-access-t4r8d\") pod \"4f8c6055-c8b4-4d05-976c-089ca6cbbfed\" (UID: \"4f8c6055-c8b4-4d05-976c-089ca6cbbfed\") " Sep 17 01:44:06.800723 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.800561 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-openshift-service-ca\") pod \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\" (UID: \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\") " Sep 17 01:44:06.800723 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.800580 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2xpf\" (UniqueName: \"kubernetes.io/projected/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-kube-api-access-h2xpf\") pod \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\" (UID: \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\") " Sep 17 01:44:06.800723 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.800608 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-maas-api-tls\") pod \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\" (UID: \"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49\") " Sep 17 01:44:06.800979 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.800865 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8c6055-c8b4-4d05-976c-089ca6cbbfed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f8c6055-c8b4-4d05-976c-089ca6cbbfed" (UID: "4f8c6055-c8b4-4d05-976c-089ca6cbbfed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:44:06.800979 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.800886 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7cc4ed03-686b-4e65-a4c6-f7cb9973dc49" (UID: "7cc4ed03-686b-4e65-a4c6-f7cb9973dc49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:44:06.801085 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.801010 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8c6055-c8b4-4d05-976c-089ca6cbbfed-plugins-config-volume\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:06.801085 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.801031 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-openshift-service-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:06.802772 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.802740 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7cc4ed03-686b-4e65-a4c6-f7cb9973dc49" (UID: "7cc4ed03-686b-4e65-a4c6-f7cb9973dc49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:44:06.802887 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.802767 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-kube-api-access-h2xpf" (OuterVolumeSpecName: "kube-api-access-h2xpf") pod "7cc4ed03-686b-4e65-a4c6-f7cb9973dc49" (UID: "7cc4ed03-686b-4e65-a4c6-f7cb9973dc49"). InnerVolumeSpecName "kube-api-access-h2xpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:44:06.803096 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.803074 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7cc4ed03-686b-4e65-a4c6-f7cb9973dc49" (UID: "7cc4ed03-686b-4e65-a4c6-f7cb9973dc49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:44:06.803167 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.803144 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8c6055-c8b4-4d05-976c-089ca6cbbfed-kube-api-access-t4r8d" (OuterVolumeSpecName: "kube-api-access-t4r8d") pod "4f8c6055-c8b4-4d05-976c-089ca6cbbfed" (UID: "4f8c6055-c8b4-4d05-976c-089ca6cbbfed"). InnerVolumeSpecName "kube-api-access-t4r8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:44:06.901685 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.901645 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2xpf\" (UniqueName: \"kubernetes.io/projected/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-kube-api-access-h2xpf\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:06.901685 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.901680 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-maas-api-tls\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:06.901685 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.901693 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49-metrics-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:06.901915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:06.901706 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4r8d\" (UniqueName: \"kubernetes.io/projected/4f8c6055-c8b4-4d05-976c-089ca6cbbfed-kube-api-access-t4r8d\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:07.414111 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:07.414069 2577 generic.go:356] "Generic (PLEG): container finished" podID="4f8c6055-c8b4-4d05-976c-089ca6cbbfed" containerID="457d65823efb865b2d396aeb65b8c528c2ac319630c5145fd2b922afac6d87ca" exitCode=0 Sep 17 01:44:07.414546 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:07.414120 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48" Sep 17 01:44:07.414546 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:07.414156 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48" event={"ID":"4f8c6055-c8b4-4d05-976c-089ca6cbbfed","Type":"ContainerDied","Data":"457d65823efb865b2d396aeb65b8c528c2ac319630c5145fd2b922afac6d87ca"} Sep 17 01:44:07.414546 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:07.414213 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48" event={"ID":"4f8c6055-c8b4-4d05-976c-089ca6cbbfed","Type":"ContainerDied","Data":"e931bc7ccc6defd784fff758f1175243dd29b2e632b1232fbb8222139beba1ca"} Sep 17 01:44:07.414546 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:07.414235 2577 scope.go:117] "RemoveContainer" containerID="457d65823efb865b2d396aeb65b8c528c2ac319630c5145fd2b922afac6d87ca" Sep 17 01:44:07.415713 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:07.415693 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" event={"ID":"7cc4ed03-686b-4e65-a4c6-f7cb9973dc49","Type":"ContainerDied","Data":"b5ce9da31a137ed66d5468a32df2ebffdbe870a0fb4b313996f7dae8e5ea27f8"} Sep 17 01:44:07.415800 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:07.415729 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6" Sep 17 01:44:07.426201 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:07.426179 2577 scope.go:117] "RemoveContainer" containerID="457d65823efb865b2d396aeb65b8c528c2ac319630c5145fd2b922afac6d87ca" Sep 17 01:44:07.426519 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:44:07.426497 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457d65823efb865b2d396aeb65b8c528c2ac319630c5145fd2b922afac6d87ca\": container with ID starting with 457d65823efb865b2d396aeb65b8c528c2ac319630c5145fd2b922afac6d87ca not found: ID does not exist" containerID="457d65823efb865b2d396aeb65b8c528c2ac319630c5145fd2b922afac6d87ca" Sep 17 01:44:07.426601 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:07.426531 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457d65823efb865b2d396aeb65b8c528c2ac319630c5145fd2b922afac6d87ca"} err="failed to get container status \"457d65823efb865b2d396aeb65b8c528c2ac319630c5145fd2b922afac6d87ca\": rpc error: code = NotFound desc = could not find container \"457d65823efb865b2d396aeb65b8c528c2ac319630c5145fd2b922afac6d87ca\": container with ID starting with 457d65823efb865b2d396aeb65b8c528c2ac319630c5145fd2b922afac6d87ca not found: ID does not exist" Sep 17 01:44:07.426601 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:07.426554 2577 scope.go:117] "RemoveContainer" containerID="47bd2cccaa20f143b7433ac4de3dc14eebc522091f1017c2131ca67989c976d5" Sep 17 01:44:07.432712 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:07.432681 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48"] Sep 17 01:44:07.436519 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:07.436496 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-07942c61-784b8b7dd9-ggx48"] Sep 17 01:44:07.445592 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:07.445563 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6"] Sep 17 01:44:07.448359 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:07.448337 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07942c61-5fdc4d5f97-ss7t6"] Sep 17 01:44:09.212820 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:09.212788 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f8c6055-c8b4-4d05-976c-089ca6cbbfed" path="/var/lib/kubelet/pods/4f8c6055-c8b4-4d05-976c-089ca6cbbfed/volumes" Sep 17 01:44:09.213258 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:09.213243 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cc4ed03-686b-4e65-a4c6-f7cb9973dc49" path="/var/lib/kubelet/pods/7cc4ed03-686b-4e65-a4c6-f7cb9973dc49/volumes" Sep 17 01:44:12.605401 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:12.605367 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh"] Sep 17 01:44:12.605870 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:12.605683 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" podUID="82709b60-32e9-4521-b9e6-f12aa4beb1f8" containerName="istio-proxy" containerID="cri-o://34c1f71d1896f8d38776204a4ed9e7c2a115222c61838ffc4556da7a4941eb25" gracePeriod=30 Sep 17 01:44:12.827535 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:12.827497 2577 patch_prober.go:28] interesting pod/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Sep 17 01:44:12.827703 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:12.827558 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" podUID="82709b60-32e9-4521-b9e6-f12aa4beb1f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 01:44:17.869806 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.869777 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:44:17.908222 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.908134 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istiod-ca-cert\") pod \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " Sep 17 01:44:17.908222 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.908178 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-podinfo\") pod \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " Sep 17 01:44:17.908222 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.908195 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-token\") pod \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " Sep 17 01:44:17.908222 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.908224 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-data\") pod \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " Sep 17 01:44:17.908548 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.908390 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-workload-certs\") pod \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " Sep 17 01:44:17.908548 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.908469 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-workload-socket\") pod \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " Sep 17 01:44:17.908548 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.908497 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-credential-socket\") pod \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " Sep 17 01:44:17.908548 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.908531 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-envoy\") pod \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " Sep 17 01:44:17.908769 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.908548 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-data" (OuterVolumeSpecName: "istio-data") pod "82709b60-32e9-4521-b9e6-f12aa4beb1f8" (UID: "82709b60-32e9-4521-b9e6-f12aa4beb1f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:44:17.908769 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.908558 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xjwp\" (UniqueName: \"kubernetes.io/projected/82709b60-32e9-4521-b9e6-f12aa4beb1f8-kube-api-access-4xjwp\") pod \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\" (UID: \"82709b60-32e9-4521-b9e6-f12aa4beb1f8\") " Sep 17 01:44:17.908769 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.908725 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "82709b60-32e9-4521-b9e6-f12aa4beb1f8" (UID: "82709b60-32e9-4521-b9e6-f12aa4beb1f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:44:17.908973 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.908772 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "82709b60-32e9-4521-b9e6-f12aa4beb1f8" (UID: "82709b60-32e9-4521-b9e6-f12aa4beb1f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:44:17.908973 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.908838 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "82709b60-32e9-4521-b9e6-f12aa4beb1f8" (UID: "82709b60-32e9-4521-b9e6-f12aa4beb1f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:44:17.909095 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.908997 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istiod-ca-cert\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:17.909095 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.909018 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-data\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:17.909095 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.909033 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-workload-socket\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:17.909095 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.909046 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-credential-socket\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:17.909095 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.909008 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "82709b60-32e9-4521-b9e6-f12aa4beb1f8" (UID: "82709b60-32e9-4521-b9e6-f12aa4beb1f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:44:17.910925 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.910875 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "82709b60-32e9-4521-b9e6-f12aa4beb1f8" (UID: "82709b60-32e9-4521-b9e6-f12aa4beb1f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 01:44:17.911090 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.911068 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-token" (OuterVolumeSpecName: "istio-token") pod "82709b60-32e9-4521-b9e6-f12aa4beb1f8" (UID: "82709b60-32e9-4521-b9e6-f12aa4beb1f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:44:17.911168 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.911132 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82709b60-32e9-4521-b9e6-f12aa4beb1f8-kube-api-access-4xjwp" (OuterVolumeSpecName: "kube-api-access-4xjwp") pod "82709b60-32e9-4521-b9e6-f12aa4beb1f8" (UID: "82709b60-32e9-4521-b9e6-f12aa4beb1f8"). InnerVolumeSpecName "kube-api-access-4xjwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:44:17.911240 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:17.911175 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "82709b60-32e9-4521-b9e6-f12aa4beb1f8" (UID: "82709b60-32e9-4521-b9e6-f12aa4beb1f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:44:18.010517 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:18.010477 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-envoy\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:18.010517 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:18.010513 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xjwp\" (UniqueName: \"kubernetes.io/projected/82709b60-32e9-4521-b9e6-f12aa4beb1f8-kube-api-access-4xjwp\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:18.010716 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:18.010526 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-podinfo\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:18.010716 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:18.010535 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82709b60-32e9-4521-b9e6-f12aa4beb1f8-istio-token\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:18.010716 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:18.010544 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82709b60-32e9-4521-b9e6-f12aa4beb1f8-workload-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:18.499048 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:18.499014 2577 generic.go:356] "Generic (PLEG): container finished" podID="82709b60-32e9-4521-b9e6-f12aa4beb1f8" containerID="34c1f71d1896f8d38776204a4ed9e7c2a115222c61838ffc4556da7a4941eb25" exitCode=0 Sep 17 01:44:18.499195 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:18.499066 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" Sep 17 01:44:18.499195 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:18.499068 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" event={"ID":"82709b60-32e9-4521-b9e6-f12aa4beb1f8","Type":"ContainerDied","Data":"34c1f71d1896f8d38776204a4ed9e7c2a115222c61838ffc4556da7a4941eb25"} Sep 17 01:44:18.499293 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:18.499196 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh" event={"ID":"82709b60-32e9-4521-b9e6-f12aa4beb1f8","Type":"ContainerDied","Data":"d75085c80018b2ad7492607b23a9651cf8f01dc2121cedb73b340c6c8b4c58ff"} Sep 17 01:44:18.499293 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:18.499223 2577 scope.go:117] "RemoveContainer" containerID="34c1f71d1896f8d38776204a4ed9e7c2a115222c61838ffc4556da7a4941eb25" Sep 17 01:44:18.509516 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:18.509499 2577 scope.go:117] "RemoveContainer" containerID="cf276112326602f261060db7f11da91fcbc99284d253f6075d1b47b1a0d4be2e" Sep 17 01:44:18.518507 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:18.518488 2577 scope.go:117] "RemoveContainer" containerID="34c1f71d1896f8d38776204a4ed9e7c2a115222c61838ffc4556da7a4941eb25" Sep 17 01:44:18.518816 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:44:18.518792 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34c1f71d1896f8d38776204a4ed9e7c2a115222c61838ffc4556da7a4941eb25\": container with ID starting with 34c1f71d1896f8d38776204a4ed9e7c2a115222c61838ffc4556da7a4941eb25 not found: ID does not exist" containerID="34c1f71d1896f8d38776204a4ed9e7c2a115222c61838ffc4556da7a4941eb25" Sep 17 01:44:18.518892 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:18.518825 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34c1f71d1896f8d38776204a4ed9e7c2a115222c61838ffc4556da7a4941eb25"} err="failed to get container status \"34c1f71d1896f8d38776204a4ed9e7c2a115222c61838ffc4556da7a4941eb25\": rpc error: code = NotFound desc = could not find container \"34c1f71d1896f8d38776204a4ed9e7c2a115222c61838ffc4556da7a4941eb25\": container with ID starting with 34c1f71d1896f8d38776204a4ed9e7c2a115222c61838ffc4556da7a4941eb25 not found: ID does not exist" Sep 17 01:44:18.518892 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:18.518846 2577 scope.go:117] "RemoveContainer" containerID="cf276112326602f261060db7f11da91fcbc99284d253f6075d1b47b1a0d4be2e" Sep 17 01:44:18.519148 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:44:18.519129 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf276112326602f261060db7f11da91fcbc99284d253f6075d1b47b1a0d4be2e\": container with ID starting with cf276112326602f261060db7f11da91fcbc99284d253f6075d1b47b1a0d4be2e not found: ID does not exist" containerID="cf276112326602f261060db7f11da91fcbc99284d253f6075d1b47b1a0d4be2e" Sep 17 01:44:18.519199 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:18.519157 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf276112326602f261060db7f11da91fcbc99284d253f6075d1b47b1a0d4be2e"} err="failed to get container status \"cf276112326602f261060db7f11da91fcbc99284d253f6075d1b47b1a0d4be2e\": rpc error: code = NotFound desc = could not find container \"cf276112326602f261060db7f11da91fcbc99284d253f6075d1b47b1a0d4be2e\": container with ID starting with cf276112326602f261060db7f11da91fcbc99284d253f6075d1b47b1a0d4be2e not found: ID does not exist" Sep 17 01:44:18.521713 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:18.521685 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh"] Sep 17 01:44:18.524406 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:18.524383 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-07942c61-openshift-default-84584dcf44-7kvqh"] Sep 17 01:44:19.213321 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:19.213288 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82709b60-32e9-4521-b9e6-f12aa4beb1f8" path="/var/lib/kubelet/pods/82709b60-32e9-4521-b9e6-f12aa4beb1f8/volumes" Sep 17 01:44:24.512896 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.512857 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8d689-bjvw4"] Sep 17 01:44:24.513590 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.513568 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc4ed03-686b-4e65-a4c6-f7cb9973dc49" containerName="maas-api" Sep 17 01:44:24.513685 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.513592 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc4ed03-686b-4e65-a4c6-f7cb9973dc49" containerName="maas-api" Sep 17 01:44:24.513685 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.513606 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82709b60-32e9-4521-b9e6-f12aa4beb1f8" containerName="istio-proxy" Sep 17 01:44:24.513685 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.513614 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="82709b60-32e9-4521-b9e6-f12aa4beb1f8" containerName="istio-proxy" Sep 17 01:44:24.513685 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.513630 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82709b60-32e9-4521-b9e6-f12aa4beb1f8" containerName="istio-proxy" Sep 17 01:44:24.513685 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.513640 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="82709b60-32e9-4521-b9e6-f12aa4beb1f8" containerName="istio-proxy" Sep 17 01:44:24.513685 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.513652 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f8c6055-c8b4-4d05-976c-089ca6cbbfed" containerName="payload-processing" Sep 17 01:44:24.513685 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.513661 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f8c6055-c8b4-4d05-976c-089ca6cbbfed" containerName="payload-processing" Sep 17 01:44:24.514067 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.513796 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f8c6055-c8b4-4d05-976c-089ca6cbbfed" containerName="payload-processing" Sep 17 01:44:24.514067 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.513815 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="82709b60-32e9-4521-b9e6-f12aa4beb1f8" containerName="istio-proxy" Sep 17 01:44:24.514067 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.513826 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cc4ed03-686b-4e65-a4c6-f7cb9973dc49" containerName="maas-api" Sep 17 01:44:24.518858 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.518833 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8d689-bjvw4" Sep 17 01:44:24.521094 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.520912 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 01:44:24.521484 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.521467 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r9tdk\"" Sep 17 01:44:24.523938 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.523908 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8d689-bjvw4"] Sep 17 01:44:24.571201 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.571172 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc10e4e-6f80-4078-86e2-5dde809978ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d8d689-bjvw4\" (UID: \"1fc10e4e-6f80-4078-86e2-5dde809978ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8d689-bjvw4" Sep 17 01:44:24.672391 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.672343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc10e4e-6f80-4078-86e2-5dde809978ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d8d689-bjvw4\" (UID: \"1fc10e4e-6f80-4078-86e2-5dde809978ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8d689-bjvw4" Sep 17 01:44:24.672978 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.672954 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc10e4e-6f80-4078-86e2-5dde809978ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d8d689-bjvw4\" (UID: \"1fc10e4e-6f80-4078-86e2-5dde809978ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8d689-bjvw4" Sep 17 01:44:24.832486 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:24.832395 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8d689-bjvw4" Sep 17 01:44:25.105500 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:25.105469 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8d689-bjvw4"] Sep 17 01:44:25.106638 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:44:25.106614 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc10e4e_6f80_4078_86e2_5dde809978ec.slice/crio-33ecded5782e2b0b2e45a7c14075c0e2112a219d89476bb7cb0622b5840fd23b WatchSource:0}: Error finding container 33ecded5782e2b0b2e45a7c14075c0e2112a219d89476bb7cb0622b5840fd23b: Status 404 returned error can't find the container with id 33ecded5782e2b0b2e45a7c14075c0e2112a219d89476bb7cb0622b5840fd23b Sep 17 01:44:25.550402 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:25.550306 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8d689-bjvw4" event={"ID":"1fc10e4e-6f80-4078-86e2-5dde809978ec","Type":"ContainerStarted","Data":"33ecded5782e2b0b2e45a7c14075c0e2112a219d89476bb7cb0622b5840fd23b"} Sep 17 01:44:26.566132 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.566092 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk"] Sep 17 01:44:26.566954 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.566920 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="82709b60-32e9-4521-b9e6-f12aa4beb1f8" containerName="istio-proxy" Sep 17 01:44:26.571424 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.571379 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.576222 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.575245 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-474b22-openshift-default-dockercfg-vxvnk\"" Sep 17 01:44:26.583016 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.582985 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk"] Sep 17 01:44:26.694243 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.694199 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg2nj\" (UniqueName: \"kubernetes.io/projected/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-kube-api-access-wg2nj\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.694243 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.694250 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-credential-socket\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.694508 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.694274 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-podinfo\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.694508 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.694308 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-workload-socket\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.694508 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.694342 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-token\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.694508 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.694364 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istiod-ca-cert\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.694508 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.694392 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-data\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.694508 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.694491 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-workload-certs\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.694749 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.694575 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-envoy\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.795838 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.795798 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-envoy\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.796009 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.795970 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg2nj\" (UniqueName: \"kubernetes.io/projected/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-kube-api-access-wg2nj\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.796084 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.796010 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-credential-socket\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.796084 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.796037 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-podinfo\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.796084 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.796074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-workload-socket\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.796252 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.796117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-token\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.796252 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.796140 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istiod-ca-cert\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.796252 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.796167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-data\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.796252 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.796209 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-workload-certs\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.796591 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.796561 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-workload-certs\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.796680 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.796603 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-workload-socket\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.796775 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.796752 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-credential-socket\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.796838 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.796804 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-data\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.797307 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.797282 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istiod-ca-cert\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.798616 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.798595 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-envoy\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.798952 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.798917 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-podinfo\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.803475 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.803451 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-token\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.803698 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.803679 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg2nj\" (UniqueName: \"kubernetes.io/projected/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-kube-api-access-wg2nj\") pod \"e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:26.889303 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:26.889269 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:27.173992 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:27.173965 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk"] Sep 17 01:44:27.605747 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:44:27.605714 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5b4aa7_9127_4b16_ba65_5f2bd797c338.slice/crio-42769eea290fc6daaf43c0258de94d57f16bd451e55364434dc4f7ec422d7c8c WatchSource:0}: Error finding container 42769eea290fc6daaf43c0258de94d57f16bd451e55364434dc4f7ec422d7c8c: Status 404 returned error can't find the container with id 42769eea290fc6daaf43c0258de94d57f16bd451e55364434dc4f7ec422d7c8c Sep 17 01:44:27.608067 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:27.608025 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:44:27.608196 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:27.608123 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:44:27.608196 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:27.608168 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:44:28.574592 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:28.574558 2577 generic.go:356] "Generic (PLEG): container finished" podID="1fc10e4e-6f80-4078-86e2-5dde809978ec" containerID="c1f253dd51f17e5a8d2e6756d749f70d0095a6a3dac3efcca7bfd64ff3b22eb1" exitCode=0 Sep 17 01:44:28.574787 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:28.574644 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8d689-bjvw4" event={"ID":"1fc10e4e-6f80-4078-86e2-5dde809978ec","Type":"ContainerDied","Data":"c1f253dd51f17e5a8d2e6756d749f70d0095a6a3dac3efcca7bfd64ff3b22eb1"} Sep 17 01:44:28.576377 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:28.576349 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" event={"ID":"8b5b4aa7-9127-4b16-ba65-5f2bd797c338","Type":"ContainerStarted","Data":"61b2576993de2535762d56784f07046d65f95d40df9f5f100d9a1c1839c10b94"} Sep 17 01:44:28.576483 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:28.576397 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" event={"ID":"8b5b4aa7-9127-4b16-ba65-5f2bd797c338","Type":"ContainerStarted","Data":"42769eea290fc6daaf43c0258de94d57f16bd451e55364434dc4f7ec422d7c8c"} Sep 17 01:44:28.607010 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:28.606958 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" podStartSLOduration=2.606920643 podStartE2EDuration="2.606920643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:44:28.603228081 +0000 UTC m=+1333.838375269" watchObservedRunningTime="2026-09-17 01:44:28.606920643 +0000 UTC m=+1333.842067831" Sep 17 01:44:28.890263 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:28.890220 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:28.895674 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:28.895646 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:29.582992 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:29.582956 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:29.583859 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:29.583840 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:44:29.726010 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:29.725981 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8d689-bjvw4" Sep 17 01:44:29.829563 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:29.829526 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc10e4e-6f80-4078-86e2-5dde809978ec-maas-api-service-ca\") pod \"1fc10e4e-6f80-4078-86e2-5dde809978ec\" (UID: \"1fc10e4e-6f80-4078-86e2-5dde809978ec\") " Sep 17 01:44:29.829870 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:29.829848 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc10e4e-6f80-4078-86e2-5dde809978ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1fc10e4e-6f80-4078-86e2-5dde809978ec" (UID: "1fc10e4e-6f80-4078-86e2-5dde809978ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:44:29.930365 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:29.930332 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc10e4e-6f80-4078-86e2-5dde809978ec-maas-api-service-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:30.589880 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:30.589844 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8d689-bjvw4" Sep 17 01:44:30.590056 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:30.589876 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8d689-bjvw4" event={"ID":"1fc10e4e-6f80-4078-86e2-5dde809978ec","Type":"ContainerDied","Data":"33ecded5782e2b0b2e45a7c14075c0e2112a219d89476bb7cb0622b5840fd23b"} Sep 17 01:44:30.590056 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:30.589952 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33ecded5782e2b0b2e45a7c14075c0e2112a219d89476bb7cb0622b5840fd23b" Sep 17 01:44:33.357421 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.357384 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55"] Sep 17 01:44:33.358189 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.358161 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fc10e4e-6f80-4078-86e2-5dde809978ec" containerName="revoke-keys" Sep 17 01:44:33.358189 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.358190 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fc10e4e-6f80-4078-86e2-5dde809978ec" containerName="revoke-keys" Sep 17 01:44:33.358383 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.358318 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fc10e4e-6f80-4078-86e2-5dde809978ec" containerName="revoke-keys" Sep 17 01:44:33.362020 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.361996 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" Sep 17 01:44:33.364183 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.364158 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-474b22-serving-cert\"" Sep 17 01:44:33.364309 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.364203 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 01:44:33.364309 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.364220 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 01:44:33.364897 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.364876 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s7lw7\"" Sep 17 01:44:33.364897 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.364887 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 01:44:33.373111 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.373085 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55"] Sep 17 01:44:33.465967 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.465841 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8385561-69c6-4247-ba38-bfc52153b6e4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55\" (UID: \"e8385561-69c6-4247-ba38-bfc52153b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" Sep 17 01:44:33.465967 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.465922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8385561-69c6-4247-ba38-bfc52153b6e4-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55\" (UID: \"e8385561-69c6-4247-ba38-bfc52153b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" Sep 17 01:44:33.466258 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.466052 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8385561-69c6-4247-ba38-bfc52153b6e4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55\" (UID: \"e8385561-69c6-4247-ba38-bfc52153b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" Sep 17 01:44:33.466258 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.466103 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxqpt\" (UniqueName: \"kubernetes.io/projected/e8385561-69c6-4247-ba38-bfc52153b6e4-kube-api-access-lxqpt\") pod \"maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55\" (UID: \"e8385561-69c6-4247-ba38-bfc52153b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" Sep 17 01:44:33.567067 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.567026 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8385561-69c6-4247-ba38-bfc52153b6e4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55\" (UID: \"e8385561-69c6-4247-ba38-bfc52153b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" Sep 17 01:44:33.567237 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.567080 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxqpt\" (UniqueName: \"kubernetes.io/projected/e8385561-69c6-4247-ba38-bfc52153b6e4-kube-api-access-lxqpt\") pod \"maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55\" (UID: \"e8385561-69c6-4247-ba38-bfc52153b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" Sep 17 01:44:33.567237 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.567147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8385561-69c6-4247-ba38-bfc52153b6e4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55\" (UID: \"e8385561-69c6-4247-ba38-bfc52153b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" Sep 17 01:44:33.567237 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.567187 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8385561-69c6-4247-ba38-bfc52153b6e4-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55\" (UID: \"e8385561-69c6-4247-ba38-bfc52153b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" Sep 17 01:44:33.567237 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:44:33.567192 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-serving-cert: secret "maas-api-e2e-shared-a-w1-474b22-serving-cert" not found Sep 17 01:44:33.567385 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:44:33.567263 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8385561-69c6-4247-ba38-bfc52153b6e4-maas-api-tls podName:e8385561-69c6-4247-ba38-bfc52153b6e4 nodeName:}" failed. No retries permitted until 2026-09-17 01:44:34.067242958 +0000 UTC m=+1339.302390144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e8385561-69c6-4247-ba38-bfc52153b6e4-maas-api-tls") pod "maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" (UID: "e8385561-69c6-4247-ba38-bfc52153b6e4") : secret "maas-api-e2e-shared-a-w1-474b22-serving-cert" not found Sep 17 01:44:33.567902 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.567881 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8385561-69c6-4247-ba38-bfc52153b6e4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55\" (UID: \"e8385561-69c6-4247-ba38-bfc52153b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" Sep 17 01:44:33.569674 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.569650 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8385561-69c6-4247-ba38-bfc52153b6e4-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55\" (UID: \"e8385561-69c6-4247-ba38-bfc52153b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" Sep 17 01:44:33.575748 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.575725 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxqpt\" (UniqueName: \"kubernetes.io/projected/e8385561-69c6-4247-ba38-bfc52153b6e4-kube-api-access-lxqpt\") pod \"maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55\" (UID: \"e8385561-69c6-4247-ba38-bfc52153b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" Sep 17 01:44:33.966759 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.966720 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6"] Sep 17 01:44:33.970729 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.970709 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6" Sep 17 01:44:33.972847 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.972830 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-474b22\"" Sep 17 01:44:33.972952 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.972870 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-474b22-dockercfg-q7scn\"" Sep 17 01:44:33.979908 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:33.979882 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6"] Sep 17 01:44:34.071809 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:34.071769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8385561-69c6-4247-ba38-bfc52153b6e4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55\" (UID: \"e8385561-69c6-4247-ba38-bfc52153b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" Sep 17 01:44:34.072026 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:34.071823 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnmn7\" (UniqueName: \"kubernetes.io/projected/e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2-kube-api-access-lnmn7\") pod \"payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6\" (UID: \"e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6" Sep 17 01:44:34.072026 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:34.071856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6\" (UID: \"e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6" Sep 17 01:44:34.074269 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:34.074250 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8385561-69c6-4247-ba38-bfc52153b6e4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55\" (UID: \"e8385561-69c6-4247-ba38-bfc52153b6e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" Sep 17 01:44:34.172708 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:34.172677 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmn7\" (UniqueName: \"kubernetes.io/projected/e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2-kube-api-access-lnmn7\") pod \"payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6\" (UID: \"e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6" Sep 17 01:44:34.172876 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:34.172722 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6\" (UID: \"e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6" Sep 17 01:44:34.173414 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:34.173393 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6\" (UID: \"e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6" Sep 17 01:44:34.182615 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:34.182580 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnmn7\" (UniqueName: \"kubernetes.io/projected/e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2-kube-api-access-lnmn7\") pod \"payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6\" (UID: \"e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6" Sep 17 01:44:34.276490 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:34.276408 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" Sep 17 01:44:34.284731 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:34.284703 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6" Sep 17 01:44:34.562260 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:34.562195 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55"] Sep 17 01:44:34.562989 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:44:34.562961 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8385561_69c6_4247_ba38_bfc52153b6e4.slice/crio-695564df018e0ff507f66b2d6123bb6dfd5b9b305c0c3e388e9d9ff03d9fcfbb WatchSource:0}: Error finding container 695564df018e0ff507f66b2d6123bb6dfd5b9b305c0c3e388e9d9ff03d9fcfbb: Status 404 returned error can't find the container with id 695564df018e0ff507f66b2d6123bb6dfd5b9b305c0c3e388e9d9ff03d9fcfbb Sep 17 01:44:34.564491 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:44:34.564467 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96f0a4e_95bc_4dbb_bf5c_7c7e801d1cd2.slice/crio-5d5010dd22b408c389307a0aba7012e2584ac0aeed42e779f519b455ad4bd1c1 WatchSource:0}: Error finding container 5d5010dd22b408c389307a0aba7012e2584ac0aeed42e779f519b455ad4bd1c1: Status 404 returned error can't find the container with id 5d5010dd22b408c389307a0aba7012e2584ac0aeed42e779f519b455ad4bd1c1 Sep 17 01:44:34.565785 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:34.565764 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6"] Sep 17 01:44:34.617642 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:34.617604 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" event={"ID":"e8385561-69c6-4247-ba38-bfc52153b6e4","Type":"ContainerStarted","Data":"695564df018e0ff507f66b2d6123bb6dfd5b9b305c0c3e388e9d9ff03d9fcfbb"} Sep 17 01:44:34.619101 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:34.619077 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6" event={"ID":"e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2","Type":"ContainerStarted","Data":"5d5010dd22b408c389307a0aba7012e2584ac0aeed42e779f519b455ad4bd1c1"} Sep 17 01:44:35.630679 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:35.630637 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" event={"ID":"e8385561-69c6-4247-ba38-bfc52153b6e4","Type":"ContainerStarted","Data":"d172014a50569c1368286af9f2cdc916fb06b5d21e118849d09fcfd2122e4d2b"} Sep 17 01:44:35.631158 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:35.630761 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" Sep 17 01:44:35.632597 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:35.632567 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6" event={"ID":"e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2","Type":"ContainerStarted","Data":"b1b31b4872a5d314942c723523bb27c6cb25cbf4feeaf148dc1f24578ebea5b7"} Sep 17 01:44:35.632734 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:35.632697 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6" Sep 17 01:44:35.650705 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:35.650656 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" podStartSLOduration=2.310579416 podStartE2EDuration="2.650643095s" totalImagesPullingTime="340.063679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:44:35.648153767 +0000 UTC m=+1340.883300956" watchObservedRunningTime="2026-09-17 01:44:35.650643095 +0000 UTC m=+1340.885790284" Sep 17 01:44:35.664520 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:35.664465 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6" podStartSLOduration=2.664450764 podStartE2EDuration="2.664450764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:44:35.662191771 +0000 UTC m=+1340.897338965" watchObservedRunningTime="2026-09-17 01:44:35.664450764 +0000 UTC m=+1340.899597952" Sep 17 01:44:37.068917 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.068878 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7"] Sep 17 01:44:37.069410 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.069163 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7" podUID="34504414-1aa5-4cd8-b033-276a0d189d75" containerName="payload-pre-processing" containerID="cri-o://24c3f164b5789c385995ebea93f35f8033a8e708b4281a98d34b0c51be40273d" gracePeriod=30 Sep 17 01:44:37.217518 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.217484 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8d689-bjvw4"] Sep 17 01:44:37.217707 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.217534 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d8d689-bjvw4"] Sep 17 01:44:37.339288 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.339257 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7" Sep 17 01:44:37.406984 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.406952 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plbvk\" (UniqueName: \"kubernetes.io/projected/34504414-1aa5-4cd8-b033-276a0d189d75-kube-api-access-plbvk\") pod \"34504414-1aa5-4cd8-b033-276a0d189d75\" (UID: \"34504414-1aa5-4cd8-b033-276a0d189d75\") " Sep 17 01:44:37.406984 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.407000 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34504414-1aa5-4cd8-b033-276a0d189d75-plugins-config-volume\") pod \"34504414-1aa5-4cd8-b033-276a0d189d75\" (UID: \"34504414-1aa5-4cd8-b033-276a0d189d75\") " Sep 17 01:44:37.407382 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.407354 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34504414-1aa5-4cd8-b033-276a0d189d75-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "34504414-1aa5-4cd8-b033-276a0d189d75" (UID: "34504414-1aa5-4cd8-b033-276a0d189d75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:44:37.409184 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.409162 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34504414-1aa5-4cd8-b033-276a0d189d75-kube-api-access-plbvk" (OuterVolumeSpecName: "kube-api-access-plbvk") pod "34504414-1aa5-4cd8-b033-276a0d189d75" (UID: "34504414-1aa5-4cd8-b033-276a0d189d75"). InnerVolumeSpecName "kube-api-access-plbvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:44:37.508900 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.508863 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plbvk\" (UniqueName: \"kubernetes.io/projected/34504414-1aa5-4cd8-b033-276a0d189d75-kube-api-access-plbvk\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:37.508900 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.508893 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34504414-1aa5-4cd8-b033-276a0d189d75-plugins-config-volume\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:37.646134 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.646099 2577 generic.go:356] "Generic (PLEG): container finished" podID="34504414-1aa5-4cd8-b033-276a0d189d75" containerID="24c3f164b5789c385995ebea93f35f8033a8e708b4281a98d34b0c51be40273d" exitCode=0 Sep 17 01:44:37.646339 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.646150 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7" Sep 17 01:44:37.646339 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.646184 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7" event={"ID":"34504414-1aa5-4cd8-b033-276a0d189d75","Type":"ContainerDied","Data":"24c3f164b5789c385995ebea93f35f8033a8e708b4281a98d34b0c51be40273d"} Sep 17 01:44:37.646339 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.646240 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7" event={"ID":"34504414-1aa5-4cd8-b033-276a0d189d75","Type":"ContainerDied","Data":"b6bf1e41d7caa53950db45f4fe11493213ef9c8bee9f78b54f4a0863d995aa8d"} Sep 17 01:44:37.646339 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.646266 2577 scope.go:117] "RemoveContainer" containerID="24c3f164b5789c385995ebea93f35f8033a8e708b4281a98d34b0c51be40273d" Sep 17 01:44:37.656645 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.656619 2577 scope.go:117] "RemoveContainer" containerID="24c3f164b5789c385995ebea93f35f8033a8e708b4281a98d34b0c51be40273d" Sep 17 01:44:37.656923 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:44:37.656895 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c3f164b5789c385995ebea93f35f8033a8e708b4281a98d34b0c51be40273d\": container with ID starting with 24c3f164b5789c385995ebea93f35f8033a8e708b4281a98d34b0c51be40273d not found: ID does not exist" containerID="24c3f164b5789c385995ebea93f35f8033a8e708b4281a98d34b0c51be40273d" Sep 17 01:44:37.657062 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.656948 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c3f164b5789c385995ebea93f35f8033a8e708b4281a98d34b0c51be40273d"} err="failed to get container status \"24c3f164b5789c385995ebea93f35f8033a8e708b4281a98d34b0c51be40273d\": rpc error: code = NotFound desc = could not find container \"24c3f164b5789c385995ebea93f35f8033a8e708b4281a98d34b0c51be40273d\": container with ID starting with 24c3f164b5789c385995ebea93f35f8033a8e708b4281a98d34b0c51be40273d not found: ID does not exist" Sep 17 01:44:37.669366 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.669322 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7"] Sep 17 01:44:37.672150 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:37.672124 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d8d689-5f57947d54-wxlb7"] Sep 17 01:44:39.213130 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:39.213097 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fc10e4e-6f80-4078-86e2-5dde809978ec" path="/var/lib/kubelet/pods/1fc10e4e-6f80-4078-86e2-5dde809978ec/volumes" Sep 17 01:44:39.213535 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:39.213518 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34504414-1aa5-4cd8-b033-276a0d189d75" path="/var/lib/kubelet/pods/34504414-1aa5-4cd8-b033-276a0d189d75/volumes" Sep 17 01:44:41.644277 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:41.644241 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" Sep 17 01:44:43.536257 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:43.536214 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c"] Sep 17 01:44:43.536644 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:43.536489 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" podUID="010061bf-7ab5-4623-826a-545d286b26e9" containerName="istio-proxy" containerID="cri-o://3b10de500dc96d811b5637f6dddb1e2183d1c584ccc0055b373bac74a8cf06b5" gracePeriod=30 Sep 17 01:44:46.641588 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:46.641554 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6" Sep 17 01:44:48.733559 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.733369 2577 generic.go:356] "Generic (PLEG): container finished" podID="010061bf-7ab5-4623-826a-545d286b26e9" containerID="3b10de500dc96d811b5637f6dddb1e2183d1c584ccc0055b373bac74a8cf06b5" exitCode=0 Sep 17 01:44:48.733559 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.733546 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" event={"ID":"010061bf-7ab5-4623-826a-545d286b26e9","Type":"ContainerDied","Data":"3b10de500dc96d811b5637f6dddb1e2183d1c584ccc0055b373bac74a8cf06b5"} Sep 17 01:44:48.734088 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.733613 2577 scope.go:117] "RemoveContainer" containerID="689714ffb0a4e2337dddc5a3d7545aaee97ec94055b54958534875ae58f9c27b" Sep 17 01:44:48.807205 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.807125 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:44:48.933098 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.933059 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-istio-envoy\") pod \"010061bf-7ab5-4623-826a-545d286b26e9\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " Sep 17 01:44:48.933098 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.933100 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-workload-certs\") pod \"010061bf-7ab5-4623-826a-545d286b26e9\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " Sep 17 01:44:48.933298 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.933131 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-istio-data\") pod \"010061bf-7ab5-4623-826a-545d286b26e9\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " Sep 17 01:44:48.933298 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.933190 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/010061bf-7ab5-4623-826a-545d286b26e9-istiod-ca-cert\") pod \"010061bf-7ab5-4623-826a-545d286b26e9\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " Sep 17 01:44:48.933298 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.933254 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/010061bf-7ab5-4623-826a-545d286b26e9-istio-podinfo\") pod \"010061bf-7ab5-4623-826a-545d286b26e9\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " Sep 17 01:44:48.933418 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.933309 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69xr8\" (UniqueName: \"kubernetes.io/projected/010061bf-7ab5-4623-826a-545d286b26e9-kube-api-access-69xr8\") pod \"010061bf-7ab5-4623-826a-545d286b26e9\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " Sep 17 01:44:48.933418 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.933349 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-credential-socket\") pod \"010061bf-7ab5-4623-826a-545d286b26e9\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " Sep 17 01:44:48.933418 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.933393 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010061bf-7ab5-4623-826a-545d286b26e9-istio-token\") pod \"010061bf-7ab5-4623-826a-545d286b26e9\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " Sep 17 01:44:48.933566 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.933431 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "010061bf-7ab5-4623-826a-545d286b26e9" (UID: "010061bf-7ab5-4623-826a-545d286b26e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:44:48.933566 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.933453 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-workload-socket\") pod \"010061bf-7ab5-4623-826a-545d286b26e9\" (UID: \"010061bf-7ab5-4623-826a-545d286b26e9\") " Sep 17 01:44:48.933566 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.933449 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-istio-data" (OuterVolumeSpecName: "istio-data") pod "010061bf-7ab5-4623-826a-545d286b26e9" (UID: "010061bf-7ab5-4623-826a-545d286b26e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:44:48.933727 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.933669 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010061bf-7ab5-4623-826a-545d286b26e9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "010061bf-7ab5-4623-826a-545d286b26e9" (UID: "010061bf-7ab5-4623-826a-545d286b26e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:44:48.933796 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.933676 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "010061bf-7ab5-4623-826a-545d286b26e9" (UID: "010061bf-7ab5-4623-826a-545d286b26e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:44:48.933903 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.933880 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "010061bf-7ab5-4623-826a-545d286b26e9" (UID: "010061bf-7ab5-4623-826a-545d286b26e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:44:48.934029 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.933897 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-credential-socket\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:48.934029 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.933918 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-workload-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:48.934029 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.933960 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-istio-data\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:48.934029 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.933970 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/010061bf-7ab5-4623-826a-545d286b26e9-istiod-ca-cert\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:48.935427 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.935400 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "010061bf-7ab5-4623-826a-545d286b26e9" (UID: "010061bf-7ab5-4623-826a-545d286b26e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:44:48.935603 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.935580 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010061bf-7ab5-4623-826a-545d286b26e9-kube-api-access-69xr8" (OuterVolumeSpecName: "kube-api-access-69xr8") pod "010061bf-7ab5-4623-826a-545d286b26e9" (UID: "010061bf-7ab5-4623-826a-545d286b26e9"). InnerVolumeSpecName "kube-api-access-69xr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:44:48.935973 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.935951 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010061bf-7ab5-4623-826a-545d286b26e9-istio-token" (OuterVolumeSpecName: "istio-token") pod "010061bf-7ab5-4623-826a-545d286b26e9" (UID: "010061bf-7ab5-4623-826a-545d286b26e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:44:48.935973 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:48.935950 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/010061bf-7ab5-4623-826a-545d286b26e9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "010061bf-7ab5-4623-826a-545d286b26e9" (UID: "010061bf-7ab5-4623-826a-545d286b26e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 01:44:49.035161 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:49.035123 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010061bf-7ab5-4623-826a-545d286b26e9-istio-token\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:49.035161 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:49.035157 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-workload-socket\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:49.035161 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:49.035169 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/010061bf-7ab5-4623-826a-545d286b26e9-istio-envoy\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:49.035429 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:49.035181 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/010061bf-7ab5-4623-826a-545d286b26e9-istio-podinfo\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:49.035429 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:49.035192 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69xr8\" (UniqueName: \"kubernetes.io/projected/010061bf-7ab5-4623-826a-545d286b26e9-kube-api-access-69xr8\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:44:49.743518 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:49.743484 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" Sep 17 01:44:49.744020 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:49.743519 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c" event={"ID":"010061bf-7ab5-4623-826a-545d286b26e9","Type":"ContainerDied","Data":"0de16394d0e785e7e43f24e10327790ba5346baf19f40495da8b309803d6c95b"} Sep 17 01:44:49.744020 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:49.743594 2577 scope.go:117] "RemoveContainer" containerID="3b10de500dc96d811b5637f6dddb1e2183d1c584ccc0055b373bac74a8cf06b5" Sep 17 01:44:49.761924 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:49.761643 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c"] Sep 17 01:44:49.764923 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:49.764896 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d8d689-openshift-default-5847cf4968-pr26c"] Sep 17 01:44:51.212979 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:51.212914 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="010061bf-7ab5-4623-826a-545d286b26e9" path="/var/lib/kubelet/pods/010061bf-7ab5-4623-826a-545d286b26e9/volumes" Sep 17 01:44:55.786662 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.786624 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr"] Sep 17 01:44:55.787420 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.787398 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="010061bf-7ab5-4623-826a-545d286b26e9" containerName="istio-proxy" Sep 17 01:44:55.787477 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.787424 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="010061bf-7ab5-4623-826a-545d286b26e9" containerName="istio-proxy" Sep 17 01:44:55.787477 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.787448 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="010061bf-7ab5-4623-826a-545d286b26e9" containerName="istio-proxy" Sep 17 01:44:55.787477 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.787456 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="010061bf-7ab5-4623-826a-545d286b26e9" containerName="istio-proxy" Sep 17 01:44:55.787477 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.787471 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34504414-1aa5-4cd8-b033-276a0d189d75" containerName="payload-pre-processing" Sep 17 01:44:55.787602 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.787480 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="34504414-1aa5-4cd8-b033-276a0d189d75" containerName="payload-pre-processing" Sep 17 01:44:55.787644 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.787619 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="010061bf-7ab5-4623-826a-545d286b26e9" containerName="istio-proxy" Sep 17 01:44:55.787644 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.787635 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="010061bf-7ab5-4623-826a-545d286b26e9" containerName="istio-proxy" Sep 17 01:44:55.787737 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.787645 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="34504414-1aa5-4cd8-b033-276a0d189d75" containerName="payload-pre-processing" Sep 17 01:44:55.792961 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.792921 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" Sep 17 01:44:55.795194 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.795170 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-725559-serving-cert\"" Sep 17 01:44:55.798720 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.798696 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr"] Sep 17 01:44:55.896300 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.896198 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ef0979e-f4b1-4881-8bf6-eb87e4291484-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr\" (UID: \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" Sep 17 01:44:55.896506 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.896332 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef0979e-f4b1-4881-8bf6-eb87e4291484-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr\" (UID: \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" Sep 17 01:44:55.896506 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.896387 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ef0979e-f4b1-4881-8bf6-eb87e4291484-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr\" (UID: \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" Sep 17 01:44:55.896506 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.896429 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwpnh\" (UniqueName: \"kubernetes.io/projected/9ef0979e-f4b1-4881-8bf6-eb87e4291484-kube-api-access-lwpnh\") pod \"maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr\" (UID: \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" Sep 17 01:44:55.997482 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.997445 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef0979e-f4b1-4881-8bf6-eb87e4291484-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr\" (UID: \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" Sep 17 01:44:55.997679 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.997502 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ef0979e-f4b1-4881-8bf6-eb87e4291484-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr\" (UID: \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" Sep 17 01:44:55.997679 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.997539 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwpnh\" (UniqueName: \"kubernetes.io/projected/9ef0979e-f4b1-4881-8bf6-eb87e4291484-kube-api-access-lwpnh\") pod \"maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr\" (UID: \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" Sep 17 01:44:55.997679 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.997618 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ef0979e-f4b1-4881-8bf6-eb87e4291484-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr\" (UID: \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" Sep 17 01:44:55.998274 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:55.998246 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef0979e-f4b1-4881-8bf6-eb87e4291484-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr\" (UID: \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" Sep 17 01:44:56.000139 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:56.000112 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ef0979e-f4b1-4881-8bf6-eb87e4291484-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr\" (UID: \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" Sep 17 01:44:56.000229 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:56.000167 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ef0979e-f4b1-4881-8bf6-eb87e4291484-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr\" (UID: \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" Sep 17 01:44:56.004991 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:56.004971 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwpnh\" (UniqueName: \"kubernetes.io/projected/9ef0979e-f4b1-4881-8bf6-eb87e4291484-kube-api-access-lwpnh\") pod \"maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr\" (UID: \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" Sep 17 01:44:56.108464 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:56.108432 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" Sep 17 01:44:56.399063 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:56.399030 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr"] Sep 17 01:44:56.400308 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:44:56.400281 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef0979e_f4b1_4881_8bf6_eb87e4291484.slice/crio-78762e9c150f9c0e9ff92aff14fd10be9164586f9618297647719d12fe12973e WatchSource:0}: Error finding container 78762e9c150f9c0e9ff92aff14fd10be9164586f9618297647719d12fe12973e: Status 404 returned error can't find the container with id 78762e9c150f9c0e9ff92aff14fd10be9164586f9618297647719d12fe12973e Sep 17 01:44:56.793825 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:56.793742 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" event={"ID":"9ef0979e-f4b1-4881-8bf6-eb87e4291484","Type":"ContainerStarted","Data":"78762e9c150f9c0e9ff92aff14fd10be9164586f9618297647719d12fe12973e"} Sep 17 01:44:57.804884 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:57.804845 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" event={"ID":"9ef0979e-f4b1-4881-8bf6-eb87e4291484","Type":"ContainerStarted","Data":"166d017c3fe9261926a142e28ffff84ab7c089e0aab299d0cb8a205d8b52d6f1"} Sep 17 01:44:57.805293 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:57.804977 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" Sep 17 01:44:57.840951 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:44:57.840858 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" podStartSLOduration=2.335041805 podStartE2EDuration="2.840835625s" totalImagesPullingTime="505.79382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:44:57.838722099 +0000 UTC m=+1363.073869288" watchObservedRunningTime="2026-09-17 01:44:57.840835625 +0000 UTC m=+1363.075982815" Sep 17 01:45:00.142217 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:00.142178 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826825-bvfzn"] Sep 17 01:45:00.152294 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:00.152265 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826825-bvfzn" Sep 17 01:45:00.157008 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:00.156748 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826825-bvfzn"] Sep 17 01:45:00.157180 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:00.157098 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r9tdk\"" Sep 17 01:45:00.166834 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:00.166811 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826825-bvfzn" Sep 17 01:45:00.449280 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:00.449253 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826825-bvfzn"] Sep 17 01:45:00.450271 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:45:00.450247 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74d398b_705d_4d53_925d_217da6b02f26.slice/crio-336fd9c1dae5af72306b2f7888b03b574707762c20d960bb08b7181680880601 WatchSource:0}: Error finding container 336fd9c1dae5af72306b2f7888b03b574707762c20d960bb08b7181680880601: Status 404 returned error can't find the container with id 336fd9c1dae5af72306b2f7888b03b574707762c20d960bb08b7181680880601 Sep 17 01:45:00.829502 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:00.829405 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826825-bvfzn" event={"ID":"a74d398b-705d-4d53-925d-217da6b02f26","Type":"ContainerStarted","Data":"7445d0b15abb5bca0471d8b3e69d2fb9f00d75d1a8ce65f2694e4442ab20b1c0"} Sep 17 01:45:00.829502 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:00.829464 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826825-bvfzn" event={"ID":"a74d398b-705d-4d53-925d-217da6b02f26","Type":"ContainerStarted","Data":"336fd9c1dae5af72306b2f7888b03b574707762c20d960bb08b7181680880601"} Sep 17 01:45:00.843516 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:00.843462 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826825-bvfzn" podStartSLOduration=0.843446016 podStartE2EDuration="843.446016ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:45:00.841842763 +0000 UTC m=+1366.076989952" watchObservedRunningTime="2026-09-17 01:45:00.843446016 +0000 UTC m=+1366.078593204" Sep 17 01:45:01.838415 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:01.838324 2577 generic.go:356] "Generic (PLEG): container finished" podID="a74d398b-705d-4d53-925d-217da6b02f26" containerID="7445d0b15abb5bca0471d8b3e69d2fb9f00d75d1a8ce65f2694e4442ab20b1c0" exitCode=0 Sep 17 01:45:01.838769 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:01.838413 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826825-bvfzn" event={"ID":"a74d398b-705d-4d53-925d-217da6b02f26","Type":"ContainerDied","Data":"7445d0b15abb5bca0471d8b3e69d2fb9f00d75d1a8ce65f2694e4442ab20b1c0"} Sep 17 01:45:02.997198 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:02.997170 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826825-bvfzn" Sep 17 01:45:03.818511 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:03.818401 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" Sep 17 01:45:03.854750 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:03.854712 2577 generic.go:356] "Generic (PLEG): container finished" podID="8b5b4aa7-9127-4b16-ba65-5f2bd797c338" containerID="61b2576993de2535762d56784f07046d65f95d40df9f5f100d9a1c1839c10b94" exitCode=0 Sep 17 01:45:03.854984 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:03.854788 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" event={"ID":"8b5b4aa7-9127-4b16-ba65-5f2bd797c338","Type":"ContainerDied","Data":"61b2576993de2535762d56784f07046d65f95d40df9f5f100d9a1c1839c10b94"} Sep 17 01:45:03.855415 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:03.855391 2577 scope.go:117] "RemoveContainer" containerID="61b2576993de2535762d56784f07046d65f95d40df9f5f100d9a1c1839c10b94" Sep 17 01:45:03.859331 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:03.859290 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:45:03.859438 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:03.859390 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:45:03.859492 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:03.859474 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:45:03.859577 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:03.859553 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826825-bvfzn" Sep 17 01:45:03.859643 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:03.859598 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826825-bvfzn" event={"ID":"a74d398b-705d-4d53-925d-217da6b02f26","Type":"ContainerDied","Data":"336fd9c1dae5af72306b2f7888b03b574707762c20d960bb08b7181680880601"} Sep 17 01:45:03.859643 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:03.859635 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="336fd9c1dae5af72306b2f7888b03b574707762c20d960bb08b7181680880601" Sep 17 01:45:04.868338 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:04.868306 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" event={"ID":"8b5b4aa7-9127-4b16-ba65-5f2bd797c338","Type":"ContainerStarted","Data":"0321df4f6a7fd0fe53936923234217f5e100f09f147c9526943f9c1338142224"} Sep 17 01:45:04.889683 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:04.889650 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:45:04.889883 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:04.889863 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 17 01:45:04.889949 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:04.889906 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" podUID="8b5b4aa7-9127-4b16-ba65-5f2bd797c338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 17 01:45:05.890599 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:05.890563 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 17 01:45:05.891008 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:05.890619 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" podUID="8b5b4aa7-9127-4b16-ba65-5f2bd797c338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 17 01:45:06.890640 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:06.890599 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 17 01:45:06.891085 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:06.890673 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" podUID="8b5b4aa7-9127-4b16-ba65-5f2bd797c338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 17 01:45:07.889906 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:07.889866 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 17 01:45:07.890114 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:07.889990 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" podUID="8b5b4aa7-9127-4b16-ba65-5f2bd797c338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 17 01:45:08.890626 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:08.890586 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 17 01:45:08.891179 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:08.890648 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" podUID="8b5b4aa7-9127-4b16-ba65-5f2bd797c338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 17 01:45:08.964062 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:08.964026 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10044-6c6fa6"] Sep 17 01:45:08.964543 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:08.964528 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a74d398b-705d-4d53-925d-217da6b02f26" containerName="cleanup" Sep 17 01:45:08.964589 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:08.964545 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a74d398b-705d-4d53-925d-217da6b02f26" containerName="cleanup" Sep 17 01:45:08.964714 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:08.964703 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a74d398b-705d-4d53-925d-217da6b02f26" containerName="cleanup" Sep 17 01:45:08.968289 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:08.968262 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10044-6c6fa6" Sep 17 01:45:08.970350 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:08.970203 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qszjr\"" Sep 17 01:45:08.971379 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:08.971353 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10044-6c6fa6"] Sep 17 01:45:09.047286 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.047253 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sjw6\" (UniqueName: \"kubernetes.io/projected/c3bb2be8-258c-47d9-b2e2-875da53fb973-kube-api-access-2sjw6\") pod \"test-curl-10044-6c6fa6\" (UID: \"c3bb2be8-258c-47d9-b2e2-875da53fb973\") " pod="opendatahub/test-curl-10044-6c6fa6" Sep 17 01:45:09.068865 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.068824 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq"] Sep 17 01:45:09.072656 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.072635 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" Sep 17 01:45:09.074553 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.074532 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-fcd402-serving-cert\"" Sep 17 01:45:09.079633 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.079600 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq"] Sep 17 01:45:09.151852 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.151760 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slv5p\" (UniqueName: \"kubernetes.io/projected/9d1c705f-8f65-435f-9747-1ba29daee4fe-kube-api-access-slv5p\") pod \"maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq\" (UID: \"9d1c705f-8f65-435f-9747-1ba29daee4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" Sep 17 01:45:09.151852 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.151830 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d1c705f-8f65-435f-9747-1ba29daee4fe-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq\" (UID: \"9d1c705f-8f65-435f-9747-1ba29daee4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" Sep 17 01:45:09.152072 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.151884 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d1c705f-8f65-435f-9747-1ba29daee4fe-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq\" (UID: \"9d1c705f-8f65-435f-9747-1ba29daee4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" Sep 17 01:45:09.152072 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.152002 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d1c705f-8f65-435f-9747-1ba29daee4fe-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq\" (UID: \"9d1c705f-8f65-435f-9747-1ba29daee4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" Sep 17 01:45:09.152208 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.152189 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sjw6\" (UniqueName: \"kubernetes.io/projected/c3bb2be8-258c-47d9-b2e2-875da53fb973-kube-api-access-2sjw6\") pod \"test-curl-10044-6c6fa6\" (UID: \"c3bb2be8-258c-47d9-b2e2-875da53fb973\") " pod="opendatahub/test-curl-10044-6c6fa6" Sep 17 01:45:09.170462 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.170431 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sjw6\" (UniqueName: \"kubernetes.io/projected/c3bb2be8-258c-47d9-b2e2-875da53fb973-kube-api-access-2sjw6\") pod \"test-curl-10044-6c6fa6\" (UID: \"c3bb2be8-258c-47d9-b2e2-875da53fb973\") " pod="opendatahub/test-curl-10044-6c6fa6" Sep 17 01:45:09.253379 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.253339 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slv5p\" (UniqueName: \"kubernetes.io/projected/9d1c705f-8f65-435f-9747-1ba29daee4fe-kube-api-access-slv5p\") pod \"maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq\" (UID: \"9d1c705f-8f65-435f-9747-1ba29daee4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" Sep 17 01:45:09.253603 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.253398 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d1c705f-8f65-435f-9747-1ba29daee4fe-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq\" (UID: \"9d1c705f-8f65-435f-9747-1ba29daee4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" Sep 17 01:45:09.253603 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.253457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d1c705f-8f65-435f-9747-1ba29daee4fe-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq\" (UID: \"9d1c705f-8f65-435f-9747-1ba29daee4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" Sep 17 01:45:09.253737 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:45:09.253684 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-serving-cert: secret "maas-api-e2e-shared-a-w2-fcd402-serving-cert" not found Sep 17 01:45:09.253806 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:45:09.253761 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1c705f-8f65-435f-9747-1ba29daee4fe-maas-api-tls podName:9d1c705f-8f65-435f-9747-1ba29daee4fe nodeName:}" failed. No retries permitted until 2026-09-17 01:45:09.753740409 +0000 UTC m=+1374.988887588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d1c705f-8f65-435f-9747-1ba29daee4fe-maas-api-tls") pod "maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" (UID: "9d1c705f-8f65-435f-9747-1ba29daee4fe") : secret "maas-api-e2e-shared-a-w2-fcd402-serving-cert" not found Sep 17 01:45:09.254328 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.254301 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d1c705f-8f65-435f-9747-1ba29daee4fe-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq\" (UID: \"9d1c705f-8f65-435f-9747-1ba29daee4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" Sep 17 01:45:09.254448 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.254356 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d1c705f-8f65-435f-9747-1ba29daee4fe-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq\" (UID: \"9d1c705f-8f65-435f-9747-1ba29daee4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" Sep 17 01:45:09.256629 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.256607 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d1c705f-8f65-435f-9747-1ba29daee4fe-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq\" (UID: \"9d1c705f-8f65-435f-9747-1ba29daee4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" Sep 17 01:45:09.263334 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.263313 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slv5p\" (UniqueName: \"kubernetes.io/projected/9d1c705f-8f65-435f-9747-1ba29daee4fe-kube-api-access-slv5p\") pod \"maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq\" (UID: \"9d1c705f-8f65-435f-9747-1ba29daee4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" Sep 17 01:45:09.281076 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.281045 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10044-6c6fa6" Sep 17 01:45:09.582194 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.582155 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10044-6c6fa6"] Sep 17 01:45:09.583357 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:45:09.583323 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3bb2be8_258c_47d9_b2e2_875da53fb973.slice/crio-f8dff352e6382cf08ed121e69d73f1af1f3813ebeaed9bdc1dc0fe84f293bf33 WatchSource:0}: Error finding container f8dff352e6382cf08ed121e69d73f1af1f3813ebeaed9bdc1dc0fe84f293bf33: Status 404 returned error can't find the container with id f8dff352e6382cf08ed121e69d73f1af1f3813ebeaed9bdc1dc0fe84f293bf33 Sep 17 01:45:09.759189 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.759147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d1c705f-8f65-435f-9747-1ba29daee4fe-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq\" (UID: \"9d1c705f-8f65-435f-9747-1ba29daee4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" Sep 17 01:45:09.761529 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.761510 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d1c705f-8f65-435f-9747-1ba29daee4fe-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq\" (UID: \"9d1c705f-8f65-435f-9747-1ba29daee4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" Sep 17 01:45:09.890162 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.890127 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 17 01:45:09.890341 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.890185 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" podUID="8b5b4aa7-9127-4b16-ba65-5f2bd797c338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 17 01:45:09.902071 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.902035 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10044-6c6fa6" event={"ID":"c3bb2be8-258c-47d9-b2e2-875da53fb973","Type":"ContainerStarted","Data":"7267bd2b163a732cf004d8608fb950394f3a29cb82aa9f81fd251b0eca61588d"} Sep 17 01:45:09.902567 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.902082 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10044-6c6fa6" event={"ID":"c3bb2be8-258c-47d9-b2e2-875da53fb973","Type":"ContainerStarted","Data":"f8dff352e6382cf08ed121e69d73f1af1f3813ebeaed9bdc1dc0fe84f293bf33"} Sep 17 01:45:09.917021 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.916955 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10044-6c6fa6" podStartSLOduration=1.916915555 podStartE2EDuration="1.916915555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:45:09.916189446 +0000 UTC m=+1375.151336634" watchObservedRunningTime="2026-09-17 01:45:09.916915555 +0000 UTC m=+1375.152062776" Sep 17 01:45:09.983869 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:09.983820 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" Sep 17 01:45:10.259915 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.259888 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq"] Sep 17 01:45:10.260565 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:45:10.260540 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1c705f_8f65_435f_9747_1ba29daee4fe.slice/crio-5edea9a53056382f7b86f79b0cf2790307518a805ebeb097d1277d98da023495 WatchSource:0}: Error finding container 5edea9a53056382f7b86f79b0cf2790307518a805ebeb097d1277d98da023495: Status 404 returned error can't find the container with id 5edea9a53056382f7b86f79b0cf2790307518a805ebeb097d1277d98da023495 Sep 17 01:45:10.262413 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.262397 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 01:45:10.478771 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.478727 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10044-6c6fa6"] Sep 17 01:45:10.482371 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.482344 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10044-6c6fa6"] Sep 17 01:45:10.589162 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.589079 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10044-77f427"] Sep 17 01:45:10.589576 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.589563 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3bb2be8-258c-47d9-b2e2-875da53fb973" containerName="test-curl-10044-6c6fa6" Sep 17 01:45:10.589618 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.589577 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3bb2be8-258c-47d9-b2e2-875da53fb973" containerName="test-curl-10044-6c6fa6" Sep 17 01:45:10.589678 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.589669 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3bb2be8-258c-47d9-b2e2-875da53fb973" containerName="test-curl-10044-6c6fa6" Sep 17 01:45:10.594454 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.594422 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10044-77f427" Sep 17 01:45:10.595885 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.595858 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10044-77f427"] Sep 17 01:45:10.667227 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.667187 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jccb2\" (UniqueName: \"kubernetes.io/projected/696c8c79-14ae-4d09-85ad-907ae2d6cb44-kube-api-access-jccb2\") pod \"test-curl-10044-77f427\" (UID: \"696c8c79-14ae-4d09-85ad-907ae2d6cb44\") " pod="opendatahub/test-curl-10044-77f427" Sep 17 01:45:10.768021 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.767988 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jccb2\" (UniqueName: \"kubernetes.io/projected/696c8c79-14ae-4d09-85ad-907ae2d6cb44-kube-api-access-jccb2\") pod \"test-curl-10044-77f427\" (UID: \"696c8c79-14ae-4d09-85ad-907ae2d6cb44\") " pod="opendatahub/test-curl-10044-77f427" Sep 17 01:45:10.776303 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.776259 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jccb2\" (UniqueName: \"kubernetes.io/projected/696c8c79-14ae-4d09-85ad-907ae2d6cb44-kube-api-access-jccb2\") pod \"test-curl-10044-77f427\" (UID: \"696c8c79-14ae-4d09-85ad-907ae2d6cb44\") " pod="opendatahub/test-curl-10044-77f427" Sep 17 01:45:10.894698 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.894667 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:45:10.894986 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.894961 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:45:10.895658 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.895638 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:45:10.908294 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.908266 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10044-77f427" Sep 17 01:45:10.910411 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.910375 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" event={"ID":"9d1c705f-8f65-435f-9747-1ba29daee4fe","Type":"ContainerStarted","Data":"08f797392dd9512e92009817e090d3da114d41960544517cd7c99ddab706fc08"} Sep 17 01:45:10.910525 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.910426 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" event={"ID":"9d1c705f-8f65-435f-9747-1ba29daee4fe","Type":"ContainerStarted","Data":"5edea9a53056382f7b86f79b0cf2790307518a805ebeb097d1277d98da023495"} Sep 17 01:45:10.910598 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:45:10.910560 2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10044-6c6fa6" container="test-curl-10044-6c6fa6" Sep 17 01:45:10.949966 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:10.948944 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" podStartSLOduration=1.52168322 podStartE2EDuration="1.948909437s" totalImagesPullingTime="427.226217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:45:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:45:10.946922036 +0000 UTC m=+1376.182069226" watchObservedRunningTime="2026-09-17 01:45:10.948909437 +0000 UTC m=+1376.184056625" Sep 17 01:45:11.189252 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:11.189217 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10044-77f427"] Sep 17 01:45:11.191709 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:45:11.191674 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod696c8c79_14ae_4d09_85ad_907ae2d6cb44.slice/crio-e976f31772a7a6bbeb09739c0bf823ac969a1b604810bc02c2cf672e1ec38176 WatchSource:0}: Error finding container e976f31772a7a6bbeb09739c0bf823ac969a1b604810bc02c2cf672e1ec38176: Status 404 returned error can't find the container with id e976f31772a7a6bbeb09739c0bf823ac969a1b604810bc02c2cf672e1ec38176 Sep 17 01:45:11.919292 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:11.919251 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10044-77f427" event={"ID":"696c8c79-14ae-4d09-85ad-907ae2d6cb44","Type":"ContainerStarted","Data":"6242be55666c638ab6d727a907e02276f0e8ecb397867bcf457819e06e769264"} Sep 17 01:45:11.919845 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:11.919743 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10044-6c6fa6" podUID="c3bb2be8-258c-47d9-b2e2-875da53fb973" containerName="test-curl-10044-6c6fa6" containerID="cri-o://7267bd2b163a732cf004d8608fb950394f3a29cb82aa9f81fd251b0eca61588d" gracePeriod=2 Sep 17 01:45:11.920058 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:11.919823 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" Sep 17 01:45:11.920160 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:11.920088 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10044-77f427" event={"ID":"696c8c79-14ae-4d09-85ad-907ae2d6cb44","Type":"ContainerStarted","Data":"e976f31772a7a6bbeb09739c0bf823ac969a1b604810bc02c2cf672e1ec38176"} Sep 17 01:45:11.932770 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:11.932719 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10044-77f427" podStartSLOduration=1.932702645 podStartE2EDuration="1.932702645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:45:11.931714882 +0000 UTC m=+1377.166862070" watchObservedRunningTime="2026-09-17 01:45:11.932702645 +0000 UTC m=+1377.167849832" Sep 17 01:45:12.509664 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:12.509612 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10044-77f427"] Sep 17 01:45:12.515887 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:12.515846 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10044-77f427"] Sep 17 01:45:12.624504 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:12.624470 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10044-8d507e"] Sep 17 01:45:12.624979 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:12.624965 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="696c8c79-14ae-4d09-85ad-907ae2d6cb44" containerName="test-curl-10044-77f427" Sep 17 01:45:12.625038 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:12.624980 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="696c8c79-14ae-4d09-85ad-907ae2d6cb44" containerName="test-curl-10044-77f427" Sep 17 01:45:12.625092 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:12.625083 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="696c8c79-14ae-4d09-85ad-907ae2d6cb44" containerName="test-curl-10044-77f427" Sep 17 01:45:12.628454 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:12.628435 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10044-8d507e" Sep 17 01:45:12.636427 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:12.636390 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10044-8d507e"] Sep 17 01:45:12.789613 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:12.789576 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg4m2\" (UniqueName: \"kubernetes.io/projected/b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0-kube-api-access-bg4m2\") pod \"test-curl-10044-8d507e\" (UID: \"b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0\") " pod="opendatahub/test-curl-10044-8d507e" Sep 17 01:45:12.890717 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:12.890619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg4m2\" (UniqueName: \"kubernetes.io/projected/b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0-kube-api-access-bg4m2\") pod \"test-curl-10044-8d507e\" (UID: \"b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0\") " pod="opendatahub/test-curl-10044-8d507e" Sep 17 01:45:12.897686 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:12.897655 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg4m2\" (UniqueName: \"kubernetes.io/projected/b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0-kube-api-access-bg4m2\") pod \"test-curl-10044-8d507e\" (UID: \"b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0\") " pod="opendatahub/test-curl-10044-8d507e" Sep 17 01:45:12.926352 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:45:12.926323 2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10044-77f427" container="test-curl-10044-77f427" Sep 17 01:45:12.941821 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:12.941795 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10044-8d507e" Sep 17 01:45:13.206663 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:13.206578 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:45:13.212539 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:45:13.212509 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb98b4f3f_0cb7_42a1_a28f_ee5997d41ef0.slice/crio-fe27baa1989cc316a10f6fd9986bb598a29ecdb52da6cb0fc4b8ed29d0f01e55 WatchSource:0}: Error finding container fe27baa1989cc316a10f6fd9986bb598a29ecdb52da6cb0fc4b8ed29d0f01e55: Status 404 returned error can't find the container with id fe27baa1989cc316a10f6fd9986bb598a29ecdb52da6cb0fc4b8ed29d0f01e55 Sep 17 01:45:13.215361 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:13.215329 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10044-8d507e"] Sep 17 01:45:13.935601 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:13.935562 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10044-8d507e" event={"ID":"b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0","Type":"ContainerStarted","Data":"9858adb0b27efb3e9875e04051d16ffe06e66821f51d24859c3e4a9845f361c4"} Sep 17 01:45:13.935984 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:13.935617 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10044-8d507e" event={"ID":"b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0","Type":"ContainerStarted","Data":"fe27baa1989cc316a10f6fd9986bb598a29ecdb52da6cb0fc4b8ed29d0f01e55"} Sep 17 01:45:13.935984 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:13.935655 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10044-77f427" podUID="696c8c79-14ae-4d09-85ad-907ae2d6cb44" containerName="test-curl-10044-77f427" containerID="cri-o://6242be55666c638ab6d727a907e02276f0e8ecb397867bcf457819e06e769264" gracePeriod=2 Sep 17 01:45:13.949234 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:13.949185 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10044-8d507e" podStartSLOduration=1.9491690400000001 podStartE2EDuration="1.94916904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:45:13.947352559 +0000 UTC m=+1379.182499745" watchObservedRunningTime="2026-09-17 01:45:13.94916904 +0000 UTC m=+1379.184316261" Sep 17 01:45:14.080549 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.080520 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10044-6c6fa6" Sep 17 01:45:14.205615 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.205523 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sjw6\" (UniqueName: \"kubernetes.io/projected/c3bb2be8-258c-47d9-b2e2-875da53fb973-kube-api-access-2sjw6\") pod \"c3bb2be8-258c-47d9-b2e2-875da53fb973\" (UID: \"c3bb2be8-258c-47d9-b2e2-875da53fb973\") " Sep 17 01:45:14.207646 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.207613 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3bb2be8-258c-47d9-b2e2-875da53fb973-kube-api-access-2sjw6" (OuterVolumeSpecName: "kube-api-access-2sjw6") pod "c3bb2be8-258c-47d9-b2e2-875da53fb973" (UID: "c3bb2be8-258c-47d9-b2e2-875da53fb973"). InnerVolumeSpecName "kube-api-access-2sjw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:45:14.306515 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.306483 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sjw6\" (UniqueName: \"kubernetes.io/projected/c3bb2be8-258c-47d9-b2e2-875da53fb973-kube-api-access-2sjw6\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:45:14.490135 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.490022 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10044-8d507e"] Sep 17 01:45:14.493345 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.493312 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10044-8d507e"] Sep 17 01:45:14.600701 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.600661 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10044-71e836"] Sep 17 01:45:14.601205 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.601189 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0" containerName="test-curl-10044-8d507e" Sep 17 01:45:14.601258 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.601206 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0" containerName="test-curl-10044-8d507e" Sep 17 01:45:14.601310 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.601300 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0" containerName="test-curl-10044-8d507e" Sep 17 01:45:14.604553 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.604534 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10044-71e836" Sep 17 01:45:14.609986 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.609945 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10044-71e836"] Sep 17 01:45:14.709823 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.709779 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqw2w\" (UniqueName: \"kubernetes.io/projected/f131778d-5a95-4a35-b9cf-c2a93d5ae34c-kube-api-access-qqw2w\") pod \"test-curl-10044-71e836\" (UID: \"f131778d-5a95-4a35-b9cf-c2a93d5ae34c\") " pod="opendatahub/test-curl-10044-71e836" Sep 17 01:45:14.810425 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.810390 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqw2w\" (UniqueName: \"kubernetes.io/projected/f131778d-5a95-4a35-b9cf-c2a93d5ae34c-kube-api-access-qqw2w\") pod \"test-curl-10044-71e836\" (UID: \"f131778d-5a95-4a35-b9cf-c2a93d5ae34c\") " pod="opendatahub/test-curl-10044-71e836" Sep 17 01:45:14.817349 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.817320 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqw2w\" (UniqueName: \"kubernetes.io/projected/f131778d-5a95-4a35-b9cf-c2a93d5ae34c-kube-api-access-qqw2w\") pod \"test-curl-10044-71e836\" (UID: \"f131778d-5a95-4a35-b9cf-c2a93d5ae34c\") " pod="opendatahub/test-curl-10044-71e836" Sep 17 01:45:14.917948 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.917902 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10044-71e836" Sep 17 01:45:14.945070 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.945038 2577 generic.go:356] "Generic (PLEG): container finished" podID="c3bb2be8-258c-47d9-b2e2-875da53fb973" containerID="7267bd2b163a732cf004d8608fb950394f3a29cb82aa9f81fd251b0eca61588d" exitCode=137 Sep 17 01:45:14.945490 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.945112 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10044-6c6fa6" Sep 17 01:45:14.945490 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.945189 2577 scope.go:117] "RemoveContainer" containerID="7267bd2b163a732cf004d8608fb950394f3a29cb82aa9f81fd251b0eca61588d" Sep 17 01:45:14.945490 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:45:14.945428 2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10044-8d507e" container="test-curl-10044-8d507e" Sep 17 01:45:14.956038 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.956018 2577 scope.go:117] "RemoveContainer" containerID="7267bd2b163a732cf004d8608fb950394f3a29cb82aa9f81fd251b0eca61588d" Sep 17 01:45:14.956381 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:45:14.956352 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7267bd2b163a732cf004d8608fb950394f3a29cb82aa9f81fd251b0eca61588d\": container with ID starting with 7267bd2b163a732cf004d8608fb950394f3a29cb82aa9f81fd251b0eca61588d not found: ID does not exist" containerID="7267bd2b163a732cf004d8608fb950394f3a29cb82aa9f81fd251b0eca61588d" Sep 17 01:45:14.956473 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:45:14.956390 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-10044-6c6fa6\" is forbidden: User \"system:node:ip-10-0-138-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-83.ec2.internal' and this object" podUID="c3bb2be8-258c-47d9-b2e2-875da53fb973" pod="opendatahub/test-curl-10044-6c6fa6" Sep 17 01:45:14.956473 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:14.956393 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7267bd2b163a732cf004d8608fb950394f3a29cb82aa9f81fd251b0eca61588d"} err="failed to get container status \"7267bd2b163a732cf004d8608fb950394f3a29cb82aa9f81fd251b0eca61588d\": rpc error: code = NotFound desc = could not find container \"7267bd2b163a732cf004d8608fb950394f3a29cb82aa9f81fd251b0eca61588d\": container with ID starting with 7267bd2b163a732cf004d8608fb950394f3a29cb82aa9f81fd251b0eca61588d not found: ID does not exist" Sep 17 01:45:15.191851 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:15.191813 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10044-71e836"] Sep 17 01:45:15.192828 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:45:15.192798 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf131778d_5a95_4a35_b9cf_c2a93d5ae34c.slice/crio-6f17f26e3a4b55f3c73510b04bb3e5d1694f37836a3b48a1fae13c125721cd5b WatchSource:0}: Error finding container 6f17f26e3a4b55f3c73510b04bb3e5d1694f37836a3b48a1fae13c125721cd5b: Status 404 returned error can't find the container with id 6f17f26e3a4b55f3c73510b04bb3e5d1694f37836a3b48a1fae13c125721cd5b Sep 17 01:45:15.210746 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:45:15.210713 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-10044-6c6fa6\" is forbidden: User \"system:node:ip-10-0-138-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-83.ec2.internal' and this object" podUID="c3bb2be8-258c-47d9-b2e2-875da53fb973" pod="opendatahub/test-curl-10044-6c6fa6" Sep 17 01:45:15.213829 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:15.213803 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3bb2be8-258c-47d9-b2e2-875da53fb973" path="/var/lib/kubelet/pods/c3bb2be8-258c-47d9-b2e2-875da53fb973/volumes" Sep 17 01:45:15.953859 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:15.953830 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10044-71e836" event={"ID":"f131778d-5a95-4a35-b9cf-c2a93d5ae34c","Type":"ContainerStarted","Data":"7336cbe4e69b764467c5ac7f6630181b67fe597129391dfcb0e2f6b8c6fe3261"} Sep 17 01:45:15.954205 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:15.953878 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10044-71e836" event={"ID":"f131778d-5a95-4a35-b9cf-c2a93d5ae34c","Type":"ContainerStarted","Data":"6f17f26e3a4b55f3c73510b04bb3e5d1694f37836a3b48a1fae13c125721cd5b"} Sep 17 01:45:15.954205 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:15.953983 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10044-8d507e" podUID="b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0" containerName="test-curl-10044-8d507e" containerID="cri-o://9858adb0b27efb3e9875e04051d16ffe06e66821f51d24859c3e4a9845f361c4" gracePeriod=2 Sep 17 01:45:15.966480 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:15.966422 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10044-71e836" podStartSLOduration=1.9664022939999999 podStartE2EDuration="1.966402294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:45:15.965321892 +0000 UTC m=+1381.200469081" watchObservedRunningTime="2026-09-17 01:45:15.966402294 +0000 UTC m=+1381.201549483" Sep 17 01:45:16.137281 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:16.137245 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10044-77f427" Sep 17 01:45:16.206595 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:16.206515 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:45:16.226652 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:16.226614 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jccb2\" (UniqueName: \"kubernetes.io/projected/696c8c79-14ae-4d09-85ad-907ae2d6cb44-kube-api-access-jccb2\") pod \"696c8c79-14ae-4d09-85ad-907ae2d6cb44\" (UID: \"696c8c79-14ae-4d09-85ad-907ae2d6cb44\") " Sep 17 01:45:16.228829 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:16.228786 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/696c8c79-14ae-4d09-85ad-907ae2d6cb44-kube-api-access-jccb2" (OuterVolumeSpecName: "kube-api-access-jccb2") pod "696c8c79-14ae-4d09-85ad-907ae2d6cb44" (UID: "696c8c79-14ae-4d09-85ad-907ae2d6cb44"). InnerVolumeSpecName "kube-api-access-jccb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:45:16.328553 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:16.328515 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jccb2\" (UniqueName: \"kubernetes.io/projected/696c8c79-14ae-4d09-85ad-907ae2d6cb44-kube-api-access-jccb2\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:45:16.527458 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:16.527361 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10044-71e836"] Sep 17 01:45:16.531348 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:16.531305 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10044-71e836"] Sep 17 01:45:16.961452 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:16.961414 2577 generic.go:356] "Generic (PLEG): container finished" podID="696c8c79-14ae-4d09-85ad-907ae2d6cb44" containerID="6242be55666c638ab6d727a907e02276f0e8ecb397867bcf457819e06e769264" exitCode=137 Sep 17 01:45:16.961888 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:16.961469 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10044-77f427" Sep 17 01:45:16.961888 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:16.961513 2577 scope.go:117] "RemoveContainer" containerID="6242be55666c638ab6d727a907e02276f0e8ecb397867bcf457819e06e769264" Sep 17 01:45:16.961888 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:45:16.961875 2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10044-71e836" container="test-curl-10044-71e836" Sep 17 01:45:16.971794 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:16.971770 2577 scope.go:117] "RemoveContainer" containerID="6242be55666c638ab6d727a907e02276f0e8ecb397867bcf457819e06e769264" Sep 17 01:45:16.972105 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:45:16.972070 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-10044-77f427\" is forbidden: User \"system:node:ip-10-0-138-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-83.ec2.internal' and this object" podUID="696c8c79-14ae-4d09-85ad-907ae2d6cb44" pod="opendatahub/test-curl-10044-77f427" Sep 17 01:45:16.972195 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:45:16.972117 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6242be55666c638ab6d727a907e02276f0e8ecb397867bcf457819e06e769264\": container with ID starting with 6242be55666c638ab6d727a907e02276f0e8ecb397867bcf457819e06e769264 not found: ID does not exist" containerID="6242be55666c638ab6d727a907e02276f0e8ecb397867bcf457819e06e769264" Sep 17 01:45:16.972195 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:16.972144 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6242be55666c638ab6d727a907e02276f0e8ecb397867bcf457819e06e769264"} err="failed to get container status \"6242be55666c638ab6d727a907e02276f0e8ecb397867bcf457819e06e769264\": rpc error: code = NotFound desc = could not find container \"6242be55666c638ab6d727a907e02276f0e8ecb397867bcf457819e06e769264\": container with ID starting with 6242be55666c638ab6d727a907e02276f0e8ecb397867bcf457819e06e769264 not found: ID does not exist" Sep 17 01:45:17.213533 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:17.213444 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="696c8c79-14ae-4d09-85ad-907ae2d6cb44" path="/var/lib/kubelet/pods/696c8c79-14ae-4d09-85ad-907ae2d6cb44/volumes" Sep 17 01:45:17.931274 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:17.931240 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" Sep 17 01:45:17.971865 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:17.971831 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10044-71e836" podUID="f131778d-5a95-4a35-b9cf-c2a93d5ae34c" containerName="test-curl-10044-71e836" containerID="cri-o://7336cbe4e69b764467c5ac7f6630181b67fe597129391dfcb0e2f6b8c6fe3261" gracePeriod=2 Sep 17 01:45:18.174137 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:18.174112 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10044-8d507e" Sep 17 01:45:18.349185 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:18.349149 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bg4m2\" (UniqueName: \"kubernetes.io/projected/b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0-kube-api-access-bg4m2\") pod \"b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0\" (UID: \"b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0\") " Sep 17 01:45:18.351238 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:18.351211 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0-kube-api-access-bg4m2" (OuterVolumeSpecName: "kube-api-access-bg4m2") pod "b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0" (UID: "b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0"). InnerVolumeSpecName "kube-api-access-bg4m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:45:18.450678 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:18.450641 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg4m2\" (UniqueName: \"kubernetes.io/projected/b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0-kube-api-access-bg4m2\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:45:18.979626 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:18.979592 2577 generic.go:356] "Generic (PLEG): container finished" podID="b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0" containerID="9858adb0b27efb3e9875e04051d16ffe06e66821f51d24859c3e4a9845f361c4" exitCode=137 Sep 17 01:45:18.980053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:18.979634 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10044-8d507e" Sep 17 01:45:18.980053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:18.979689 2577 scope.go:117] "RemoveContainer" containerID="9858adb0b27efb3e9875e04051d16ffe06e66821f51d24859c3e4a9845f361c4" Sep 17 01:45:18.989769 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:45:18.989631 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-10044-8d507e\" is forbidden: User \"system:node:ip-10-0-138-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-83.ec2.internal' and this object" podUID="b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0" pod="opendatahub/test-curl-10044-8d507e" Sep 17 01:45:18.989831 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:18.989785 2577 scope.go:117] "RemoveContainer" containerID="9858adb0b27efb3e9875e04051d16ffe06e66821f51d24859c3e4a9845f361c4" Sep 17 01:45:18.990109 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:45:18.990087 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9858adb0b27efb3e9875e04051d16ffe06e66821f51d24859c3e4a9845f361c4\": container with ID starting with 9858adb0b27efb3e9875e04051d16ffe06e66821f51d24859c3e4a9845f361c4 not found: ID does not exist" containerID="9858adb0b27efb3e9875e04051d16ffe06e66821f51d24859c3e4a9845f361c4" Sep 17 01:45:18.990160 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:18.990118 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9858adb0b27efb3e9875e04051d16ffe06e66821f51d24859c3e4a9845f361c4"} err="failed to get container status \"9858adb0b27efb3e9875e04051d16ffe06e66821f51d24859c3e4a9845f361c4\": rpc error: code = NotFound desc = could not find container \"9858adb0b27efb3e9875e04051d16ffe06e66821f51d24859c3e4a9845f361c4\": container with ID starting with 9858adb0b27efb3e9875e04051d16ffe06e66821f51d24859c3e4a9845f361c4 not found: ID does not exist" Sep 17 01:45:19.214680 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:19.214646 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0" path="/var/lib/kubelet/pods/b98b4f3f-0cb7-42a1-a28f-ee5997d41ef0/volumes" Sep 17 01:45:20.163858 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:20.163827 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10044-71e836" Sep 17 01:45:20.170273 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:20.170246 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqw2w\" (UniqueName: \"kubernetes.io/projected/f131778d-5a95-4a35-b9cf-c2a93d5ae34c-kube-api-access-qqw2w\") pod \"f131778d-5a95-4a35-b9cf-c2a93d5ae34c\" (UID: \"f131778d-5a95-4a35-b9cf-c2a93d5ae34c\") " Sep 17 01:45:20.172237 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:20.172214 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f131778d-5a95-4a35-b9cf-c2a93d5ae34c-kube-api-access-qqw2w" (OuterVolumeSpecName: "kube-api-access-qqw2w") pod "f131778d-5a95-4a35-b9cf-c2a93d5ae34c" (UID: "f131778d-5a95-4a35-b9cf-c2a93d5ae34c"). InnerVolumeSpecName "kube-api-access-qqw2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:45:20.271744 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:20.271643 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqw2w\" (UniqueName: \"kubernetes.io/projected/f131778d-5a95-4a35-b9cf-c2a93d5ae34c-kube-api-access-qqw2w\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:45:21.002808 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:21.002767 2577 generic.go:356] "Generic (PLEG): container finished" podID="f131778d-5a95-4a35-b9cf-c2a93d5ae34c" containerID="7336cbe4e69b764467c5ac7f6630181b67fe597129391dfcb0e2f6b8c6fe3261" exitCode=137 Sep 17 01:45:21.003013 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:21.002813 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10044-71e836" Sep 17 01:45:21.003013 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:21.002825 2577 scope.go:117] "RemoveContainer" containerID="7336cbe4e69b764467c5ac7f6630181b67fe597129391dfcb0e2f6b8c6fe3261" Sep 17 01:45:21.013114 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:21.013091 2577 scope.go:117] "RemoveContainer" containerID="7336cbe4e69b764467c5ac7f6630181b67fe597129391dfcb0e2f6b8c6fe3261" Sep 17 01:45:21.013217 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:45:21.013179 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-10044-71e836\" is forbidden: User \"system:node:ip-10-0-138-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-83.ec2.internal' and this object" podUID="f131778d-5a95-4a35-b9cf-c2a93d5ae34c" pod="opendatahub/test-curl-10044-71e836" Sep 17 01:45:21.013359 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:45:21.013341 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7336cbe4e69b764467c5ac7f6630181b67fe597129391dfcb0e2f6b8c6fe3261\": container with ID starting with 7336cbe4e69b764467c5ac7f6630181b67fe597129391dfcb0e2f6b8c6fe3261 not found: ID does not exist" containerID="7336cbe4e69b764467c5ac7f6630181b67fe597129391dfcb0e2f6b8c6fe3261" Sep 17 01:45:21.013406 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:21.013369 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7336cbe4e69b764467c5ac7f6630181b67fe597129391dfcb0e2f6b8c6fe3261"} err="failed to get container status \"7336cbe4e69b764467c5ac7f6630181b67fe597129391dfcb0e2f6b8c6fe3261\": rpc error: code = NotFound desc = could not find container \"7336cbe4e69b764467c5ac7f6630181b67fe597129391dfcb0e2f6b8c6fe3261\": container with ID starting with 7336cbe4e69b764467c5ac7f6630181b67fe597129391dfcb0e2f6b8c6fe3261 not found: ID does not exist" Sep 17 01:45:21.212885 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:21.212852 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f131778d-5a95-4a35-b9cf-c2a93d5ae34c" path="/var/lib/kubelet/pods/f131778d-5a95-4a35-b9cf-c2a93d5ae34c/volumes" Sep 17 01:45:26.157243 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.157205 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb"] Sep 17 01:45:26.157772 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.157758 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f131778d-5a95-4a35-b9cf-c2a93d5ae34c" containerName="test-curl-10044-71e836" Sep 17 01:45:26.157817 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.157775 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f131778d-5a95-4a35-b9cf-c2a93d5ae34c" containerName="test-curl-10044-71e836" Sep 17 01:45:26.157865 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.157847 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f131778d-5a95-4a35-b9cf-c2a93d5ae34c" containerName="test-curl-10044-71e836" Sep 17 01:45:26.162459 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.162439 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.164366 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.164345 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-361849-openshift-default-dockercfg-jh9lp\"" Sep 17 01:45:26.175028 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.174998 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb"] Sep 17 01:45:26.234023 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.233979 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-istio-data\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.234023 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.234027 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-workload-socket\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.234231 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.234057 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-credential-socket\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.234231 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.234077 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-workload-certs\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.234231 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.234100 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029db114-e409-4496-b364-f1ebac568192-istiod-ca-cert\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.234231 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.234175 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rglbt\" (UniqueName: \"kubernetes.io/projected/029db114-e409-4496-b364-f1ebac568192-kube-api-access-rglbt\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.234361 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.234228 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029db114-e409-4496-b364-f1ebac568192-istio-token\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.234361 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.234276 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-istio-envoy\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.234361 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.234313 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029db114-e409-4496-b364-f1ebac568192-istio-podinfo\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.334984 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.334910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-workload-certs\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.335162 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.335005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029db114-e409-4496-b364-f1ebac568192-istiod-ca-cert\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.335162 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.335041 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rglbt\" (UniqueName: \"kubernetes.io/projected/029db114-e409-4496-b364-f1ebac568192-kube-api-access-rglbt\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.335162 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.335074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029db114-e409-4496-b364-f1ebac568192-istio-token\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.335162 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.335127 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-istio-envoy\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.335369 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.335180 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029db114-e409-4496-b364-f1ebac568192-istio-podinfo\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.335369 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.335231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-istio-data\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.335369 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.335267 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-workload-socket\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.335369 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.335308 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-credential-socket\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.335369 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.335334 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-workload-certs\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.335612 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.335593 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-istio-data\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.335684 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.335616 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-workload-socket\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.335684 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.335671 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-credential-socket\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.335780 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.335724 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029db114-e409-4496-b364-f1ebac568192-istiod-ca-cert\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.337522 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.337497 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-istio-envoy\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.337656 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.337637 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029db114-e409-4496-b364-f1ebac568192-istio-podinfo\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.341944 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.341905 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029db114-e409-4496-b364-f1ebac568192-istio-token\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.342141 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.342123 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rglbt\" (UniqueName: \"kubernetes.io/projected/029db114-e409-4496-b364-f1ebac568192-kube-api-access-rglbt\") pod \"e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.476385 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.476300 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:26.768351 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.768317 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb"] Sep 17 01:45:26.768570 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:45:26.768546 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029db114_e409_4496_b364_f1ebac568192.slice/crio-5aed25fdb78531497655a75b1bcfe33b7334c75f91aeb04d4ee43b907c6fae88 WatchSource:0}: Error finding container 5aed25fdb78531497655a75b1bcfe33b7334c75f91aeb04d4ee43b907c6fae88: Status 404 returned error can't find the container with id 5aed25fdb78531497655a75b1bcfe33b7334c75f91aeb04d4ee43b907c6fae88 Sep 17 01:45:26.772080 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.772043 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:45:26.772208 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.772137 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:45:26.772208 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:26.772183 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:45:27.050073 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:27.049971 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" event={"ID":"029db114-e409-4496-b364-f1ebac568192","Type":"ContainerStarted","Data":"46ad107a814f5ffdcbfba14dee438ee474bde2f8052f80c7dd52ae44479d92ed"} Sep 17 01:45:27.050073 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:27.050034 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" event={"ID":"029db114-e409-4496-b364-f1ebac568192","Type":"ContainerStarted","Data":"5aed25fdb78531497655a75b1bcfe33b7334c75f91aeb04d4ee43b907c6fae88"} Sep 17 01:45:27.066638 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:27.066568 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" podStartSLOduration=1.066547009 podStartE2EDuration="1.066547009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:45:27.065760523 +0000 UTC m=+1392.300907715" watchObservedRunningTime="2026-09-17 01:45:27.066547009 +0000 UTC m=+1392.301694201" Sep 17 01:45:27.477276 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:27.477231 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:27.482421 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:27.482397 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:28.056866 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:28.056823 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:28.057736 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:28.057716 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:34.299521 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:34.299482 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w"] Sep 17 01:45:34.303755 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:34.303728 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" Sep 17 01:45:34.305919 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:34.305897 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-361849-serving-cert\"" Sep 17 01:45:34.312025 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:34.311998 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w"] Sep 17 01:45:34.425015 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:34.424981 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w\" (UID: \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" Sep 17 01:45:34.425216 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:34.425041 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w\" (UID: \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" Sep 17 01:45:34.425285 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:34.425231 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w\" (UID: \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" Sep 17 01:45:34.425285 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:34.425270 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc6tx\" (UniqueName: \"kubernetes.io/projected/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-kube-api-access-fc6tx\") pod \"maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w\" (UID: \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" Sep 17 01:45:34.526262 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:34.526220 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w\" (UID: \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" Sep 17 01:45:34.526262 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:34.526268 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc6tx\" (UniqueName: \"kubernetes.io/projected/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-kube-api-access-fc6tx\") pod \"maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w\" (UID: \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" Sep 17 01:45:34.526535 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:34.526389 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w\" (UID: \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" Sep 17 01:45:34.526535 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:34.526458 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w\" (UID: \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" Sep 17 01:45:34.526535 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:45:34.526518 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-serving-cert: secret "maas-api-e2e-shared-b-w2-361849-serving-cert" not found Sep 17 01:45:34.526735 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:45:34.526601 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-maas-api-tls podName:02e4bd03-22eb-4bb1-9329-d5e787a8ecc3 nodeName:}" failed. No retries permitted until 2026-09-17 01:45:35.026582632 +0000 UTC m=+1400.261729812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-maas-api-tls") pod "maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" (UID: "02e4bd03-22eb-4bb1-9329-d5e787a8ecc3") : secret "maas-api-e2e-shared-b-w2-361849-serving-cert" not found Sep 17 01:45:34.527015 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:34.526990 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w\" (UID: \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" Sep 17 01:45:34.528891 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:34.528871 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w\" (UID: \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" Sep 17 01:45:34.535879 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:34.535861 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc6tx\" (UniqueName: \"kubernetes.io/projected/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-kube-api-access-fc6tx\") pod \"maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w\" (UID: \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" Sep 17 01:45:35.031963 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:35.031895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w\" (UID: \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" Sep 17 01:45:35.034560 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:35.034527 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w\" (UID: \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" Sep 17 01:45:35.220156 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:35.220116 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" Sep 17 01:45:35.500466 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:35.500438 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w"] Sep 17 01:45:35.501315 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:45:35.501290 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e4bd03_22eb_4bb1_9329_d5e787a8ecc3.slice/crio-e793ad340c1e0b6321fbf14ac03c3a05c49df0cb4078043e2a84683b095f595e WatchSource:0}: Error finding container e793ad340c1e0b6321fbf14ac03c3a05c49df0cb4078043e2a84683b095f595e: Status 404 returned error can't find the container with id e793ad340c1e0b6321fbf14ac03c3a05c49df0cb4078043e2a84683b095f595e Sep 17 01:45:36.110732 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:36.110698 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" event={"ID":"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3","Type":"ContainerStarted","Data":"e793ad340c1e0b6321fbf14ac03c3a05c49df0cb4078043e2a84683b095f595e"} Sep 17 01:45:37.119259 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:37.119219 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" event={"ID":"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3","Type":"ContainerStarted","Data":"45f1abd33e6ab43bc7cda3bd4f4ad37fdc95c3d6e4a770f48c649edff966040c"} Sep 17 01:45:37.119660 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:37.119282 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" Sep 17 01:45:37.136974 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:37.136907 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" podStartSLOduration=2.58894803 podStartE2EDuration="3.136893164s" totalImagesPullingTime="547.945134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:45:37.134042272 +0000 UTC m=+1402.369189460" watchObservedRunningTime="2026-09-17 01:45:37.136893164 +0000 UTC m=+1402.372040352" Sep 17 01:45:43.131674 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:43.131640 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" Sep 17 01:45:58.059260 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:58.059177 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Sep 17 01:45:58.059636 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:58.059262 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" podUID="029db114-e409-4496-b364-f1ebac568192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Sep 17 01:45:58.281855 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:58.281766 2577 generic.go:356] "Generic (PLEG): container finished" podID="029db114-e409-4496-b364-f1ebac568192" containerID="46ad107a814f5ffdcbfba14dee438ee474bde2f8052f80c7dd52ae44479d92ed" exitCode=0 Sep 17 01:45:58.282027 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:58.281845 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" event={"ID":"029db114-e409-4496-b364-f1ebac568192","Type":"ContainerDied","Data":"46ad107a814f5ffdcbfba14dee438ee474bde2f8052f80c7dd52ae44479d92ed"} Sep 17 01:45:58.282363 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:58.282347 2577 scope.go:117] "RemoveContainer" containerID="46ad107a814f5ffdcbfba14dee438ee474bde2f8052f80c7dd52ae44479d92ed" Sep 17 01:45:58.283591 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:58.283557 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:45:58.283652 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:58.283625 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:45:58.283688 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:58.283657 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 01:45:59.290925 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:59.290890 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" event={"ID":"029db114-e409-4496-b364-f1ebac568192","Type":"ContainerStarted","Data":"42b6ec91979e81a9b8f5241068643c3310cc00cc56c0c69dc1a0a87bdaff19cd"} Sep 17 01:45:59.476501 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:59.476467 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:45:59.476692 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:59.476664 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Sep 17 01:45:59.476761 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:45:59.476721 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" podUID="029db114-e409-4496-b364-f1ebac568192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Sep 17 01:46:00.477030 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:46:00.476992 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Sep 17 01:46:00.477406 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:46:00.477055 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" podUID="029db114-e409-4496-b364-f1ebac568192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Sep 17 01:46:01.477535 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:46:01.477499 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Sep 17 01:46:01.477956 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:46:01.477552 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" podUID="029db114-e409-4496-b364-f1ebac568192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Sep 17 01:46:02.477537 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:46:02.477492 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Sep 17 01:46:02.477723 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:46:02.477559 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" podUID="029db114-e409-4496-b364-f1ebac568192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Sep 17 01:46:03.477696 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:46:03.477656 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Sep 17 01:46:03.477883 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:46:03.477722 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" podUID="029db114-e409-4496-b364-f1ebac568192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Sep 17 01:46:04.476950 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:46:04.476894 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Sep 17 01:46:04.477146 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:46:04.476972 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" podUID="029db114-e409-4496-b364-f1ebac568192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Sep 17 01:46:05.476821 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:46:05.476782 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Sep 17 01:46:05.477231 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:46:05.476844 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" podUID="029db114-e409-4496-b364-f1ebac568192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Sep 17 01:46:06.480888 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:46:06.480854 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:46:06.481306 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:46:06.481285 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:46:06.481880 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:46:06.481862 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:46:21.206424 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:46:21.206385 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:46:32.205598 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:46:32.205557 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:47:11.716771 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:11.716731 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz"] Sep 17 01:47:15.281263 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:15.281235 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 01:47:15.284202 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:15.284178 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 01:47:22.454815 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:22.454778 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr"] Sep 17 01:47:22.455287 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:22.455037 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr" podUID="cf6fbf20-070e-4f55-84e1-ff4021d0570a" containerName="payload-processing" containerID="cri-o://5699bfaa435846010a73997d67b0dc08b0004905a71a7dc35ccd352f497b881e" gracePeriod=30 Sep 17 01:47:22.736901 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:22.736873 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr" Sep 17 01:47:22.867386 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:22.867344 2577 generic.go:356] "Generic (PLEG): container finished" podID="cf6fbf20-070e-4f55-84e1-ff4021d0570a" containerID="5699bfaa435846010a73997d67b0dc08b0004905a71a7dc35ccd352f497b881e" exitCode=0 Sep 17 01:47:22.867599 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:22.867396 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr" event={"ID":"cf6fbf20-070e-4f55-84e1-ff4021d0570a","Type":"ContainerDied","Data":"5699bfaa435846010a73997d67b0dc08b0004905a71a7dc35ccd352f497b881e"} Sep 17 01:47:22.867599 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:22.867409 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr" Sep 17 01:47:22.867599 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:22.867446 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr" event={"ID":"cf6fbf20-070e-4f55-84e1-ff4021d0570a","Type":"ContainerDied","Data":"d15c2ca2ead1168647dbed407f696cf326c06974dcc7bc10ea47ae18afc314a6"} Sep 17 01:47:22.867599 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:22.867472 2577 scope.go:117] "RemoveContainer" containerID="5699bfaa435846010a73997d67b0dc08b0004905a71a7dc35ccd352f497b881e" Sep 17 01:47:22.872710 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:22.872588 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzdsr\" (UniqueName: \"kubernetes.io/projected/cf6fbf20-070e-4f55-84e1-ff4021d0570a-kube-api-access-pzdsr\") pod \"cf6fbf20-070e-4f55-84e1-ff4021d0570a\" (UID: \"cf6fbf20-070e-4f55-84e1-ff4021d0570a\") " Sep 17 01:47:22.872710 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:22.872706 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6fbf20-070e-4f55-84e1-ff4021d0570a-plugins-config-volume\") pod \"cf6fbf20-070e-4f55-84e1-ff4021d0570a\" (UID: \"cf6fbf20-070e-4f55-84e1-ff4021d0570a\") " Sep 17 01:47:22.873545 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:22.873517 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf6fbf20-070e-4f55-84e1-ff4021d0570a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cf6fbf20-070e-4f55-84e1-ff4021d0570a" (UID: "cf6fbf20-070e-4f55-84e1-ff4021d0570a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:47:22.876789 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:22.876764 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6fbf20-070e-4f55-84e1-ff4021d0570a-kube-api-access-pzdsr" (OuterVolumeSpecName: "kube-api-access-pzdsr") pod "cf6fbf20-070e-4f55-84e1-ff4021d0570a" (UID: "cf6fbf20-070e-4f55-84e1-ff4021d0570a"). InnerVolumeSpecName "kube-api-access-pzdsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:47:22.881492 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:22.880327 2577 scope.go:117] "RemoveContainer" containerID="5699bfaa435846010a73997d67b0dc08b0004905a71a7dc35ccd352f497b881e" Sep 17 01:47:22.881842 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:47:22.881783 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5699bfaa435846010a73997d67b0dc08b0004905a71a7dc35ccd352f497b881e\": container with ID starting with 5699bfaa435846010a73997d67b0dc08b0004905a71a7dc35ccd352f497b881e not found: ID does not exist" containerID="5699bfaa435846010a73997d67b0dc08b0004905a71a7dc35ccd352f497b881e" Sep 17 01:47:22.881842 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:22.881825 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5699bfaa435846010a73997d67b0dc08b0004905a71a7dc35ccd352f497b881e"} err="failed to get container status \"5699bfaa435846010a73997d67b0dc08b0004905a71a7dc35ccd352f497b881e\": rpc error: code = NotFound desc = could not find container \"5699bfaa435846010a73997d67b0dc08b0004905a71a7dc35ccd352f497b881e\": container with ID starting with 5699bfaa435846010a73997d67b0dc08b0004905a71a7dc35ccd352f497b881e not found: ID does not exist" Sep 17 01:47:22.974334 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:22.974225 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzdsr\" (UniqueName: \"kubernetes.io/projected/cf6fbf20-070e-4f55-84e1-ff4021d0570a-kube-api-access-pzdsr\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:47:22.974334 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:22.974257 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6fbf20-070e-4f55-84e1-ff4021d0570a-plugins-config-volume\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:47:23.191748 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:23.191704 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr"] Sep 17 01:47:23.194975 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:23.194944 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cc98e0-56597c87f7-q2njr"] Sep 17 01:47:23.213798 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:23.213763 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf6fbf20-070e-4f55-84e1-ff4021d0570a" path="/var/lib/kubelet/pods/cf6fbf20-070e-4f55-84e1-ff4021d0570a/volumes" Sep 17 01:47:26.737377 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:26.737326 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" podUID="e8f10f54-d924-4598-aff2-020ddeb3d487" containerName="main" containerID="cri-o://f1827c0788d34018a9aa579e132d7de176455c9ecb2a9ddb98ca3f1239081a57" gracePeriod=45 Sep 17 01:47:26.897159 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:26.897127 2577 generic.go:356] "Generic (PLEG): container finished" podID="e8f10f54-d924-4598-aff2-020ddeb3d487" containerID="f1827c0788d34018a9aa579e132d7de176455c9ecb2a9ddb98ca3f1239081a57" exitCode=0 Sep 17 01:47:26.897312 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:26.897197 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" event={"ID":"e8f10f54-d924-4598-aff2-020ddeb3d487","Type":"ContainerDied","Data":"f1827c0788d34018a9aa579e132d7de176455c9ecb2a9ddb98ca3f1239081a57"} Sep 17 01:47:26.989434 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:26.989374 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:47:27.116701 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.116665 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-tmp-dir\") pod \"e8f10f54-d924-4598-aff2-020ddeb3d487\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " Sep 17 01:47:27.116851 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.116730 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-dshm\") pod \"e8f10f54-d924-4598-aff2-020ddeb3d487\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " Sep 17 01:47:27.116851 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.116781 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-model-cache\") pod \"e8f10f54-d924-4598-aff2-020ddeb3d487\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " Sep 17 01:47:27.116851 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.116834 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-home\") pod \"e8f10f54-d924-4598-aff2-020ddeb3d487\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " Sep 17 01:47:27.117048 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.116889 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f10f54-d924-4598-aff2-020ddeb3d487-tls-certs\") pod \"e8f10f54-d924-4598-aff2-020ddeb3d487\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " Sep 17 01:47:27.117048 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.116925 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tm7mq\" (UniqueName: \"kubernetes.io/projected/e8f10f54-d924-4598-aff2-020ddeb3d487-kube-api-access-tm7mq\") pod \"e8f10f54-d924-4598-aff2-020ddeb3d487\" (UID: \"e8f10f54-d924-4598-aff2-020ddeb3d487\") " Sep 17 01:47:27.117048 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.117026 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8f10f54-d924-4598-aff2-020ddeb3d487" (UID: "e8f10f54-d924-4598-aff2-020ddeb3d487"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:47:27.117210 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.117041 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8f10f54-d924-4598-aff2-020ddeb3d487" (UID: "e8f10f54-d924-4598-aff2-020ddeb3d487"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:47:27.117274 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.117205 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-home" (OuterVolumeSpecName: "home") pod "e8f10f54-d924-4598-aff2-020ddeb3d487" (UID: "e8f10f54-d924-4598-aff2-020ddeb3d487"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:47:27.117385 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.117367 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-tmp-dir\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:47:27.117445 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.117392 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-model-cache\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:47:27.117445 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.117407 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-home\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:47:27.119180 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.119147 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-dshm" (OuterVolumeSpecName: "dshm") pod "e8f10f54-d924-4598-aff2-020ddeb3d487" (UID: "e8f10f54-d924-4598-aff2-020ddeb3d487"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:47:27.119478 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.119455 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f10f54-d924-4598-aff2-020ddeb3d487-kube-api-access-tm7mq" (OuterVolumeSpecName: "kube-api-access-tm7mq") pod "e8f10f54-d924-4598-aff2-020ddeb3d487" (UID: "e8f10f54-d924-4598-aff2-020ddeb3d487"). InnerVolumeSpecName "kube-api-access-tm7mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:47:27.119478 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.119462 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8f10f54-d924-4598-aff2-020ddeb3d487-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8f10f54-d924-4598-aff2-020ddeb3d487" (UID: "e8f10f54-d924-4598-aff2-020ddeb3d487"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:47:27.217979 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.217948 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f10f54-d924-4598-aff2-020ddeb3d487-dshm\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:47:27.217979 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.217975 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f10f54-d924-4598-aff2-020ddeb3d487-tls-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:47:27.217979 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.217989 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm7mq\" (UniqueName: \"kubernetes.io/projected/e8f10f54-d924-4598-aff2-020ddeb3d487-kube-api-access-tm7mq\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:47:27.904911 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.904880 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" Sep 17 01:47:27.905334 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.904910 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz" event={"ID":"e8f10f54-d924-4598-aff2-020ddeb3d487","Type":"ContainerDied","Data":"4e48227a4243fc826e9e4a8c7c261b3b41073920c548aed7722f33dc0b209f5c"} Sep 17 01:47:27.905334 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.904971 2577 scope.go:117] "RemoveContainer" containerID="f1827c0788d34018a9aa579e132d7de176455c9ecb2a9ddb98ca3f1239081a57" Sep 17 01:47:27.924550 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.924524 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz"] Sep 17 01:47:27.927544 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:27.927505 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cc98e0/premium-w3-cc98e0-kserve-65c74f5987-szdwz"] Sep 17 01:47:28.873199 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:28.873164 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92"] Sep 17 01:47:28.873412 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:28.873385 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" podUID="fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" containerName="istio-proxy" containerID="cri-o://6804637bdbe70e3b30e2b2494789334eb256a729d459044523794cfddcd10167" gracePeriod=30 Sep 17 01:47:29.213470 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:29.213386 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8f10f54-d924-4598-aff2-020ddeb3d487" path="/var/lib/kubelet/pods/e8f10f54-d924-4598-aff2-020ddeb3d487/volumes" Sep 17 01:47:31.946232 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:31.946193 2577 patch_prober.go:28] interesting pod/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 01:47:31.946597 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:31.946264 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" podUID="fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 01:47:34.139877 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.139811 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:47:34.193347 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.193315 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-credential-socket\") pod \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " Sep 17 01:47:34.193512 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.193350 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-workload-socket\") pod \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " Sep 17 01:47:34.193512 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.193382 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-podinfo\") pod \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " Sep 17 01:47:34.193512 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.193429 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-workload-certs\") pod \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " Sep 17 01:47:34.193512 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.193462 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istiod-ca-cert\") pod \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " Sep 17 01:47:34.193743 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.193585 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" (UID: "fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:47:34.193743 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.193603 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-envoy\") pod \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " Sep 17 01:47:34.193743 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.193606 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" (UID: "fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:47:34.193743 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.193650 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-token\") pod \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " Sep 17 01:47:34.193743 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.193677 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqmqn\" (UniqueName: \"kubernetes.io/projected/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-kube-api-access-jqmqn\") pod \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " Sep 17 01:47:34.193743 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.193715 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" (UID: "fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:47:34.194083 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.193761 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-data\") pod \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\" (UID: \"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1\") " Sep 17 01:47:34.194083 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.193796 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" (UID: "fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:47:34.194190 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.194100 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-data" (OuterVolumeSpecName: "istio-data") pod "fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" (UID: "fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:47:34.194248 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.194216 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-data\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:47:34.194248 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.194237 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-credential-socket\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:47:34.194352 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.194252 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-workload-socket\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:47:34.194352 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.194265 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-workload-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:47:34.194352 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.194277 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istiod-ca-cert\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:47:34.195790 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.195761 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" (UID: "fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 01:47:34.195908 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.195855 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" (UID: "fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:47:34.195908 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.195885 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-kube-api-access-jqmqn" (OuterVolumeSpecName: "kube-api-access-jqmqn") pod "fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" (UID: "fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1"). InnerVolumeSpecName "kube-api-access-jqmqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:47:34.196015 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.195975 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-token" (OuterVolumeSpecName: "istio-token") pod "fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" (UID: "fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:47:34.295246 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.295212 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-envoy\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:47:34.295246 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.295246 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-token\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:47:34.295454 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.295259 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqmqn\" (UniqueName: \"kubernetes.io/projected/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-kube-api-access-jqmqn\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:47:34.295454 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.295272 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1-istio-podinfo\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:47:34.952456 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.952420 2577 generic.go:356] "Generic (PLEG): container finished" podID="fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" containerID="6804637bdbe70e3b30e2b2494789334eb256a729d459044523794cfddcd10167" exitCode=0 Sep 17 01:47:34.952789 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.952478 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" Sep 17 01:47:34.952789 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.952544 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" event={"ID":"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1","Type":"ContainerDied","Data":"6804637bdbe70e3b30e2b2494789334eb256a729d459044523794cfddcd10167"} Sep 17 01:47:34.952789 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.952623 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92" event={"ID":"fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1","Type":"ContainerDied","Data":"b8079016b4475e1468ee2f5408f4625dbdfd878e86cd2104cd4d4a7e782d266f"} Sep 17 01:47:34.952789 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.952646 2577 scope.go:117] "RemoveContainer" containerID="6804637bdbe70e3b30e2b2494789334eb256a729d459044523794cfddcd10167" Sep 17 01:47:34.966768 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.966749 2577 scope.go:117] "RemoveContainer" containerID="5f07311e4c731682d82c04d67c9ced9ece88b36003bf62c37f0b98cd0c39f5c1" Sep 17 01:47:34.977004 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.976982 2577 scope.go:117] "RemoveContainer" containerID="6804637bdbe70e3b30e2b2494789334eb256a729d459044523794cfddcd10167" Sep 17 01:47:34.977347 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:47:34.977324 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6804637bdbe70e3b30e2b2494789334eb256a729d459044523794cfddcd10167\": container with ID starting with 6804637bdbe70e3b30e2b2494789334eb256a729d459044523794cfddcd10167 not found: ID does not exist" containerID="6804637bdbe70e3b30e2b2494789334eb256a729d459044523794cfddcd10167" Sep 17 01:47:34.977420 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.977354 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6804637bdbe70e3b30e2b2494789334eb256a729d459044523794cfddcd10167"} err="failed to get container status \"6804637bdbe70e3b30e2b2494789334eb256a729d459044523794cfddcd10167\": rpc error: code = NotFound desc = could not find container \"6804637bdbe70e3b30e2b2494789334eb256a729d459044523794cfddcd10167\": container with ID starting with 6804637bdbe70e3b30e2b2494789334eb256a729d459044523794cfddcd10167 not found: ID does not exist" Sep 17 01:47:34.977420 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.977371 2577 scope.go:117] "RemoveContainer" containerID="5f07311e4c731682d82c04d67c9ced9ece88b36003bf62c37f0b98cd0c39f5c1" Sep 17 01:47:34.977733 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:47:34.977704 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f07311e4c731682d82c04d67c9ced9ece88b36003bf62c37f0b98cd0c39f5c1\": container with ID starting with 5f07311e4c731682d82c04d67c9ced9ece88b36003bf62c37f0b98cd0c39f5c1 not found: ID does not exist" containerID="5f07311e4c731682d82c04d67c9ced9ece88b36003bf62c37f0b98cd0c39f5c1" Sep 17 01:47:34.977789 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.977741 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f07311e4c731682d82c04d67c9ced9ece88b36003bf62c37f0b98cd0c39f5c1"} err="failed to get container status \"5f07311e4c731682d82c04d67c9ced9ece88b36003bf62c37f0b98cd0c39f5c1\": rpc error: code = NotFound desc = could not find container \"5f07311e4c731682d82c04d67c9ced9ece88b36003bf62c37f0b98cd0c39f5c1\": container with ID starting with 5f07311e4c731682d82c04d67c9ced9ece88b36003bf62c37f0b98cd0c39f5c1 not found: ID does not exist" Sep 17 01:47:34.978917 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.978883 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92"] Sep 17 01:47:34.981738 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:34.981719 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cc98e0-openshift-default-656b649fd6-wrk92"] Sep 17 01:47:35.215510 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:35.215424 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" path="/var/lib/kubelet/pods/fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1/volumes" Sep 17 01:47:36.969243 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:36.969209 2577 generic.go:356] "Generic (PLEG): container finished" podID="36a411dd-11a4-4abb-9ff1-677dfc09cac7" containerID="c297fc2df366a5cc16cb060a7c457fde28758e19f602ca5a98349403c0bd9093" exitCode=137 Sep 17 01:47:36.969640 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:36.969280 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" event={"ID":"36a411dd-11a4-4abb-9ff1-677dfc09cac7","Type":"ContainerDied","Data":"c297fc2df366a5cc16cb060a7c457fde28758e19f602ca5a98349403c0bd9093"} Sep 17 01:47:36.969708 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:36.969693 2577 scope.go:117] "RemoveContainer" containerID="c297fc2df366a5cc16cb060a7c457fde28758e19f602ca5a98349403c0bd9093" Sep 17 01:47:37.977815 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:37.977727 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" event={"ID":"36a411dd-11a4-4abb-9ff1-677dfc09cac7","Type":"ContainerStarted","Data":"a1fd8160f6e45518e92d738340a41873976e3ffa64082c4041d9dc10fa7b394c"} Sep 17 01:47:37.978239 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:37.977953 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" Sep 17 01:47:38.206379 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:38.206347 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:47:48.987355 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:48.987319 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" Sep 17 01:47:50.981781 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:50.981747 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj"] Sep 17 01:47:50.982249 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:50.982233 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf6fbf20-070e-4f55-84e1-ff4021d0570a" containerName="payload-processing" Sep 17 01:47:50.982299 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:50.982250 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf6fbf20-070e-4f55-84e1-ff4021d0570a" containerName="payload-processing" Sep 17 01:47:50.982299 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:50.982262 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" containerName="istio-proxy" Sep 17 01:47:50.982299 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:50.982267 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" containerName="istio-proxy" Sep 17 01:47:50.982299 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:50.982289 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" containerName="istio-proxy" Sep 17 01:47:50.982299 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:50.982295 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" containerName="istio-proxy" Sep 17 01:47:50.982457 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:50.982304 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8f10f54-d924-4598-aff2-020ddeb3d487" containerName="main" Sep 17 01:47:50.982457 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:50.982309 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8f10f54-d924-4598-aff2-020ddeb3d487" containerName="main" Sep 17 01:47:50.982457 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:50.982391 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" containerName="istio-proxy" Sep 17 01:47:50.982457 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:50.982400 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8f10f54-d924-4598-aff2-020ddeb3d487" containerName="main" Sep 17 01:47:50.982457 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:50.982406 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdfe78aa-ff8e-4e7b-9ca7-84d1598bfdf1" containerName="istio-proxy" Sep 17 01:47:50.982457 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:50.982415 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf6fbf20-070e-4f55-84e1-ff4021d0570a" containerName="payload-processing" Sep 17 01:47:50.985668 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:50.985647 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:50.988685 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:50.988662 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-8be734\"/\"kube-root-ca.crt\"" Sep 17 01:47:50.988818 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:50.988690 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-8be734\"/\"openshift-service-ca.crt\"" Sep 17 01:47:50.988818 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:50.988690 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8be734\"/\"embedding-w0-8be734-kserve-self-signed-certs\"" Sep 17 01:47:50.988818 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:50.988666 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8be734\"/\"default-dockercfg-fx72h\"" Sep 17 01:47:50.995513 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:50.995493 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj"] Sep 17 01:47:51.065234 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.065204 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-model-cache\") pod \"embedding-w0-8be734-kserve-6ddff6598c-vc7jj\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:51.065398 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.065346 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7pfw\" (UniqueName: \"kubernetes.io/projected/c720f2fb-2303-4994-b363-4880c0ed144e-kube-api-access-z7pfw\") pod \"embedding-w0-8be734-kserve-6ddff6598c-vc7jj\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:51.065398 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.065373 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-home\") pod \"embedding-w0-8be734-kserve-6ddff6598c-vc7jj\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:51.065398 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.065392 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-dshm\") pod \"embedding-w0-8be734-kserve-6ddff6598c-vc7jj\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:51.065526 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.065429 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c720f2fb-2303-4994-b363-4880c0ed144e-tls-certs\") pod \"embedding-w0-8be734-kserve-6ddff6598c-vc7jj\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:51.065571 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.065532 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-tmp-dir\") pod \"embedding-w0-8be734-kserve-6ddff6598c-vc7jj\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:51.166493 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.166459 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-tmp-dir\") pod \"embedding-w0-8be734-kserve-6ddff6598c-vc7jj\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:51.166493 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.166492 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-model-cache\") pod \"embedding-w0-8be734-kserve-6ddff6598c-vc7jj\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:51.166683 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.166558 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7pfw\" (UniqueName: \"kubernetes.io/projected/c720f2fb-2303-4994-b363-4880c0ed144e-kube-api-access-z7pfw\") pod \"embedding-w0-8be734-kserve-6ddff6598c-vc7jj\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:51.166683 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.166671 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-home\") pod \"embedding-w0-8be734-kserve-6ddff6598c-vc7jj\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:51.166778 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.166703 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-dshm\") pod \"embedding-w0-8be734-kserve-6ddff6598c-vc7jj\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:51.166778 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.166750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c720f2fb-2303-4994-b363-4880c0ed144e-tls-certs\") pod \"embedding-w0-8be734-kserve-6ddff6598c-vc7jj\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:51.166861 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.166772 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-tmp-dir\") pod \"embedding-w0-8be734-kserve-6ddff6598c-vc7jj\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:51.166861 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.166825 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-model-cache\") pod \"embedding-w0-8be734-kserve-6ddff6598c-vc7jj\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:51.166989 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.166973 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-home\") pod \"embedding-w0-8be734-kserve-6ddff6598c-vc7jj\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:51.168864 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.168840 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-dshm\") pod \"embedding-w0-8be734-kserve-6ddff6598c-vc7jj\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:51.169153 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.169134 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c720f2fb-2303-4994-b363-4880c0ed144e-tls-certs\") pod \"embedding-w0-8be734-kserve-6ddff6598c-vc7jj\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:51.173543 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.173522 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7pfw\" (UniqueName: \"kubernetes.io/projected/c720f2fb-2303-4994-b363-4880c0ed144e-kube-api-access-z7pfw\") pod \"embedding-w0-8be734-kserve-6ddff6598c-vc7jj\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:51.295984 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.295867 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:47:51.568906 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:51.568875 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj"] Sep 17 01:47:51.569309 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:47:51.569282 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc720f2fb_2303_4994_b363_4880c0ed144e.slice/crio-07ad22cec224ac5ce48dbbf6efe4e792e0fd9fcb88863ae6c4c9a38cee082939 WatchSource:0}: Error finding container 07ad22cec224ac5ce48dbbf6efe4e792e0fd9fcb88863ae6c4c9a38cee082939: Status 404 returned error can't find the container with id 07ad22cec224ac5ce48dbbf6efe4e792e0fd9fcb88863ae6c4c9a38cee082939 Sep 17 01:47:52.072483 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:52.072437 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" event={"ID":"c720f2fb-2303-4994-b363-4880c0ed144e","Type":"ContainerStarted","Data":"0bdbce104d94d55898a05e0f2f2785b45aaa7aed2ce9b4ff28ffe5d0cd868499"} Sep 17 01:47:52.072832 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:52.072491 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" event={"ID":"c720f2fb-2303-4994-b363-4880c0ed144e","Type":"ContainerStarted","Data":"07ad22cec224ac5ce48dbbf6efe4e792e0fd9fcb88863ae6c4c9a38cee082939"} Sep 17 01:47:52.087885 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:52.087836 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" podStartSLOduration=2.08782206 podStartE2EDuration="2.08782206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:47:52.085666994 +0000 UTC m=+1537.320814181" watchObservedRunningTime="2026-09-17 01:47:52.08782206 +0000 UTC m=+1537.322969248" Sep 17 01:47:55.210040 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:47:55.210002 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:48:01.295999 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:48:01.295927 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:48:01.295999 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:48:01.296006 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:48:01.308744 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:48:01.308716 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:48:02.150642 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:48:02.150605 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:48:48.206498 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:48:48.206458 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:49:11.607445 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:11.607351 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj"] Sep 17 01:49:22.553148 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:22.553092 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt"] Sep 17 01:49:22.553818 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:22.553400 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt" podUID="b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6" containerName="payload-processing" containerID="cri-o://c31983f21c84e7fe5412dbc9e4002d994f8036c703fc7a55a596872eb36ee7c9" gracePeriod=30 Sep 17 01:49:22.561905 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:22.561804 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d"] Sep 17 01:49:22.562105 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:22.562051 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d" podUID="81a31583-90a7-4f7a-90e0-4870220056b1" containerName="payload-pre-processing" containerID="cri-o://8d2d294ce7da059cdfe09f9f3bd21c6b1b86bad2ca74c54a7fc28c634bd3b848" gracePeriod=30 Sep 17 01:49:22.685519 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:22.685483 2577 generic.go:356] "Generic (PLEG): container finished" podID="81a31583-90a7-4f7a-90e0-4870220056b1" containerID="8d2d294ce7da059cdfe09f9f3bd21c6b1b86bad2ca74c54a7fc28c634bd3b848" exitCode=0 Sep 17 01:49:22.685687 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:22.685562 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d" event={"ID":"81a31583-90a7-4f7a-90e0-4870220056b1","Type":"ContainerDied","Data":"8d2d294ce7da059cdfe09f9f3bd21c6b1b86bad2ca74c54a7fc28c634bd3b848"} Sep 17 01:49:22.687655 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:22.687620 2577 generic.go:356] "Generic (PLEG): container finished" podID="b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6" containerID="c31983f21c84e7fe5412dbc9e4002d994f8036c703fc7a55a596872eb36ee7c9" exitCode=0 Sep 17 01:49:22.687877 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:22.687677 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt" event={"ID":"b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6","Type":"ContainerDied","Data":"c31983f21c84e7fe5412dbc9e4002d994f8036c703fc7a55a596872eb36ee7c9"} Sep 17 01:49:22.842886 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:22.842861 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt" Sep 17 01:49:22.856634 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:22.856612 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d" Sep 17 01:49:22.963912 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:22.963878 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a31583-90a7-4f7a-90e0-4870220056b1-plugins-config-volume\") pod \"81a31583-90a7-4f7a-90e0-4870220056b1\" (UID: \"81a31583-90a7-4f7a-90e0-4870220056b1\") " Sep 17 01:49:22.964095 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:22.963960 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lglz\" (UniqueName: \"kubernetes.io/projected/b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6-kube-api-access-4lglz\") pod \"b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6\" (UID: \"b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6\") " Sep 17 01:49:22.964095 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:22.964023 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6-plugins-config-volume\") pod \"b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6\" (UID: \"b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6\") " Sep 17 01:49:22.964095 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:22.964053 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dghf7\" (UniqueName: \"kubernetes.io/projected/81a31583-90a7-4f7a-90e0-4870220056b1-kube-api-access-dghf7\") pod \"81a31583-90a7-4f7a-90e0-4870220056b1\" (UID: \"81a31583-90a7-4f7a-90e0-4870220056b1\") " Sep 17 01:49:22.964296 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:22.964268 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a31583-90a7-4f7a-90e0-4870220056b1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "81a31583-90a7-4f7a-90e0-4870220056b1" (UID: "81a31583-90a7-4f7a-90e0-4870220056b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:49:22.964392 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:22.964370 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6" (UID: "b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:49:22.964470 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:22.964455 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a31583-90a7-4f7a-90e0-4870220056b1-plugins-config-volume\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:49:22.966154 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:22.966132 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a31583-90a7-4f7a-90e0-4870220056b1-kube-api-access-dghf7" (OuterVolumeSpecName: "kube-api-access-dghf7") pod "81a31583-90a7-4f7a-90e0-4870220056b1" (UID: "81a31583-90a7-4f7a-90e0-4870220056b1"). InnerVolumeSpecName "kube-api-access-dghf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:49:22.966205 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:22.966187 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6-kube-api-access-4lglz" (OuterVolumeSpecName: "kube-api-access-4lglz") pod "b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6" (UID: "b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6"). InnerVolumeSpecName "kube-api-access-4lglz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:49:23.065741 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:23.065701 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lglz\" (UniqueName: \"kubernetes.io/projected/b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6-kube-api-access-4lglz\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:49:23.065741 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:23.065733 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6-plugins-config-volume\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:49:23.065741 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:23.065744 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dghf7\" (UniqueName: \"kubernetes.io/projected/81a31583-90a7-4f7a-90e0-4870220056b1-kube-api-access-dghf7\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:49:23.206219 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:23.206179 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:49:23.695612 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:23.695583 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt" Sep 17 01:49:23.696085 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:23.695620 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt" event={"ID":"b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6","Type":"ContainerDied","Data":"282fccdc63f0a4d273b21ec610c177b3a0245e1a33f14f3dcf1e714be4f58282"} Sep 17 01:49:23.696085 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:23.695682 2577 scope.go:117] "RemoveContainer" containerID="c31983f21c84e7fe5412dbc9e4002d994f8036c703fc7a55a596872eb36ee7c9" Sep 17 01:49:23.697284 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:23.697250 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d" Sep 17 01:49:23.697284 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:23.697269 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d" event={"ID":"81a31583-90a7-4f7a-90e0-4870220056b1","Type":"ContainerDied","Data":"09d2b115e225fb9c7beb363a270f5853ba07676784c798645406b0f2eb18dd78"} Sep 17 01:49:23.706027 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:23.706008 2577 scope.go:117] "RemoveContainer" containerID="8d2d294ce7da059cdfe09f9f3bd21c6b1b86bad2ca74c54a7fc28c634bd3b848" Sep 17 01:49:23.715361 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:23.715344 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d"] Sep 17 01:49:23.718869 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:23.718843 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-8be734-777dc4db98-h6c5d"] Sep 17 01:49:23.727382 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:23.727359 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt"] Sep 17 01:49:23.729919 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:23.729894 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-8be734-776b56bf6f-x6zdt"] Sep 17 01:49:25.213560 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:25.213520 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81a31583-90a7-4f7a-90e0-4870220056b1" path="/var/lib/kubelet/pods/81a31583-90a7-4f7a-90e0-4870220056b1/volumes" Sep 17 01:49:25.214004 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:25.213987 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6" path="/var/lib/kubelet/pods/b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6/volumes" Sep 17 01:49:26.627772 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:26.627725 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" podUID="c720f2fb-2303-4994-b363-4880c0ed144e" containerName="main" containerID="cri-o://0bdbce104d94d55898a05e0f2f2785b45aaa7aed2ce9b4ff28ffe5d0cd868499" gracePeriod=45 Sep 17 01:49:26.890451 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:26.890386 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:49:26.899977 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:26.899940 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-tmp-dir\") pod \"c720f2fb-2303-4994-b363-4880c0ed144e\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " Sep 17 01:49:26.900145 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:26.900026 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-model-cache\") pod \"c720f2fb-2303-4994-b363-4880c0ed144e\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " Sep 17 01:49:26.900145 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:26.900058 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c720f2fb-2303-4994-b363-4880c0ed144e-tls-certs\") pod \"c720f2fb-2303-4994-b363-4880c0ed144e\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " Sep 17 01:49:26.900145 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:26.900100 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-dshm\") pod \"c720f2fb-2303-4994-b363-4880c0ed144e\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " Sep 17 01:49:26.900145 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:26.900144 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7pfw\" (UniqueName: \"kubernetes.io/projected/c720f2fb-2303-4994-b363-4880c0ed144e-kube-api-access-z7pfw\") pod \"c720f2fb-2303-4994-b363-4880c0ed144e\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " Sep 17 01:49:26.900374 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:26.900139 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c720f2fb-2303-4994-b363-4880c0ed144e" (UID: "c720f2fb-2303-4994-b363-4880c0ed144e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:49:26.900374 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:26.900166 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-home\") pod \"c720f2fb-2303-4994-b363-4880c0ed144e\" (UID: \"c720f2fb-2303-4994-b363-4880c0ed144e\") " Sep 17 01:49:26.900374 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:26.900309 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-model-cache" (OuterVolumeSpecName: "model-cache") pod "c720f2fb-2303-4994-b363-4880c0ed144e" (UID: "c720f2fb-2303-4994-b363-4880c0ed144e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:49:26.900499 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:26.900476 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-model-cache\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:49:26.900499 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:26.900496 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-tmp-dir\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:49:26.900566 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:26.900515 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-home" (OuterVolumeSpecName: "home") pod "c720f2fb-2303-4994-b363-4880c0ed144e" (UID: "c720f2fb-2303-4994-b363-4880c0ed144e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:49:26.902546 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:26.902519 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c720f2fb-2303-4994-b363-4880c0ed144e-kube-api-access-z7pfw" (OuterVolumeSpecName: "kube-api-access-z7pfw") pod "c720f2fb-2303-4994-b363-4880c0ed144e" (UID: "c720f2fb-2303-4994-b363-4880c0ed144e"). InnerVolumeSpecName "kube-api-access-z7pfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:49:26.902675 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:26.902652 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-dshm" (OuterVolumeSpecName: "dshm") pod "c720f2fb-2303-4994-b363-4880c0ed144e" (UID: "c720f2fb-2303-4994-b363-4880c0ed144e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:49:26.902720 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:26.902652 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c720f2fb-2303-4994-b363-4880c0ed144e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c720f2fb-2303-4994-b363-4880c0ed144e" (UID: "c720f2fb-2303-4994-b363-4880c0ed144e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:49:27.001051 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:27.001013 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7pfw\" (UniqueName: \"kubernetes.io/projected/c720f2fb-2303-4994-b363-4880c0ed144e-kube-api-access-z7pfw\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:49:27.001051 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:27.001044 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-home\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:49:27.001051 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:27.001055 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c720f2fb-2303-4994-b363-4880c0ed144e-tls-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:49:27.001275 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:27.001065 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c720f2fb-2303-4994-b363-4880c0ed144e-dshm\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:49:27.729226 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:27.729185 2577 generic.go:356] "Generic (PLEG): container finished" podID="c720f2fb-2303-4994-b363-4880c0ed144e" containerID="0bdbce104d94d55898a05e0f2f2785b45aaa7aed2ce9b4ff28ffe5d0cd868499" exitCode=0 Sep 17 01:49:27.729667 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:27.729230 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" Sep 17 01:49:27.729667 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:27.729267 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" event={"ID":"c720f2fb-2303-4994-b363-4880c0ed144e","Type":"ContainerDied","Data":"0bdbce104d94d55898a05e0f2f2785b45aaa7aed2ce9b4ff28ffe5d0cd868499"} Sep 17 01:49:27.729667 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:27.729311 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj" event={"ID":"c720f2fb-2303-4994-b363-4880c0ed144e","Type":"ContainerDied","Data":"07ad22cec224ac5ce48dbbf6efe4e792e0fd9fcb88863ae6c4c9a38cee082939"} Sep 17 01:49:27.729667 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:27.729334 2577 scope.go:117] "RemoveContainer" containerID="0bdbce104d94d55898a05e0f2f2785b45aaa7aed2ce9b4ff28ffe5d0cd868499" Sep 17 01:49:27.739893 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:27.739868 2577 scope.go:117] "RemoveContainer" containerID="0bdbce104d94d55898a05e0f2f2785b45aaa7aed2ce9b4ff28ffe5d0cd868499" Sep 17 01:49:27.740257 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:49:27.740234 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bdbce104d94d55898a05e0f2f2785b45aaa7aed2ce9b4ff28ffe5d0cd868499\": container with ID starting with 0bdbce104d94d55898a05e0f2f2785b45aaa7aed2ce9b4ff28ffe5d0cd868499 not found: ID does not exist" containerID="0bdbce104d94d55898a05e0f2f2785b45aaa7aed2ce9b4ff28ffe5d0cd868499" Sep 17 01:49:27.740356 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:27.740265 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bdbce104d94d55898a05e0f2f2785b45aaa7aed2ce9b4ff28ffe5d0cd868499"} err="failed to get container status \"0bdbce104d94d55898a05e0f2f2785b45aaa7aed2ce9b4ff28ffe5d0cd868499\": rpc error: code = NotFound desc = could not find container \"0bdbce104d94d55898a05e0f2f2785b45aaa7aed2ce9b4ff28ffe5d0cd868499\": container with ID starting with 0bdbce104d94d55898a05e0f2f2785b45aaa7aed2ce9b4ff28ffe5d0cd868499 not found: ID does not exist" Sep 17 01:49:27.746877 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:27.746851 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj"] Sep 17 01:49:27.751243 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:27.751213 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8be734/embedding-w0-8be734-kserve-6ddff6598c-vc7jj"] Sep 17 01:49:29.213971 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:49:29.213914 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c720f2fb-2303-4994-b363-4880c0ed144e" path="/var/lib/kubelet/pods/c720f2fb-2303-4994-b363-4880c0ed144e/volumes" Sep 17 01:50:03.206034 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:03.205999 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:50:22.357794 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:22.357755 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-10047-fced50"] Sep 17 01:50:22.358262 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:22.358247 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6" containerName="payload-processing" Sep 17 01:50:22.358262 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:22.358263 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6" containerName="payload-processing" Sep 17 01:50:22.358346 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:22.358276 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81a31583-90a7-4f7a-90e0-4870220056b1" containerName="payload-pre-processing" Sep 17 01:50:22.358346 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:22.358281 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="81a31583-90a7-4f7a-90e0-4870220056b1" containerName="payload-pre-processing" Sep 17 01:50:22.358346 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:22.358290 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c720f2fb-2303-4994-b363-4880c0ed144e" containerName="main" Sep 17 01:50:22.358346 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:22.358296 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c720f2fb-2303-4994-b363-4880c0ed144e" containerName="main" Sep 17 01:50:22.358471 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:22.358375 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="81a31583-90a7-4f7a-90e0-4870220056b1" containerName="payload-pre-processing" Sep 17 01:50:22.358471 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:22.358387 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c720f2fb-2303-4994-b363-4880c0ed144e" containerName="main" Sep 17 01:50:22.358471 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:22.358396 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2d0bc32-5bcd-4cc0-8e45-558b9b9356b6" containerName="payload-processing" Sep 17 01:50:22.361728 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:22.361707 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10047-fced50" Sep 17 01:50:22.363892 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:22.363868 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qszjr\"" Sep 17 01:50:22.368681 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:22.368649 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10047-fced50"] Sep 17 01:50:22.440630 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:22.440595 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4vrg\" (UniqueName: \"kubernetes.io/projected/d52f8321-1164-452e-a65a-0995c084a100-kube-api-access-c4vrg\") pod \"test-curl-10047-fced50\" (UID: \"d52f8321-1164-452e-a65a-0995c084a100\") " pod="opendatahub/test-curl-10047-fced50" Sep 17 01:50:22.542192 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:22.542151 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4vrg\" (UniqueName: \"kubernetes.io/projected/d52f8321-1164-452e-a65a-0995c084a100-kube-api-access-c4vrg\") pod \"test-curl-10047-fced50\" (UID: \"d52f8321-1164-452e-a65a-0995c084a100\") " pod="opendatahub/test-curl-10047-fced50" Sep 17 01:50:22.550279 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:22.550246 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4vrg\" (UniqueName: \"kubernetes.io/projected/d52f8321-1164-452e-a65a-0995c084a100-kube-api-access-c4vrg\") pod \"test-curl-10047-fced50\" (UID: \"d52f8321-1164-452e-a65a-0995c084a100\") " pod="opendatahub/test-curl-10047-fced50" Sep 17 01:50:22.673879 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:22.673793 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10047-fced50" Sep 17 01:50:22.950851 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:22.950820 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-10047-fced50"] Sep 17 01:50:22.952002 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:50:22.951975 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd52f8321_1164_452e_a65a_0995c084a100.slice/crio-d7ea15bcc6a5d04a6d54db6b1f76b1a666f6f8dd741983de0e6b673b955b95a1 WatchSource:0}: Error finding container d7ea15bcc6a5d04a6d54db6b1f76b1a666f6f8dd741983de0e6b673b955b95a1: Status 404 returned error can't find the container with id d7ea15bcc6a5d04a6d54db6b1f76b1a666f6f8dd741983de0e6b673b955b95a1 Sep 17 01:50:23.105478 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:23.105434 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10047-fced50" event={"ID":"d52f8321-1164-452e-a65a-0995c084a100","Type":"ContainerStarted","Data":"0d45eeb4d2b310ec629fd8420940b4c81fa72b4881dffcb9013511e6c6852905"} Sep 17 01:50:23.105656 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:23.105490 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-10047-fced50" event={"ID":"d52f8321-1164-452e-a65a-0995c084a100","Type":"ContainerStarted","Data":"d7ea15bcc6a5d04a6d54db6b1f76b1a666f6f8dd741983de0e6b673b955b95a1"} Sep 17 01:50:23.124321 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:23.123642 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-10047-fced50" podStartSLOduration=1.123618838 podStartE2EDuration="1.123618838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:50:23.121082112 +0000 UTC m=+1688.356229300" watchObservedRunningTime="2026-09-17 01:50:23.123618838 +0000 UTC m=+1688.358766050" Sep 17 01:50:23.714851 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:23.714815 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-10047-fced50"] Sep 17 01:50:23.719190 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:23.719151 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-10047-fced50"] Sep 17 01:50:24.113022 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:50:24.112991 2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-10047-fced50" container="test-curl-10047-fced50" Sep 17 01:50:25.120035 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:25.119979 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-10047-fced50" podUID="d52f8321-1164-452e-a65a-0995c084a100" containerName="test-curl-10047-fced50" containerID="cri-o://0d45eeb4d2b310ec629fd8420940b4c81fa72b4881dffcb9013511e6c6852905" gracePeriod=2 Sep 17 01:50:25.121894 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:50:25.121861 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-10047-fced50\" is forbidden: User \"system:node:ip-10-0-138-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-83.ec2.internal' and this object" podUID="d52f8321-1164-452e-a65a-0995c084a100" pod="opendatahub/test-curl-10047-fced50" Sep 17 01:50:25.212664 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:50:25.212619 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-10047-fced50\" is forbidden: User \"system:node:ip-10-0-138-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-83.ec2.internal' and this object" podUID="d52f8321-1164-452e-a65a-0995c084a100" pod="opendatahub/test-curl-10047-fced50" Sep 17 01:50:27.304903 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:27.304875 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10047-fced50" Sep 17 01:50:27.393851 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:27.393733 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4vrg\" (UniqueName: \"kubernetes.io/projected/d52f8321-1164-452e-a65a-0995c084a100-kube-api-access-c4vrg\") pod \"d52f8321-1164-452e-a65a-0995c084a100\" (UID: \"d52f8321-1164-452e-a65a-0995c084a100\") " Sep 17 01:50:27.395912 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:27.395885 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52f8321-1164-452e-a65a-0995c084a100-kube-api-access-c4vrg" (OuterVolumeSpecName: "kube-api-access-c4vrg") pod "d52f8321-1164-452e-a65a-0995c084a100" (UID: "d52f8321-1164-452e-a65a-0995c084a100"). InnerVolumeSpecName "kube-api-access-c4vrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:50:27.495418 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:27.495371 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4vrg\" (UniqueName: \"kubernetes.io/projected/d52f8321-1164-452e-a65a-0995c084a100-kube-api-access-c4vrg\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:50:28.142542 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:28.142499 2577 generic.go:356] "Generic (PLEG): container finished" podID="d52f8321-1164-452e-a65a-0995c084a100" containerID="0d45eeb4d2b310ec629fd8420940b4c81fa72b4881dffcb9013511e6c6852905" exitCode=137 Sep 17 01:50:28.142542 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:28.142540 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-10047-fced50" Sep 17 01:50:28.142837 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:28.142601 2577 scope.go:117] "RemoveContainer" containerID="0d45eeb4d2b310ec629fd8420940b4c81fa72b4881dffcb9013511e6c6852905" Sep 17 01:50:28.152765 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:28.152741 2577 scope.go:117] "RemoveContainer" containerID="0d45eeb4d2b310ec629fd8420940b4c81fa72b4881dffcb9013511e6c6852905" Sep 17 01:50:28.153065 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:50:28.153043 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d45eeb4d2b310ec629fd8420940b4c81fa72b4881dffcb9013511e6c6852905\": container with ID starting with 0d45eeb4d2b310ec629fd8420940b4c81fa72b4881dffcb9013511e6c6852905 not found: ID does not exist" containerID="0d45eeb4d2b310ec629fd8420940b4c81fa72b4881dffcb9013511e6c6852905" Sep 17 01:50:28.153153 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:28.153074 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d45eeb4d2b310ec629fd8420940b4c81fa72b4881dffcb9013511e6c6852905"} err="failed to get container status \"0d45eeb4d2b310ec629fd8420940b4c81fa72b4881dffcb9013511e6c6852905\": rpc error: code = NotFound desc = could not find container \"0d45eeb4d2b310ec629fd8420940b4c81fa72b4881dffcb9013511e6c6852905\": container with ID starting with 0d45eeb4d2b310ec629fd8420940b4c81fa72b4881dffcb9013511e6c6852905 not found: ID does not exist" Sep 17 01:50:28.153153 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:50:28.153094 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-10047-fced50\" is forbidden: User \"system:node:ip-10-0-138-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-83.ec2.internal' and this object" podUID="d52f8321-1164-452e-a65a-0995c084a100" pod="opendatahub/test-curl-10047-fced50" Sep 17 01:50:29.213235 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:29.213192 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d52f8321-1164-452e-a65a-0995c084a100" path="/var/lib/kubelet/pods/d52f8321-1164-452e-a65a-0995c084a100/volumes" Sep 17 01:50:41.213807 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:41.213769 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:50:54.369118 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:54.369075 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn"] Sep 17 01:50:54.369612 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:54.369549 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52f8321-1164-452e-a65a-0995c084a100" containerName="test-curl-10047-fced50" Sep 17 01:50:54.369612 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:54.369562 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52f8321-1164-452e-a65a-0995c084a100" containerName="test-curl-10047-fced50" Sep 17 01:50:54.369686 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:54.369662 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52f8321-1164-452e-a65a-0995c084a100" containerName="test-curl-10047-fced50" Sep 17 01:50:54.372498 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:54.372475 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" Sep 17 01:50:54.374488 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:54.374464 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6f3c6f-serving-cert\"" Sep 17 01:50:54.400504 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:54.400380 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn"] Sep 17 01:50:54.457252 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:54.457215 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmkmm\" (UniqueName: \"kubernetes.io/projected/e218e6e6-5b90-4da9-b683-1db5349e5d12-kube-api-access-fmkmm\") pod \"maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn\" (UID: \"e218e6e6-5b90-4da9-b683-1db5349e5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" Sep 17 01:50:54.457403 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:54.457305 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e218e6e6-5b90-4da9-b683-1db5349e5d12-openshift-service-ca\") pod \"maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn\" (UID: \"e218e6e6-5b90-4da9-b683-1db5349e5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" Sep 17 01:50:54.457403 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:54.457347 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e218e6e6-5b90-4da9-b683-1db5349e5d12-metrics-certs\") pod \"maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn\" (UID: \"e218e6e6-5b90-4da9-b683-1db5349e5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" Sep 17 01:50:54.457403 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:54.457398 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e218e6e6-5b90-4da9-b683-1db5349e5d12-maas-api-tls\") pod \"maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn\" (UID: \"e218e6e6-5b90-4da9-b683-1db5349e5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" Sep 17 01:50:54.558379 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:54.558347 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e218e6e6-5b90-4da9-b683-1db5349e5d12-openshift-service-ca\") pod \"maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn\" (UID: \"e218e6e6-5b90-4da9-b683-1db5349e5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" Sep 17 01:50:54.558546 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:54.558392 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e218e6e6-5b90-4da9-b683-1db5349e5d12-metrics-certs\") pod \"maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn\" (UID: \"e218e6e6-5b90-4da9-b683-1db5349e5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" Sep 17 01:50:54.558546 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:54.558432 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e218e6e6-5b90-4da9-b683-1db5349e5d12-maas-api-tls\") pod \"maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn\" (UID: \"e218e6e6-5b90-4da9-b683-1db5349e5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" Sep 17 01:50:54.558546 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:54.558465 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmkmm\" (UniqueName: \"kubernetes.io/projected/e218e6e6-5b90-4da9-b683-1db5349e5d12-kube-api-access-fmkmm\") pod \"maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn\" (UID: \"e218e6e6-5b90-4da9-b683-1db5349e5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" Sep 17 01:50:54.558672 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:50:54.558602 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-serving-cert: secret "maas-api-e2e-api-b-6f3c6f-serving-cert" not found Sep 17 01:50:54.558711 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:50:54.558687 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e218e6e6-5b90-4da9-b683-1db5349e5d12-maas-api-tls podName:e218e6e6-5b90-4da9-b683-1db5349e5d12 nodeName:}" failed. No retries permitted until 2026-09-17 01:50:55.058669626 +0000 UTC m=+1720.293816792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e218e6e6-5b90-4da9-b683-1db5349e5d12-maas-api-tls") pod "maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" (UID: "e218e6e6-5b90-4da9-b683-1db5349e5d12") : secret "maas-api-e2e-api-b-6f3c6f-serving-cert" not found Sep 17 01:50:54.559117 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:54.559096 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e218e6e6-5b90-4da9-b683-1db5349e5d12-openshift-service-ca\") pod \"maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn\" (UID: \"e218e6e6-5b90-4da9-b683-1db5349e5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" Sep 17 01:50:54.560997 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:54.560973 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e218e6e6-5b90-4da9-b683-1db5349e5d12-metrics-certs\") pod \"maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn\" (UID: \"e218e6e6-5b90-4da9-b683-1db5349e5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" Sep 17 01:50:54.566727 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:54.566703 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmkmm\" (UniqueName: \"kubernetes.io/projected/e218e6e6-5b90-4da9-b683-1db5349e5d12-kube-api-access-fmkmm\") pod \"maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn\" (UID: \"e218e6e6-5b90-4da9-b683-1db5349e5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" Sep 17 01:50:55.064442 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:55.064403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e218e6e6-5b90-4da9-b683-1db5349e5d12-maas-api-tls\") pod \"maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn\" (UID: \"e218e6e6-5b90-4da9-b683-1db5349e5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" Sep 17 01:50:55.067341 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:55.067308 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e218e6e6-5b90-4da9-b683-1db5349e5d12-maas-api-tls\") pod \"maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn\" (UID: \"e218e6e6-5b90-4da9-b683-1db5349e5d12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" Sep 17 01:50:55.284253 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:55.284216 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" Sep 17 01:50:55.558528 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:55.558496 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn"] Sep 17 01:50:55.559473 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:50:55.559444 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode218e6e6_5b90_4da9_b683_1db5349e5d12.slice/crio-3d8cf59ed08592868ea7ebfce1daeb3586a7bcb33a7d79dcf6f7b1f88e08873c WatchSource:0}: Error finding container 3d8cf59ed08592868ea7ebfce1daeb3586a7bcb33a7d79dcf6f7b1f88e08873c: Status 404 returned error can't find the container with id 3d8cf59ed08592868ea7ebfce1daeb3586a7bcb33a7d79dcf6f7b1f88e08873c Sep 17 01:50:55.560873 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:55.560854 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 01:50:56.324556 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:56.324459 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" event={"ID":"e218e6e6-5b90-4da9-b683-1db5349e5d12","Type":"ContainerStarted","Data":"020701fae3d332aba1817323e62a3c4cd2617c6fd3226ffbe7a7547f95ea71af"} Sep 17 01:50:56.324556 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:56.324508 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" event={"ID":"e218e6e6-5b90-4da9-b683-1db5349e5d12","Type":"ContainerStarted","Data":"3d8cf59ed08592868ea7ebfce1daeb3586a7bcb33a7d79dcf6f7b1f88e08873c"} Sep 17 01:50:56.324758 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:56.324557 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" Sep 17 01:50:56.341429 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:50:56.341379 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" podStartSLOduration=2.028868988 podStartE2EDuration="2.341363026s" totalImagesPullingTime="312.494038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:50:56.339785847 +0000 UTC m=+1721.574933037" watchObservedRunningTime="2026-09-17 01:50:56.341363026 +0000 UTC m=+1721.576510340" Sep 17 01:51:02.337409 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:51:02.337377 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" Sep 17 01:51:15.469391 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:51:15.469347 2577 scope.go:117] "RemoveContainer" containerID="c1f253dd51f17e5a8d2e6756d749f70d0095a6a3dac3efcca7bfd64ff3b22eb1" Sep 17 01:51:23.205871 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:51:23.205825 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:51:56.205878 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:51:56.205842 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:52:01.530139 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:01.530103 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn"] Sep 17 01:52:01.530694 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:01.530371 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" podUID="e218e6e6-5b90-4da9-b683-1db5349e5d12" containerName="maas-api" containerID="cri-o://020701fae3d332aba1817323e62a3c4cd2617c6fd3226ffbe7a7547f95ea71af" gracePeriod=30 Sep 17 01:52:01.758684 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:01.758649 2577 generic.go:356] "Generic (PLEG): container finished" podID="e218e6e6-5b90-4da9-b683-1db5349e5d12" containerID="020701fae3d332aba1817323e62a3c4cd2617c6fd3226ffbe7a7547f95ea71af" exitCode=0 Sep 17 01:52:01.758863 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:01.758726 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" event={"ID":"e218e6e6-5b90-4da9-b683-1db5349e5d12","Type":"ContainerDied","Data":"020701fae3d332aba1817323e62a3c4cd2617c6fd3226ffbe7a7547f95ea71af"} Sep 17 01:52:01.791997 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:01.791893 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" Sep 17 01:52:01.923594 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:01.923560 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e218e6e6-5b90-4da9-b683-1db5349e5d12-openshift-service-ca\") pod \"e218e6e6-5b90-4da9-b683-1db5349e5d12\" (UID: \"e218e6e6-5b90-4da9-b683-1db5349e5d12\") " Sep 17 01:52:01.923594 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:01.923599 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmkmm\" (UniqueName: \"kubernetes.io/projected/e218e6e6-5b90-4da9-b683-1db5349e5d12-kube-api-access-fmkmm\") pod \"e218e6e6-5b90-4da9-b683-1db5349e5d12\" (UID: \"e218e6e6-5b90-4da9-b683-1db5349e5d12\") " Sep 17 01:52:01.923834 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:01.923670 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e218e6e6-5b90-4da9-b683-1db5349e5d12-metrics-certs\") pod \"e218e6e6-5b90-4da9-b683-1db5349e5d12\" (UID: \"e218e6e6-5b90-4da9-b683-1db5349e5d12\") " Sep 17 01:52:01.923834 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:01.923697 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e218e6e6-5b90-4da9-b683-1db5349e5d12-maas-api-tls\") pod \"e218e6e6-5b90-4da9-b683-1db5349e5d12\" (UID: \"e218e6e6-5b90-4da9-b683-1db5349e5d12\") " Sep 17 01:52:01.923985 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:01.923926 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e218e6e6-5b90-4da9-b683-1db5349e5d12-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e218e6e6-5b90-4da9-b683-1db5349e5d12" (UID: "e218e6e6-5b90-4da9-b683-1db5349e5d12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:52:01.925889 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:01.925858 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e218e6e6-5b90-4da9-b683-1db5349e5d12-kube-api-access-fmkmm" (OuterVolumeSpecName: "kube-api-access-fmkmm") pod "e218e6e6-5b90-4da9-b683-1db5349e5d12" (UID: "e218e6e6-5b90-4da9-b683-1db5349e5d12"). InnerVolumeSpecName "kube-api-access-fmkmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:52:01.926022 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:01.925890 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e218e6e6-5b90-4da9-b683-1db5349e5d12-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e218e6e6-5b90-4da9-b683-1db5349e5d12" (UID: "e218e6e6-5b90-4da9-b683-1db5349e5d12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:52:01.926022 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:01.925910 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e218e6e6-5b90-4da9-b683-1db5349e5d12-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e218e6e6-5b90-4da9-b683-1db5349e5d12" (UID: "e218e6e6-5b90-4da9-b683-1db5349e5d12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:52:02.025427 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:02.025390 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e218e6e6-5b90-4da9-b683-1db5349e5d12-openshift-service-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:02.025427 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:02.025425 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmkmm\" (UniqueName: \"kubernetes.io/projected/e218e6e6-5b90-4da9-b683-1db5349e5d12-kube-api-access-fmkmm\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:02.025648 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:02.025438 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e218e6e6-5b90-4da9-b683-1db5349e5d12-metrics-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:02.025648 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:02.025451 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e218e6e6-5b90-4da9-b683-1db5349e5d12-maas-api-tls\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:02.766345 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:02.766313 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" Sep 17 01:52:02.766815 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:02.766351 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn" event={"ID":"e218e6e6-5b90-4da9-b683-1db5349e5d12","Type":"ContainerDied","Data":"3d8cf59ed08592868ea7ebfce1daeb3586a7bcb33a7d79dcf6f7b1f88e08873c"} Sep 17 01:52:02.766815 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:02.766417 2577 scope.go:117] "RemoveContainer" containerID="020701fae3d332aba1817323e62a3c4cd2617c6fd3226ffbe7a7547f95ea71af" Sep 17 01:52:02.786603 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:02.786579 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn"] Sep 17 01:52:02.790548 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:02.790513 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6f3c6f-55fb8dc6f7-8jjsn"] Sep 17 01:52:03.214434 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:03.214397 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e218e6e6-5b90-4da9-b683-1db5349e5d12" path="/var/lib/kubelet/pods/e218e6e6-5b90-4da9-b683-1db5349e5d12/volumes" Sep 17 01:52:15.333700 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:15.333660 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 01:52:15.338848 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:15.338827 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 01:52:18.846507 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:18.846455 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55"] Sep 17 01:52:18.847065 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:18.846712 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" podUID="e8385561-69c6-4247-ba38-bfc52153b6e4" containerName="maas-api" containerID="cri-o://d172014a50569c1368286af9f2cdc916fb06b5d21e118849d09fcfd2122e4d2b" gracePeriod=30 Sep 17 01:52:18.911208 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:18.911172 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6"] Sep 17 01:52:18.911435 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:18.911385 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6" podUID="e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2" containerName="payload-processing" containerID="cri-o://b1b31b4872a5d314942c723523bb27c6cb25cbf4feeaf148dc1f24578ebea5b7" gracePeriod=30 Sep 17 01:52:19.194802 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.194769 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" Sep 17 01:52:19.226753 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.226714 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6" Sep 17 01:52:19.298907 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.298858 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxqpt\" (UniqueName: \"kubernetes.io/projected/e8385561-69c6-4247-ba38-bfc52153b6e4-kube-api-access-lxqpt\") pod \"e8385561-69c6-4247-ba38-bfc52153b6e4\" (UID: \"e8385561-69c6-4247-ba38-bfc52153b6e4\") " Sep 17 01:52:19.299136 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.298963 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2-plugins-config-volume\") pod \"e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2\" (UID: \"e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2\") " Sep 17 01:52:19.299136 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.299011 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8385561-69c6-4247-ba38-bfc52153b6e4-metrics-certs\") pod \"e8385561-69c6-4247-ba38-bfc52153b6e4\" (UID: \"e8385561-69c6-4247-ba38-bfc52153b6e4\") " Sep 17 01:52:19.299136 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.299064 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8385561-69c6-4247-ba38-bfc52153b6e4-openshift-service-ca\") pod \"e8385561-69c6-4247-ba38-bfc52153b6e4\" (UID: \"e8385561-69c6-4247-ba38-bfc52153b6e4\") " Sep 17 01:52:19.299136 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.299109 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnmn7\" (UniqueName: \"kubernetes.io/projected/e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2-kube-api-access-lnmn7\") pod \"e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2\" (UID: \"e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2\") " Sep 17 01:52:19.299357 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.299156 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8385561-69c6-4247-ba38-bfc52153b6e4-maas-api-tls\") pod \"e8385561-69c6-4247-ba38-bfc52153b6e4\" (UID: \"e8385561-69c6-4247-ba38-bfc52153b6e4\") " Sep 17 01:52:19.299429 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.299406 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2" (UID: "e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:52:19.299762 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.299686 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2-plugins-config-volume\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:19.299762 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.299683 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8385561-69c6-4247-ba38-bfc52153b6e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e8385561-69c6-4247-ba38-bfc52153b6e4" (UID: "e8385561-69c6-4247-ba38-bfc52153b6e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:52:19.301676 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.301644 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8385561-69c6-4247-ba38-bfc52153b6e4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e8385561-69c6-4247-ba38-bfc52153b6e4" (UID: "e8385561-69c6-4247-ba38-bfc52153b6e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:52:19.301768 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.301725 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8385561-69c6-4247-ba38-bfc52153b6e4-kube-api-access-lxqpt" (OuterVolumeSpecName: "kube-api-access-lxqpt") pod "e8385561-69c6-4247-ba38-bfc52153b6e4" (UID: "e8385561-69c6-4247-ba38-bfc52153b6e4"). InnerVolumeSpecName "kube-api-access-lxqpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:52:19.301954 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.301908 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8385561-69c6-4247-ba38-bfc52153b6e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e8385561-69c6-4247-ba38-bfc52153b6e4" (UID: "e8385561-69c6-4247-ba38-bfc52153b6e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:52:19.302288 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.302261 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2-kube-api-access-lnmn7" (OuterVolumeSpecName: "kube-api-access-lnmn7") pod "e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2" (UID: "e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2"). InnerVolumeSpecName "kube-api-access-lnmn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:52:19.400787 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.400682 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8385561-69c6-4247-ba38-bfc52153b6e4-maas-api-tls\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:19.400787 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.400719 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxqpt\" (UniqueName: \"kubernetes.io/projected/e8385561-69c6-4247-ba38-bfc52153b6e4-kube-api-access-lxqpt\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:19.400787 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.400732 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8385561-69c6-4247-ba38-bfc52153b6e4-metrics-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:19.400787 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.400743 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8385561-69c6-4247-ba38-bfc52153b6e4-openshift-service-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:19.400787 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.400754 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnmn7\" (UniqueName: \"kubernetes.io/projected/e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2-kube-api-access-lnmn7\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:19.881671 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.881630 2577 generic.go:356] "Generic (PLEG): container finished" podID="e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2" containerID="b1b31b4872a5d314942c723523bb27c6cb25cbf4feeaf148dc1f24578ebea5b7" exitCode=0 Sep 17 01:52:19.882139 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.881680 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6" Sep 17 01:52:19.882139 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.881715 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6" event={"ID":"e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2","Type":"ContainerDied","Data":"b1b31b4872a5d314942c723523bb27c6cb25cbf4feeaf148dc1f24578ebea5b7"} Sep 17 01:52:19.882139 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.881790 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6" event={"ID":"e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2","Type":"ContainerDied","Data":"5d5010dd22b408c389307a0aba7012e2584ac0aeed42e779f519b455ad4bd1c1"} Sep 17 01:52:19.882139 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.881827 2577 scope.go:117] "RemoveContainer" containerID="b1b31b4872a5d314942c723523bb27c6cb25cbf4feeaf148dc1f24578ebea5b7" Sep 17 01:52:19.883437 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.883417 2577 generic.go:356] "Generic (PLEG): container finished" podID="e8385561-69c6-4247-ba38-bfc52153b6e4" containerID="d172014a50569c1368286af9f2cdc916fb06b5d21e118849d09fcfd2122e4d2b" exitCode=0 Sep 17 01:52:19.883534 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.883451 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" event={"ID":"e8385561-69c6-4247-ba38-bfc52153b6e4","Type":"ContainerDied","Data":"d172014a50569c1368286af9f2cdc916fb06b5d21e118849d09fcfd2122e4d2b"} Sep 17 01:52:19.883534 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.883495 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" event={"ID":"e8385561-69c6-4247-ba38-bfc52153b6e4","Type":"ContainerDied","Data":"695564df018e0ff507f66b2d6123bb6dfd5b9b305c0c3e388e9d9ff03d9fcfbb"} Sep 17 01:52:19.883534 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.883456 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55" Sep 17 01:52:19.893105 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.893082 2577 scope.go:117] "RemoveContainer" containerID="b1b31b4872a5d314942c723523bb27c6cb25cbf4feeaf148dc1f24578ebea5b7" Sep 17 01:52:19.893423 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:52:19.893391 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b31b4872a5d314942c723523bb27c6cb25cbf4feeaf148dc1f24578ebea5b7\": container with ID starting with b1b31b4872a5d314942c723523bb27c6cb25cbf4feeaf148dc1f24578ebea5b7 not found: ID does not exist" containerID="b1b31b4872a5d314942c723523bb27c6cb25cbf4feeaf148dc1f24578ebea5b7" Sep 17 01:52:19.893532 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.893434 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b31b4872a5d314942c723523bb27c6cb25cbf4feeaf148dc1f24578ebea5b7"} err="failed to get container status \"b1b31b4872a5d314942c723523bb27c6cb25cbf4feeaf148dc1f24578ebea5b7\": rpc error: code = NotFound desc = could not find container \"b1b31b4872a5d314942c723523bb27c6cb25cbf4feeaf148dc1f24578ebea5b7\": container with ID starting with b1b31b4872a5d314942c723523bb27c6cb25cbf4feeaf148dc1f24578ebea5b7 not found: ID does not exist" Sep 17 01:52:19.893532 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.893460 2577 scope.go:117] "RemoveContainer" containerID="d172014a50569c1368286af9f2cdc916fb06b5d21e118849d09fcfd2122e4d2b" Sep 17 01:52:19.903328 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.903305 2577 scope.go:117] "RemoveContainer" containerID="d172014a50569c1368286af9f2cdc916fb06b5d21e118849d09fcfd2122e4d2b" Sep 17 01:52:19.903602 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:52:19.903585 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d172014a50569c1368286af9f2cdc916fb06b5d21e118849d09fcfd2122e4d2b\": container with ID starting with d172014a50569c1368286af9f2cdc916fb06b5d21e118849d09fcfd2122e4d2b not found: ID does not exist" containerID="d172014a50569c1368286af9f2cdc916fb06b5d21e118849d09fcfd2122e4d2b" Sep 17 01:52:19.903656 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.903610 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d172014a50569c1368286af9f2cdc916fb06b5d21e118849d09fcfd2122e4d2b"} err="failed to get container status \"d172014a50569c1368286af9f2cdc916fb06b5d21e118849d09fcfd2122e4d2b\": rpc error: code = NotFound desc = could not find container \"d172014a50569c1368286af9f2cdc916fb06b5d21e118849d09fcfd2122e4d2b\": container with ID starting with d172014a50569c1368286af9f2cdc916fb06b5d21e118849d09fcfd2122e4d2b not found: ID does not exist" Sep 17 01:52:19.905381 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.905360 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6"] Sep 17 01:52:19.909746 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.909723 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-474b22-68f48b46df-whdk6"] Sep 17 01:52:19.918413 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.918387 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55"] Sep 17 01:52:19.922682 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:19.922657 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-474b22-57969b6595-tbm55"] Sep 17 01:52:21.212485 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:21.212451 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8385561-69c6-4247-ba38-bfc52153b6e4" path="/var/lib/kubelet/pods/e8385561-69c6-4247-ba38-bfc52153b6e4/volumes" Sep 17 01:52:21.212890 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:21.212824 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2" path="/var/lib/kubelet/pods/e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2/volumes" Sep 17 01:52:23.830751 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:52:23.830719 2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/NetworkManager-dispatcher.service\": RecentStats: unable to find data in memory cache]" Sep 17 01:52:23.831100 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:52:23.830874 2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/NetworkManager-dispatcher.service\": RecentStats: unable to find data in memory cache]" Sep 17 01:52:25.879847 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:25.879807 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk"] Sep 17 01:52:25.880386 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:25.880260 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" podUID="8b5b4aa7-9127-4b16-ba65-5f2bd797c338" containerName="istio-proxy" containerID="cri-o://0321df4f6a7fd0fe53936923234217f5e100f09f147c9526943f9c1338142224" gracePeriod=30 Sep 17 01:52:25.897355 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:25.897320 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 01:52:25.897472 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:25.897377 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" podUID="8b5b4aa7-9127-4b16-ba65-5f2bd797c338" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 01:52:31.132463 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.132435 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:52:31.225214 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.225140 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-podinfo\") pod \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " Sep 17 01:52:31.225214 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.225170 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istiod-ca-cert\") pod \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " Sep 17 01:52:31.225214 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.225194 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-envoy\") pod \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " Sep 17 01:52:31.225479 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.225261 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-workload-certs\") pod \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " Sep 17 01:52:31.225479 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.225287 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-token\") pod \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " Sep 17 01:52:31.225479 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.225320 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wg2nj\" (UniqueName: \"kubernetes.io/projected/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-kube-api-access-wg2nj\") pod \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " Sep 17 01:52:31.225479 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.225387 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-data\") pod \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " Sep 17 01:52:31.225479 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.225427 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-workload-socket\") pod \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " Sep 17 01:52:31.225479 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.225475 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-credential-socket\") pod \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\" (UID: \"8b5b4aa7-9127-4b16-ba65-5f2bd797c338\") " Sep 17 01:52:31.225787 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.225596 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8b5b4aa7-9127-4b16-ba65-5f2bd797c338" (UID: "8b5b4aa7-9127-4b16-ba65-5f2bd797c338"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:52:31.225856 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.225794 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8b5b4aa7-9127-4b16-ba65-5f2bd797c338" (UID: "8b5b4aa7-9127-4b16-ba65-5f2bd797c338"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:52:31.225856 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.225831 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8b5b4aa7-9127-4b16-ba65-5f2bd797c338" (UID: "8b5b4aa7-9127-4b16-ba65-5f2bd797c338"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:52:31.226002 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.225977 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-workload-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:31.226060 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.226035 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-credential-socket\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:31.226060 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.226051 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istiod-ca-cert\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:31.226169 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.226070 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-data" (OuterVolumeSpecName: "istio-data") pod "8b5b4aa7-9127-4b16-ba65-5f2bd797c338" (UID: "8b5b4aa7-9127-4b16-ba65-5f2bd797c338"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:52:31.226258 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.226236 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8b5b4aa7-9127-4b16-ba65-5f2bd797c338" (UID: "8b5b4aa7-9127-4b16-ba65-5f2bd797c338"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:52:31.227540 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.227513 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8b5b4aa7-9127-4b16-ba65-5f2bd797c338" (UID: "8b5b4aa7-9127-4b16-ba65-5f2bd797c338"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:52:31.227685 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.227658 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8b5b4aa7-9127-4b16-ba65-5f2bd797c338" (UID: "8b5b4aa7-9127-4b16-ba65-5f2bd797c338"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 01:52:31.227972 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.227919 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-token" (OuterVolumeSpecName: "istio-token") pod "8b5b4aa7-9127-4b16-ba65-5f2bd797c338" (UID: "8b5b4aa7-9127-4b16-ba65-5f2bd797c338"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:52:31.228165 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.228137 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-kube-api-access-wg2nj" (OuterVolumeSpecName: "kube-api-access-wg2nj") pod "8b5b4aa7-9127-4b16-ba65-5f2bd797c338" (UID: "8b5b4aa7-9127-4b16-ba65-5f2bd797c338"). InnerVolumeSpecName "kube-api-access-wg2nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:52:31.326819 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.326781 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-data\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:31.326819 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.326813 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-workload-socket\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:31.326819 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.326823 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-podinfo\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:31.327075 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.326832 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-envoy\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:31.327075 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.326840 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-istio-token\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:31.327075 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.326849 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg2nj\" (UniqueName: \"kubernetes.io/projected/8b5b4aa7-9127-4b16-ba65-5f2bd797c338-kube-api-access-wg2nj\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:31.966405 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.966369 2577 generic.go:356] "Generic (PLEG): container finished" podID="8b5b4aa7-9127-4b16-ba65-5f2bd797c338" containerID="0321df4f6a7fd0fe53936923234217f5e100f09f147c9526943f9c1338142224" exitCode=0 Sep 17 01:52:31.966590 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.966417 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" Sep 17 01:52:31.966590 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.966456 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" event={"ID":"8b5b4aa7-9127-4b16-ba65-5f2bd797c338","Type":"ContainerDied","Data":"0321df4f6a7fd0fe53936923234217f5e100f09f147c9526943f9c1338142224"} Sep 17 01:52:31.966590 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.966513 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk" event={"ID":"8b5b4aa7-9127-4b16-ba65-5f2bd797c338","Type":"ContainerDied","Data":"42769eea290fc6daaf43c0258de94d57f16bd451e55364434dc4f7ec422d7c8c"} Sep 17 01:52:31.966590 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.966535 2577 scope.go:117] "RemoveContainer" containerID="0321df4f6a7fd0fe53936923234217f5e100f09f147c9526943f9c1338142224" Sep 17 01:52:31.976950 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.976914 2577 scope.go:117] "RemoveContainer" containerID="61b2576993de2535762d56784f07046d65f95d40df9f5f100d9a1c1839c10b94" Sep 17 01:52:31.985609 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.985582 2577 scope.go:117] "RemoveContainer" containerID="0321df4f6a7fd0fe53936923234217f5e100f09f147c9526943f9c1338142224" Sep 17 01:52:31.985970 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:52:31.985917 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0321df4f6a7fd0fe53936923234217f5e100f09f147c9526943f9c1338142224\": container with ID starting with 0321df4f6a7fd0fe53936923234217f5e100f09f147c9526943f9c1338142224 not found: ID does not exist" containerID="0321df4f6a7fd0fe53936923234217f5e100f09f147c9526943f9c1338142224" Sep 17 01:52:31.986065 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.985971 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0321df4f6a7fd0fe53936923234217f5e100f09f147c9526943f9c1338142224"} err="failed to get container status \"0321df4f6a7fd0fe53936923234217f5e100f09f147c9526943f9c1338142224\": rpc error: code = NotFound desc = could not find container \"0321df4f6a7fd0fe53936923234217f5e100f09f147c9526943f9c1338142224\": container with ID starting with 0321df4f6a7fd0fe53936923234217f5e100f09f147c9526943f9c1338142224 not found: ID does not exist" Sep 17 01:52:31.986065 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.985994 2577 scope.go:117] "RemoveContainer" containerID="61b2576993de2535762d56784f07046d65f95d40df9f5f100d9a1c1839c10b94" Sep 17 01:52:31.986430 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:52:31.986365 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b2576993de2535762d56784f07046d65f95d40df9f5f100d9a1c1839c10b94\": container with ID starting with 61b2576993de2535762d56784f07046d65f95d40df9f5f100d9a1c1839c10b94 not found: ID does not exist" containerID="61b2576993de2535762d56784f07046d65f95d40df9f5f100d9a1c1839c10b94" Sep 17 01:52:31.986430 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.986409 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b2576993de2535762d56784f07046d65f95d40df9f5f100d9a1c1839c10b94"} err="failed to get container status \"61b2576993de2535762d56784f07046d65f95d40df9f5f100d9a1c1839c10b94\": rpc error: code = NotFound desc = could not find container \"61b2576993de2535762d56784f07046d65f95d40df9f5f100d9a1c1839c10b94\": container with ID starting with 61b2576993de2535762d56784f07046d65f95d40df9f5f100d9a1c1839c10b94 not found: ID does not exist" Sep 17 01:52:31.989321 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.989301 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk"] Sep 17 01:52:31.994203 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:31.994181 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-474b22-openshift-default-c79bf6646-c8mqk"] Sep 17 01:52:33.221115 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:33.221074 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b5b4aa7-9127-4b16-ba65-5f2bd797c338" path="/var/lib/kubelet/pods/8b5b4aa7-9127-4b16-ba65-5f2bd797c338/volumes" Sep 17 01:52:36.678617 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:36.678580 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr"] Sep 17 01:52:36.679156 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:36.678795 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" podUID="9ef0979e-f4b1-4881-8bf6-eb87e4291484" containerName="maas-api" containerID="cri-o://166d017c3fe9261926a142e28ffff84ab7c089e0aab299d0cb8a205d8b52d6f1" gracePeriod=30 Sep 17 01:52:36.933044 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:36.932984 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" Sep 17 01:52:36.998874 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:36.998843 2577 generic.go:356] "Generic (PLEG): container finished" podID="9ef0979e-f4b1-4881-8bf6-eb87e4291484" containerID="166d017c3fe9261926a142e28ffff84ab7c089e0aab299d0cb8a205d8b52d6f1" exitCode=0 Sep 17 01:52:36.999087 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:36.998897 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" Sep 17 01:52:36.999087 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:36.998923 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" event={"ID":"9ef0979e-f4b1-4881-8bf6-eb87e4291484","Type":"ContainerDied","Data":"166d017c3fe9261926a142e28ffff84ab7c089e0aab299d0cb8a205d8b52d6f1"} Sep 17 01:52:36.999087 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:36.998994 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr" event={"ID":"9ef0979e-f4b1-4881-8bf6-eb87e4291484","Type":"ContainerDied","Data":"78762e9c150f9c0e9ff92aff14fd10be9164586f9618297647719d12fe12973e"} Sep 17 01:52:36.999087 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:36.999018 2577 scope.go:117] "RemoveContainer" containerID="166d017c3fe9261926a142e28ffff84ab7c089e0aab299d0cb8a205d8b52d6f1" Sep 17 01:52:37.008155 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:37.008053 2577 scope.go:117] "RemoveContainer" containerID="166d017c3fe9261926a142e28ffff84ab7c089e0aab299d0cb8a205d8b52d6f1" Sep 17 01:52:37.008369 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:52:37.008349 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"166d017c3fe9261926a142e28ffff84ab7c089e0aab299d0cb8a205d8b52d6f1\": container with ID starting with 166d017c3fe9261926a142e28ffff84ab7c089e0aab299d0cb8a205d8b52d6f1 not found: ID does not exist" containerID="166d017c3fe9261926a142e28ffff84ab7c089e0aab299d0cb8a205d8b52d6f1" Sep 17 01:52:37.008428 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:37.008380 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"166d017c3fe9261926a142e28ffff84ab7c089e0aab299d0cb8a205d8b52d6f1"} err="failed to get container status \"166d017c3fe9261926a142e28ffff84ab7c089e0aab299d0cb8a205d8b52d6f1\": rpc error: code = NotFound desc = could not find container \"166d017c3fe9261926a142e28ffff84ab7c089e0aab299d0cb8a205d8b52d6f1\": container with ID starting with 166d017c3fe9261926a142e28ffff84ab7c089e0aab299d0cb8a205d8b52d6f1 not found: ID does not exist" Sep 17 01:52:37.081658 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:37.081629 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ef0979e-f4b1-4881-8bf6-eb87e4291484-metrics-certs\") pod \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\" (UID: \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\") " Sep 17 01:52:37.081804 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:37.081689 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef0979e-f4b1-4881-8bf6-eb87e4291484-openshift-service-ca\") pod \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\" (UID: \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\") " Sep 17 01:52:37.081804 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:37.081759 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ef0979e-f4b1-4881-8bf6-eb87e4291484-maas-api-tls\") pod \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\" (UID: \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\") " Sep 17 01:52:37.081804 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:37.081802 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwpnh\" (UniqueName: \"kubernetes.io/projected/9ef0979e-f4b1-4881-8bf6-eb87e4291484-kube-api-access-lwpnh\") pod \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\" (UID: \"9ef0979e-f4b1-4881-8bf6-eb87e4291484\") " Sep 17 01:52:37.082109 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:37.082074 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef0979e-f4b1-4881-8bf6-eb87e4291484-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9ef0979e-f4b1-4881-8bf6-eb87e4291484" (UID: "9ef0979e-f4b1-4881-8bf6-eb87e4291484"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:52:37.083765 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:37.083734 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef0979e-f4b1-4881-8bf6-eb87e4291484-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9ef0979e-f4b1-4881-8bf6-eb87e4291484" (UID: "9ef0979e-f4b1-4881-8bf6-eb87e4291484"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:52:37.083860 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:37.083836 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef0979e-f4b1-4881-8bf6-eb87e4291484-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9ef0979e-f4b1-4881-8bf6-eb87e4291484" (UID: "9ef0979e-f4b1-4881-8bf6-eb87e4291484"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:52:37.083910 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:37.083891 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef0979e-f4b1-4881-8bf6-eb87e4291484-kube-api-access-lwpnh" (OuterVolumeSpecName: "kube-api-access-lwpnh") pod "9ef0979e-f4b1-4881-8bf6-eb87e4291484" (UID: "9ef0979e-f4b1-4881-8bf6-eb87e4291484"). InnerVolumeSpecName "kube-api-access-lwpnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:52:37.183582 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:37.183496 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ef0979e-f4b1-4881-8bf6-eb87e4291484-maas-api-tls\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:37.183582 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:37.183526 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwpnh\" (UniqueName: \"kubernetes.io/projected/9ef0979e-f4b1-4881-8bf6-eb87e4291484-kube-api-access-lwpnh\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:37.183582 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:37.183535 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ef0979e-f4b1-4881-8bf6-eb87e4291484-metrics-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:37.183582 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:37.183544 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef0979e-f4b1-4881-8bf6-eb87e4291484-openshift-service-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:52:37.314526 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:37.314492 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr"] Sep 17 01:52:37.320420 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:37.320392 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-725559-9f598f97d-2shjr"] Sep 17 01:52:39.212910 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:39.212874 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ef0979e-f4b1-4881-8bf6-eb87e4291484" path="/var/lib/kubelet/pods/9ef0979e-f4b1-4881-8bf6-eb87e4291484/volumes" Sep 17 01:52:49.206087 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:52:49.206051 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:53:16.206509 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:16.206478 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:53:31.064302 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.064265 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn"] Sep 17 01:53:31.064788 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.064701 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b5b4aa7-9127-4b16-ba65-5f2bd797c338" containerName="istio-proxy" Sep 17 01:53:31.064788 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.064712 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b5b4aa7-9127-4b16-ba65-5f2bd797c338" containerName="istio-proxy" Sep 17 01:53:31.064788 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.064732 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e218e6e6-5b90-4da9-b683-1db5349e5d12" containerName="maas-api" Sep 17 01:53:31.064788 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.064738 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e218e6e6-5b90-4da9-b683-1db5349e5d12" containerName="maas-api" Sep 17 01:53:31.064788 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.064746 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ef0979e-f4b1-4881-8bf6-eb87e4291484" containerName="maas-api" Sep 17 01:53:31.064788 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.064752 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ef0979e-f4b1-4881-8bf6-eb87e4291484" containerName="maas-api" Sep 17 01:53:31.064788 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.064760 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2" containerName="payload-processing" Sep 17 01:53:31.064788 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.064767 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2" containerName="payload-processing" Sep 17 01:53:31.064788 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.064779 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8385561-69c6-4247-ba38-bfc52153b6e4" containerName="maas-api" Sep 17 01:53:31.064788 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.064784 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8385561-69c6-4247-ba38-bfc52153b6e4" containerName="maas-api" Sep 17 01:53:31.065155 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.064844 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8385561-69c6-4247-ba38-bfc52153b6e4" containerName="maas-api" Sep 17 01:53:31.065155 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.064852 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e218e6e6-5b90-4da9-b683-1db5349e5d12" containerName="maas-api" Sep 17 01:53:31.065155 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.064859 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e96f0a4e-95bc-4dbb-bf5c-7c7e801d1cd2" containerName="payload-processing" Sep 17 01:53:31.065155 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.064865 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b5b4aa7-9127-4b16-ba65-5f2bd797c338" containerName="istio-proxy" Sep 17 01:53:31.065155 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.064872 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ef0979e-f4b1-4881-8bf6-eb87e4291484" containerName="maas-api" Sep 17 01:53:31.065155 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.064880 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b5b4aa7-9127-4b16-ba65-5f2bd797c338" containerName="istio-proxy" Sep 17 01:53:31.068233 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.068215 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" Sep 17 01:53:31.070192 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.070161 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-647268-serving-cert\"" Sep 17 01:53:31.077758 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.077735 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn"] Sep 17 01:53:31.104768 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.104739 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtn9b\" (UniqueName: \"kubernetes.io/projected/ec2076b0-1172-42e9-b76a-1daa329d28ba-kube-api-access-rtn9b\") pod \"maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn\" (UID: \"ec2076b0-1172-42e9-b76a-1daa329d28ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" Sep 17 01:53:31.104946 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.104790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2076b0-1172-42e9-b76a-1daa329d28ba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn\" (UID: \"ec2076b0-1172-42e9-b76a-1daa329d28ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" Sep 17 01:53:31.104946 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.104864 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2076b0-1172-42e9-b76a-1daa329d28ba-metrics-certs\") pod \"maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn\" (UID: \"ec2076b0-1172-42e9-b76a-1daa329d28ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" Sep 17 01:53:31.104946 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.104882 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2076b0-1172-42e9-b76a-1daa329d28ba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn\" (UID: \"ec2076b0-1172-42e9-b76a-1daa329d28ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" Sep 17 01:53:31.205797 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.205756 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2076b0-1172-42e9-b76a-1daa329d28ba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn\" (UID: \"ec2076b0-1172-42e9-b76a-1daa329d28ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" Sep 17 01:53:31.206016 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.205807 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2076b0-1172-42e9-b76a-1daa329d28ba-metrics-certs\") pod \"maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn\" (UID: \"ec2076b0-1172-42e9-b76a-1daa329d28ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" Sep 17 01:53:31.206016 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.205850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2076b0-1172-42e9-b76a-1daa329d28ba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn\" (UID: \"ec2076b0-1172-42e9-b76a-1daa329d28ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" Sep 17 01:53:31.206016 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.205922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtn9b\" (UniqueName: \"kubernetes.io/projected/ec2076b0-1172-42e9-b76a-1daa329d28ba-kube-api-access-rtn9b\") pod \"maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn\" (UID: \"ec2076b0-1172-42e9-b76a-1daa329d28ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" Sep 17 01:53:31.206246 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:53:31.206128 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-serving-cert: secret "maas-api-e2e-ipp-a-647268-serving-cert" not found Sep 17 01:53:31.206306 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:53:31.206243 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec2076b0-1172-42e9-b76a-1daa329d28ba-maas-api-tls podName:ec2076b0-1172-42e9-b76a-1daa329d28ba nodeName:}" failed. No retries permitted until 2026-09-17 01:53:31.706221347 +0000 UTC m=+1876.941368534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec2076b0-1172-42e9-b76a-1daa329d28ba-maas-api-tls") pod "maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" (UID: "ec2076b0-1172-42e9-b76a-1daa329d28ba") : secret "maas-api-e2e-ipp-a-647268-serving-cert" not found Sep 17 01:53:31.206616 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.206592 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2076b0-1172-42e9-b76a-1daa329d28ba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn\" (UID: \"ec2076b0-1172-42e9-b76a-1daa329d28ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" Sep 17 01:53:31.212968 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.208794 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2076b0-1172-42e9-b76a-1daa329d28ba-metrics-certs\") pod \"maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn\" (UID: \"ec2076b0-1172-42e9-b76a-1daa329d28ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" Sep 17 01:53:31.220397 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.220346 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtn9b\" (UniqueName: \"kubernetes.io/projected/ec2076b0-1172-42e9-b76a-1daa329d28ba-kube-api-access-rtn9b\") pod \"maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn\" (UID: \"ec2076b0-1172-42e9-b76a-1daa329d28ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" Sep 17 01:53:31.711465 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.711417 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2076b0-1172-42e9-b76a-1daa329d28ba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn\" (UID: \"ec2076b0-1172-42e9-b76a-1daa329d28ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" Sep 17 01:53:31.713829 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.713800 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2076b0-1172-42e9-b76a-1daa329d28ba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn\" (UID: \"ec2076b0-1172-42e9-b76a-1daa329d28ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" Sep 17 01:53:31.980197 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:31.980104 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" Sep 17 01:53:32.255027 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:32.252164 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn"] Sep 17 01:53:32.255636 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:53:32.255607 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2076b0_1172_42e9_b76a_1daa329d28ba.slice/crio-099acd22ddf24697a3064e317412e757c8fed763b9329ba537e3818247f65f2a WatchSource:0}: Error finding container 099acd22ddf24697a3064e317412e757c8fed763b9329ba537e3818247f65f2a: Status 404 returned error can't find the container with id 099acd22ddf24697a3064e317412e757c8fed763b9329ba537e3818247f65f2a Sep 17 01:53:32.340862 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:32.340824 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" event={"ID":"ec2076b0-1172-42e9-b76a-1daa329d28ba","Type":"ContainerStarted","Data":"099acd22ddf24697a3064e317412e757c8fed763b9329ba537e3818247f65f2a"} Sep 17 01:53:33.348745 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:33.348705 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" event={"ID":"ec2076b0-1172-42e9-b76a-1daa329d28ba","Type":"ContainerStarted","Data":"5c9cb1dbbdb2b784bfe70a56f7bd35565820192a1d329b4e466e4d014a8ccb19"} Sep 17 01:53:33.349165 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:33.348818 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" Sep 17 01:53:33.364099 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:33.364050 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" podStartSLOduration=2.027768101 podStartE2EDuration="2.364035302s" totalImagesPullingTime="336.267201ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:53:33.362826624 +0000 UTC m=+1878.597973812" watchObservedRunningTime="2026-09-17 01:53:33.364035302 +0000 UTC m=+1878.599182491" Sep 17 01:53:39.360854 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:53:39.360823 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" Sep 17 01:54:00.829896 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:00.829862 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb"] Sep 17 01:54:00.830401 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:00.830382 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b5b4aa7-9127-4b16-ba65-5f2bd797c338" containerName="istio-proxy" Sep 17 01:54:00.830467 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:00.830402 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b5b4aa7-9127-4b16-ba65-5f2bd797c338" containerName="istio-proxy" Sep 17 01:54:00.834204 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:00.834177 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" Sep 17 01:54:00.836346 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:00.836325 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-29e19c-serving-cert\"" Sep 17 01:54:00.846337 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:00.846306 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb"] Sep 17 01:54:00.994588 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:00.994545 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb\" (UID: \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" Sep 17 01:54:00.994772 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:00.994597 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdmj2\" (UniqueName: \"kubernetes.io/projected/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-kube-api-access-zdmj2\") pod \"maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb\" (UID: \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" Sep 17 01:54:00.994772 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:00.994698 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-metrics-certs\") pod \"maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb\" (UID: \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" Sep 17 01:54:00.994772 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:00.994741 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb\" (UID: \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" Sep 17 01:54:01.096043 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:01.096005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb\" (UID: \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" Sep 17 01:54:01.096245 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:01.096122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb\" (UID: \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" Sep 17 01:54:01.096245 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:01.096166 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdmj2\" (UniqueName: \"kubernetes.io/projected/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-kube-api-access-zdmj2\") pod \"maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb\" (UID: \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" Sep 17 01:54:01.096245 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:01.096227 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-metrics-certs\") pod \"maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb\" (UID: \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" Sep 17 01:54:01.096661 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:01.096636 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb\" (UID: \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" Sep 17 01:54:01.098593 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:01.098569 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb\" (UID: \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" Sep 17 01:54:01.098688 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:01.098619 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-metrics-certs\") pod \"maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb\" (UID: \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" Sep 17 01:54:01.103641 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:01.103619 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdmj2\" (UniqueName: \"kubernetes.io/projected/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-kube-api-access-zdmj2\") pod \"maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb\" (UID: \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" Sep 17 01:54:01.149282 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:01.149250 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" Sep 17 01:54:01.426820 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:01.426787 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb"] Sep 17 01:54:01.427882 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:54:01.427849 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563df3a3_8ff9_4f1d_8a6d_2607b0b6e6c9.slice/crio-05fa0804ea1a9e5f81a827f9eb20ec05bbe63fea8e511b651e041d196753123a WatchSource:0}: Error finding container 05fa0804ea1a9e5f81a827f9eb20ec05bbe63fea8e511b651e041d196753123a: Status 404 returned error can't find the container with id 05fa0804ea1a9e5f81a827f9eb20ec05bbe63fea8e511b651e041d196753123a Sep 17 01:54:01.523407 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:01.523367 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" event={"ID":"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9","Type":"ContainerStarted","Data":"05fa0804ea1a9e5f81a827f9eb20ec05bbe63fea8e511b651e041d196753123a"} Sep 17 01:54:02.531170 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:02.531132 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" event={"ID":"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9","Type":"ContainerStarted","Data":"01a1a37c70ef7acacd3e50269870ef9a551d856a9518067174519cc969f4df74"} Sep 17 01:54:02.531662 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:02.531197 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" Sep 17 01:54:02.548256 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:02.548203 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" podStartSLOduration=2.177659576 podStartE2EDuration="2.548184964s" totalImagesPullingTime="370.525388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:54:02.545278101 +0000 UTC m=+1907.780425289" watchObservedRunningTime="2026-09-17 01:54:02.548184964 +0000 UTC m=+1907.783332151" Sep 17 01:54:08.544507 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:08.544474 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" Sep 17 01:54:13.206468 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:13.206428 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:54:22.206347 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:22.206311 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:54:33.676103 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:33.676071 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-wh5fn_6a26ca5a-c33d-42f7-96c2-cfa79451016c/payload-processing/0.log" Sep 17 01:54:33.782844 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:54:33.782816 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-wh5fn_6a26ca5a-c33d-42f7-96c2-cfa79451016c/payload-processing/0.log" Sep 17 01:55:10.101579 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:10.101504 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-wh5fn_6a26ca5a-c33d-42f7-96c2-cfa79451016c/payload-processing/0.log" Sep 17 01:55:10.225266 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:10.225236 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-wh5fn_6a26ca5a-c33d-42f7-96c2-cfa79451016c/payload-processing/0.log" Sep 17 01:55:16.897467 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:16.897431 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv"] Sep 17 01:55:16.904234 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:16.904198 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" Sep 17 01:55:16.907726 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:16.907697 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-cb9976-serving-cert\"" Sep 17 01:55:16.916592 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:16.916561 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv"] Sep 17 01:55:17.033882 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:17.033852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dfb40a0-707c-4618-b833-000bae90e50f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv\" (UID: \"3dfb40a0-707c-4618-b833-000bae90e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" Sep 17 01:55:17.034078 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:17.033896 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dfb40a0-707c-4618-b833-000bae90e50f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv\" (UID: \"3dfb40a0-707c-4618-b833-000bae90e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" Sep 17 01:55:17.034078 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:17.034037 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lzxc\" (UniqueName: \"kubernetes.io/projected/3dfb40a0-707c-4618-b833-000bae90e50f-kube-api-access-7lzxc\") pod \"maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv\" (UID: \"3dfb40a0-707c-4618-b833-000bae90e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" Sep 17 01:55:17.034193 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:17.034080 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dfb40a0-707c-4618-b833-000bae90e50f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv\" (UID: \"3dfb40a0-707c-4618-b833-000bae90e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" Sep 17 01:55:17.135119 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:17.135082 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dfb40a0-707c-4618-b833-000bae90e50f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv\" (UID: \"3dfb40a0-707c-4618-b833-000bae90e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" Sep 17 01:55:17.135309 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:17.135128 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dfb40a0-707c-4618-b833-000bae90e50f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv\" (UID: \"3dfb40a0-707c-4618-b833-000bae90e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" Sep 17 01:55:17.135309 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:17.135183 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lzxc\" (UniqueName: \"kubernetes.io/projected/3dfb40a0-707c-4618-b833-000bae90e50f-kube-api-access-7lzxc\") pod \"maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv\" (UID: \"3dfb40a0-707c-4618-b833-000bae90e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" Sep 17 01:55:17.135309 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:17.135207 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dfb40a0-707c-4618-b833-000bae90e50f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv\" (UID: \"3dfb40a0-707c-4618-b833-000bae90e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" Sep 17 01:55:17.135501 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:55:17.135310 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-serving-cert: secret "maas-api-e2e-ipp-cleanup-cb9976-serving-cert" not found Sep 17 01:55:17.135501 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:55:17.135402 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dfb40a0-707c-4618-b833-000bae90e50f-maas-api-tls podName:3dfb40a0-707c-4618-b833-000bae90e50f nodeName:}" failed. No retries permitted until 2026-09-17 01:55:17.635383157 +0000 UTC m=+1982.870530326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3dfb40a0-707c-4618-b833-000bae90e50f-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" (UID: "3dfb40a0-707c-4618-b833-000bae90e50f") : secret "maas-api-e2e-ipp-cleanup-cb9976-serving-cert" not found Sep 17 01:55:17.135855 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:17.135833 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dfb40a0-707c-4618-b833-000bae90e50f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv\" (UID: \"3dfb40a0-707c-4618-b833-000bae90e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" Sep 17 01:55:17.137724 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:17.137702 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dfb40a0-707c-4618-b833-000bae90e50f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv\" (UID: \"3dfb40a0-707c-4618-b833-000bae90e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" Sep 17 01:55:17.145384 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:17.145360 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lzxc\" (UniqueName: \"kubernetes.io/projected/3dfb40a0-707c-4618-b833-000bae90e50f-kube-api-access-7lzxc\") pod \"maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv\" (UID: \"3dfb40a0-707c-4618-b833-000bae90e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" Sep 17 01:55:17.639562 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:17.639519 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dfb40a0-707c-4618-b833-000bae90e50f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv\" (UID: \"3dfb40a0-707c-4618-b833-000bae90e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" Sep 17 01:55:17.642063 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:17.642034 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dfb40a0-707c-4618-b833-000bae90e50f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv\" (UID: \"3dfb40a0-707c-4618-b833-000bae90e50f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" Sep 17 01:55:17.825884 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:17.825849 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" Sep 17 01:55:18.096353 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:18.096327 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv"] Sep 17 01:55:18.098094 ip-10-0-138-83 kubenswrapper[2577]: W0917 01:55:18.098062 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dfb40a0_707c_4618_b833_000bae90e50f.slice/crio-3040027b63d79ba281988fe9e14139608aafd653267fd4be42c72d7a7092f298 WatchSource:0}: Error finding container 3040027b63d79ba281988fe9e14139608aafd653267fd4be42c72d7a7092f298: Status 404 returned error can't find the container with id 3040027b63d79ba281988fe9e14139608aafd653267fd4be42c72d7a7092f298 Sep 17 01:55:19.009969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:19.009909 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" event={"ID":"3dfb40a0-707c-4618-b833-000bae90e50f","Type":"ContainerStarted","Data":"1ab334b2ca0a987404a7b26aa3d9c5dea8682f8067b6df99f32e065bbe20b426"} Sep 17 01:55:19.010148 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:19.009991 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" Sep 17 01:55:19.010148 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:19.010019 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" event={"ID":"3dfb40a0-707c-4618-b833-000bae90e50f","Type":"ContainerStarted","Data":"3040027b63d79ba281988fe9e14139608aafd653267fd4be42c72d7a7092f298"} Sep 17 01:55:19.028995 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:19.028949 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" podStartSLOduration=2.700805968 podStartE2EDuration="3.028915149s" totalImagesPullingTime="328.109181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:55:19.026170421 +0000 UTC m=+1984.261317611" watchObservedRunningTime="2026-09-17 01:55:19.028915149 +0000 UTC m=+1984.264062337" Sep 17 01:55:25.022097 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:25.022067 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" Sep 17 01:55:27.205636 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:27.205590 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:55:43.206069 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:43.206033 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:55:45.911361 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:45.911322 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv"] Sep 17 01:55:45.911833 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:45.911574 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" podUID="3dfb40a0-707c-4618-b833-000bae90e50f" containerName="maas-api" containerID="cri-o://1ab334b2ca0a987404a7b26aa3d9c5dea8682f8067b6df99f32e065bbe20b426" gracePeriod=30 Sep 17 01:55:46.159927 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.159900 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" Sep 17 01:55:46.196748 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.196657 2577 generic.go:356] "Generic (PLEG): container finished" podID="3dfb40a0-707c-4618-b833-000bae90e50f" containerID="1ab334b2ca0a987404a7b26aa3d9c5dea8682f8067b6df99f32e065bbe20b426" exitCode=0 Sep 17 01:55:46.196748 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.196713 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" event={"ID":"3dfb40a0-707c-4618-b833-000bae90e50f","Type":"ContainerDied","Data":"1ab334b2ca0a987404a7b26aa3d9c5dea8682f8067b6df99f32e065bbe20b426"} Sep 17 01:55:46.196977 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.196774 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" event={"ID":"3dfb40a0-707c-4618-b833-000bae90e50f","Type":"ContainerDied","Data":"3040027b63d79ba281988fe9e14139608aafd653267fd4be42c72d7a7092f298"} Sep 17 01:55:46.196977 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.196714 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv" Sep 17 01:55:46.196977 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.196795 2577 scope.go:117] "RemoveContainer" containerID="1ab334b2ca0a987404a7b26aa3d9c5dea8682f8067b6df99f32e065bbe20b426" Sep 17 01:55:46.206981 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.206952 2577 scope.go:117] "RemoveContainer" containerID="1ab334b2ca0a987404a7b26aa3d9c5dea8682f8067b6df99f32e065bbe20b426" Sep 17 01:55:46.207318 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:55:46.207292 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab334b2ca0a987404a7b26aa3d9c5dea8682f8067b6df99f32e065bbe20b426\": container with ID starting with 1ab334b2ca0a987404a7b26aa3d9c5dea8682f8067b6df99f32e065bbe20b426 not found: ID does not exist" containerID="1ab334b2ca0a987404a7b26aa3d9c5dea8682f8067b6df99f32e065bbe20b426" Sep 17 01:55:46.207390 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.207327 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab334b2ca0a987404a7b26aa3d9c5dea8682f8067b6df99f32e065bbe20b426"} err="failed to get container status \"1ab334b2ca0a987404a7b26aa3d9c5dea8682f8067b6df99f32e065bbe20b426\": rpc error: code = NotFound desc = could not find container \"1ab334b2ca0a987404a7b26aa3d9c5dea8682f8067b6df99f32e065bbe20b426\": container with ID starting with 1ab334b2ca0a987404a7b26aa3d9c5dea8682f8067b6df99f32e065bbe20b426 not found: ID does not exist" Sep 17 01:55:46.217739 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.217708 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dfb40a0-707c-4618-b833-000bae90e50f-maas-api-tls\") pod \"3dfb40a0-707c-4618-b833-000bae90e50f\" (UID: \"3dfb40a0-707c-4618-b833-000bae90e50f\") " Sep 17 01:55:46.217873 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.217812 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lzxc\" (UniqueName: \"kubernetes.io/projected/3dfb40a0-707c-4618-b833-000bae90e50f-kube-api-access-7lzxc\") pod \"3dfb40a0-707c-4618-b833-000bae90e50f\" (UID: \"3dfb40a0-707c-4618-b833-000bae90e50f\") " Sep 17 01:55:46.217873 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.217860 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dfb40a0-707c-4618-b833-000bae90e50f-metrics-certs\") pod \"3dfb40a0-707c-4618-b833-000bae90e50f\" (UID: \"3dfb40a0-707c-4618-b833-000bae90e50f\") " Sep 17 01:55:46.217980 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.217962 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dfb40a0-707c-4618-b833-000bae90e50f-openshift-service-ca\") pod \"3dfb40a0-707c-4618-b833-000bae90e50f\" (UID: \"3dfb40a0-707c-4618-b833-000bae90e50f\") " Sep 17 01:55:46.218513 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.218484 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dfb40a0-707c-4618-b833-000bae90e50f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3dfb40a0-707c-4618-b833-000bae90e50f" (UID: "3dfb40a0-707c-4618-b833-000bae90e50f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:55:46.220005 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.219975 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dfb40a0-707c-4618-b833-000bae90e50f-kube-api-access-7lzxc" (OuterVolumeSpecName: "kube-api-access-7lzxc") pod "3dfb40a0-707c-4618-b833-000bae90e50f" (UID: "3dfb40a0-707c-4618-b833-000bae90e50f"). InnerVolumeSpecName "kube-api-access-7lzxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:55:46.220421 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.220399 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dfb40a0-707c-4618-b833-000bae90e50f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3dfb40a0-707c-4618-b833-000bae90e50f" (UID: "3dfb40a0-707c-4618-b833-000bae90e50f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:55:46.220497 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.220418 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dfb40a0-707c-4618-b833-000bae90e50f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3dfb40a0-707c-4618-b833-000bae90e50f" (UID: "3dfb40a0-707c-4618-b833-000bae90e50f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:55:46.319483 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.319438 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lzxc\" (UniqueName: \"kubernetes.io/projected/3dfb40a0-707c-4618-b833-000bae90e50f-kube-api-access-7lzxc\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:55:46.319483 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.319477 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dfb40a0-707c-4618-b833-000bae90e50f-metrics-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:55:46.319704 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.319510 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dfb40a0-707c-4618-b833-000bae90e50f-openshift-service-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:55:46.319704 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.319522 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dfb40a0-707c-4618-b833-000bae90e50f-maas-api-tls\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:55:46.522543 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.522505 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv"] Sep 17 01:55:46.526834 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:46.526799 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cb9976-69c4bbb896-lvrpv"] Sep 17 01:55:47.212728 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:55:47.212691 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dfb40a0-707c-4618-b833-000bae90e50f" path="/var/lib/kubelet/pods/3dfb40a0-707c-4618-b833-000bae90e50f/volumes" Sep 17 01:56:11.684967 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:11.684918 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn"] Sep 17 01:56:11.687492 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:11.685203 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" podUID="ec2076b0-1172-42e9-b76a-1daa329d28ba" containerName="maas-api" containerID="cri-o://5c9cb1dbbdb2b784bfe70a56f7bd35565820192a1d329b4e466e4d014a8ccb19" gracePeriod=30 Sep 17 01:56:11.948591 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:11.948522 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" Sep 17 01:56:11.964304 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:11.964191 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtn9b\" (UniqueName: \"kubernetes.io/projected/ec2076b0-1172-42e9-b76a-1daa329d28ba-kube-api-access-rtn9b\") pod \"ec2076b0-1172-42e9-b76a-1daa329d28ba\" (UID: \"ec2076b0-1172-42e9-b76a-1daa329d28ba\") " Sep 17 01:56:11.966817 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:11.964704 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2076b0-1172-42e9-b76a-1daa329d28ba-maas-api-tls\") pod \"ec2076b0-1172-42e9-b76a-1daa329d28ba\" (UID: \"ec2076b0-1172-42e9-b76a-1daa329d28ba\") " Sep 17 01:56:11.966817 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:11.964747 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2076b0-1172-42e9-b76a-1daa329d28ba-openshift-service-ca\") pod \"ec2076b0-1172-42e9-b76a-1daa329d28ba\" (UID: \"ec2076b0-1172-42e9-b76a-1daa329d28ba\") " Sep 17 01:56:11.966817 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:11.964809 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2076b0-1172-42e9-b76a-1daa329d28ba-metrics-certs\") pod \"ec2076b0-1172-42e9-b76a-1daa329d28ba\" (UID: \"ec2076b0-1172-42e9-b76a-1daa329d28ba\") " Sep 17 01:56:11.966817 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:11.966243 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2076b0-1172-42e9-b76a-1daa329d28ba-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec2076b0-1172-42e9-b76a-1daa329d28ba" (UID: "ec2076b0-1172-42e9-b76a-1daa329d28ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:56:11.968402 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:11.968362 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2076b0-1172-42e9-b76a-1daa329d28ba-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ec2076b0-1172-42e9-b76a-1daa329d28ba" (UID: "ec2076b0-1172-42e9-b76a-1daa329d28ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:56:11.971421 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:11.971394 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2076b0-1172-42e9-b76a-1daa329d28ba-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec2076b0-1172-42e9-b76a-1daa329d28ba" (UID: "ec2076b0-1172-42e9-b76a-1daa329d28ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:56:11.973048 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:11.973006 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2076b0-1172-42e9-b76a-1daa329d28ba-kube-api-access-rtn9b" (OuterVolumeSpecName: "kube-api-access-rtn9b") pod "ec2076b0-1172-42e9-b76a-1daa329d28ba" (UID: "ec2076b0-1172-42e9-b76a-1daa329d28ba"). InnerVolumeSpecName "kube-api-access-rtn9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:56:12.066053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:12.066015 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtn9b\" (UniqueName: \"kubernetes.io/projected/ec2076b0-1172-42e9-b76a-1daa329d28ba-kube-api-access-rtn9b\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:56:12.066053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:12.066047 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2076b0-1172-42e9-b76a-1daa329d28ba-maas-api-tls\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:56:12.066053 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:12.066059 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2076b0-1172-42e9-b76a-1daa329d28ba-openshift-service-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:56:12.066367 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:12.066068 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2076b0-1172-42e9-b76a-1daa329d28ba-metrics-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:56:12.371120 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:12.371081 2577 generic.go:356] "Generic (PLEG): container finished" podID="ec2076b0-1172-42e9-b76a-1daa329d28ba" containerID="5c9cb1dbbdb2b784bfe70a56f7bd35565820192a1d329b4e466e4d014a8ccb19" exitCode=0 Sep 17 01:56:12.371340 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:12.371129 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" Sep 17 01:56:12.371340 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:12.371168 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" event={"ID":"ec2076b0-1172-42e9-b76a-1daa329d28ba","Type":"ContainerDied","Data":"5c9cb1dbbdb2b784bfe70a56f7bd35565820192a1d329b4e466e4d014a8ccb19"} Sep 17 01:56:12.371340 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:12.371233 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn" event={"ID":"ec2076b0-1172-42e9-b76a-1daa329d28ba","Type":"ContainerDied","Data":"099acd22ddf24697a3064e317412e757c8fed763b9329ba537e3818247f65f2a"} Sep 17 01:56:12.371340 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:12.371264 2577 scope.go:117] "RemoveContainer" containerID="5c9cb1dbbdb2b784bfe70a56f7bd35565820192a1d329b4e466e4d014a8ccb19" Sep 17 01:56:12.381944 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:12.381908 2577 scope.go:117] "RemoveContainer" containerID="5c9cb1dbbdb2b784bfe70a56f7bd35565820192a1d329b4e466e4d014a8ccb19" Sep 17 01:56:12.382293 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:56:12.382273 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c9cb1dbbdb2b784bfe70a56f7bd35565820192a1d329b4e466e4d014a8ccb19\": container with ID starting with 5c9cb1dbbdb2b784bfe70a56f7bd35565820192a1d329b4e466e4d014a8ccb19 not found: ID does not exist" containerID="5c9cb1dbbdb2b784bfe70a56f7bd35565820192a1d329b4e466e4d014a8ccb19" Sep 17 01:56:12.382362 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:12.382302 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c9cb1dbbdb2b784bfe70a56f7bd35565820192a1d329b4e466e4d014a8ccb19"} err="failed to get container status \"5c9cb1dbbdb2b784bfe70a56f7bd35565820192a1d329b4e466e4d014a8ccb19\": rpc error: code = NotFound desc = could not find container \"5c9cb1dbbdb2b784bfe70a56f7bd35565820192a1d329b4e466e4d014a8ccb19\": container with ID starting with 5c9cb1dbbdb2b784bfe70a56f7bd35565820192a1d329b4e466e4d014a8ccb19 not found: ID does not exist" Sep 17 01:56:12.396319 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:12.396287 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn"] Sep 17 01:56:12.399016 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:12.398992 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-647268-545b99c7b6-nhwgn"] Sep 17 01:56:13.213218 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:13.213187 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec2076b0-1172-42e9-b76a-1daa329d28ba" path="/var/lib/kubelet/pods/ec2076b0-1172-42e9-b76a-1daa329d28ba/volumes" Sep 17 01:56:50.205636 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:50.205600 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:56:59.351802 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.351761 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb"] Sep 17 01:56:59.352439 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.352041 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" podUID="563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9" containerName="maas-api" containerID="cri-o://01a1a37c70ef7acacd3e50269870ef9a551d856a9518067174519cc969f4df74" gracePeriod=30 Sep 17 01:56:59.612038 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.611960 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" Sep 17 01:56:59.672789 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.672756 2577 generic.go:356] "Generic (PLEG): container finished" podID="563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9" containerID="01a1a37c70ef7acacd3e50269870ef9a551d856a9518067174519cc969f4df74" exitCode=0 Sep 17 01:56:59.672982 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.672812 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" Sep 17 01:56:59.672982 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.672816 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" event={"ID":"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9","Type":"ContainerDied","Data":"01a1a37c70ef7acacd3e50269870ef9a551d856a9518067174519cc969f4df74"} Sep 17 01:56:59.672982 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.672942 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb" event={"ID":"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9","Type":"ContainerDied","Data":"05fa0804ea1a9e5f81a827f9eb20ec05bbe63fea8e511b651e041d196753123a"} Sep 17 01:56:59.672982 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.672971 2577 scope.go:117] "RemoveContainer" containerID="01a1a37c70ef7acacd3e50269870ef9a551d856a9518067174519cc969f4df74" Sep 17 01:56:59.683532 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.683505 2577 scope.go:117] "RemoveContainer" containerID="01a1a37c70ef7acacd3e50269870ef9a551d856a9518067174519cc969f4df74" Sep 17 01:56:59.683903 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:56:59.683880 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a1a37c70ef7acacd3e50269870ef9a551d856a9518067174519cc969f4df74\": container with ID starting with 01a1a37c70ef7acacd3e50269870ef9a551d856a9518067174519cc969f4df74 not found: ID does not exist" containerID="01a1a37c70ef7acacd3e50269870ef9a551d856a9518067174519cc969f4df74" Sep 17 01:56:59.683996 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.683914 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a1a37c70ef7acacd3e50269870ef9a551d856a9518067174519cc969f4df74"} err="failed to get container status \"01a1a37c70ef7acacd3e50269870ef9a551d856a9518067174519cc969f4df74\": rpc error: code = NotFound desc = could not find container \"01a1a37c70ef7acacd3e50269870ef9a551d856a9518067174519cc969f4df74\": container with ID starting with 01a1a37c70ef7acacd3e50269870ef9a551d856a9518067174519cc969f4df74 not found: ID does not exist" Sep 17 01:56:59.705118 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.705079 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-metrics-certs\") pod \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\" (UID: \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\") " Sep 17 01:56:59.705295 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.705131 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-openshift-service-ca\") pod \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\" (UID: \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\") " Sep 17 01:56:59.705295 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.705163 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdmj2\" (UniqueName: \"kubernetes.io/projected/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-kube-api-access-zdmj2\") pod \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\" (UID: \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\") " Sep 17 01:56:59.705295 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.705181 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-maas-api-tls\") pod \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\" (UID: \"563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9\") " Sep 17 01:56:59.705557 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.705533 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9" (UID: "563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:56:59.707362 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.707331 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9" (UID: "563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:56:59.707495 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.707359 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9" (UID: "563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:56:59.707495 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.707395 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-kube-api-access-zdmj2" (OuterVolumeSpecName: "kube-api-access-zdmj2") pod "563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9" (UID: "563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9"). InnerVolumeSpecName "kube-api-access-zdmj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:56:59.806386 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.806340 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-metrics-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:56:59.806386 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.806378 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-openshift-service-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:56:59.806386 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.806394 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdmj2\" (UniqueName: \"kubernetes.io/projected/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-kube-api-access-zdmj2\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:56:59.806644 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.806406 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9-maas-api-tls\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:56:59.994960 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:56:59.994886 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb"] Sep 17 01:57:00.001302 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:00.001271 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-29e19c-c7dc96bf-7hthb"] Sep 17 01:57:00.205954 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:00.205899 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:57:01.220359 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:01.220322 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9" path="/var/lib/kubelet/pods/563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9/volumes" Sep 17 01:57:15.378273 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:15.378240 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 01:57:15.387758 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:15.387729 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 01:57:16.682138 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:16.682101 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq"] Sep 17 01:57:16.682591 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:16.682436 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" podUID="9d1c705f-8f65-435f-9747-1ba29daee4fe" containerName="maas-api" containerID="cri-o://08f797392dd9512e92009817e090d3da114d41960544517cd7c99ddab706fc08" gracePeriod=30 Sep 17 01:57:16.940916 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:16.940835 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" Sep 17 01:57:17.066299 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.066258 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slv5p\" (UniqueName: \"kubernetes.io/projected/9d1c705f-8f65-435f-9747-1ba29daee4fe-kube-api-access-slv5p\") pod \"9d1c705f-8f65-435f-9747-1ba29daee4fe\" (UID: \"9d1c705f-8f65-435f-9747-1ba29daee4fe\") " Sep 17 01:57:17.066543 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.066370 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d1c705f-8f65-435f-9747-1ba29daee4fe-openshift-service-ca\") pod \"9d1c705f-8f65-435f-9747-1ba29daee4fe\" (UID: \"9d1c705f-8f65-435f-9747-1ba29daee4fe\") " Sep 17 01:57:17.066543 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.066416 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d1c705f-8f65-435f-9747-1ba29daee4fe-maas-api-tls\") pod \"9d1c705f-8f65-435f-9747-1ba29daee4fe\" (UID: \"9d1c705f-8f65-435f-9747-1ba29daee4fe\") " Sep 17 01:57:17.066543 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.066481 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d1c705f-8f65-435f-9747-1ba29daee4fe-metrics-certs\") pod \"9d1c705f-8f65-435f-9747-1ba29daee4fe\" (UID: \"9d1c705f-8f65-435f-9747-1ba29daee4fe\") " Sep 17 01:57:17.066763 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.066733 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d1c705f-8f65-435f-9747-1ba29daee4fe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d1c705f-8f65-435f-9747-1ba29daee4fe" (UID: "9d1c705f-8f65-435f-9747-1ba29daee4fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:57:17.068492 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.068454 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1c705f-8f65-435f-9747-1ba29daee4fe-kube-api-access-slv5p" (OuterVolumeSpecName: "kube-api-access-slv5p") pod "9d1c705f-8f65-435f-9747-1ba29daee4fe" (UID: "9d1c705f-8f65-435f-9747-1ba29daee4fe"). InnerVolumeSpecName "kube-api-access-slv5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:57:17.068622 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.068546 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d1c705f-8f65-435f-9747-1ba29daee4fe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d1c705f-8f65-435f-9747-1ba29daee4fe" (UID: "9d1c705f-8f65-435f-9747-1ba29daee4fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:57:17.068682 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.068615 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d1c705f-8f65-435f-9747-1ba29daee4fe-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9d1c705f-8f65-435f-9747-1ba29daee4fe" (UID: "9d1c705f-8f65-435f-9747-1ba29daee4fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:57:17.167425 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.167380 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d1c705f-8f65-435f-9747-1ba29daee4fe-metrics-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:57:17.167425 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.167417 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slv5p\" (UniqueName: \"kubernetes.io/projected/9d1c705f-8f65-435f-9747-1ba29daee4fe-kube-api-access-slv5p\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:57:17.167425 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.167431 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d1c705f-8f65-435f-9747-1ba29daee4fe-openshift-service-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:57:17.167663 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.167442 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d1c705f-8f65-435f-9747-1ba29daee4fe-maas-api-tls\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:57:17.784491 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.784453 2577 generic.go:356] "Generic (PLEG): container finished" podID="9d1c705f-8f65-435f-9747-1ba29daee4fe" containerID="08f797392dd9512e92009817e090d3da114d41960544517cd7c99ddab706fc08" exitCode=0 Sep 17 01:57:17.784907 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.784499 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" Sep 17 01:57:17.784907 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.784540 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" event={"ID":"9d1c705f-8f65-435f-9747-1ba29daee4fe","Type":"ContainerDied","Data":"08f797392dd9512e92009817e090d3da114d41960544517cd7c99ddab706fc08"} Sep 17 01:57:17.784907 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.784593 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq" event={"ID":"9d1c705f-8f65-435f-9747-1ba29daee4fe","Type":"ContainerDied","Data":"5edea9a53056382f7b86f79b0cf2790307518a805ebeb097d1277d98da023495"} Sep 17 01:57:17.784907 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.784617 2577 scope.go:117] "RemoveContainer" containerID="08f797392dd9512e92009817e090d3da114d41960544517cd7c99ddab706fc08" Sep 17 01:57:17.794172 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.794144 2577 scope.go:117] "RemoveContainer" containerID="08f797392dd9512e92009817e090d3da114d41960544517cd7c99ddab706fc08" Sep 17 01:57:17.794450 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:57:17.794429 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f797392dd9512e92009817e090d3da114d41960544517cd7c99ddab706fc08\": container with ID starting with 08f797392dd9512e92009817e090d3da114d41960544517cd7c99ddab706fc08 not found: ID does not exist" containerID="08f797392dd9512e92009817e090d3da114d41960544517cd7c99ddab706fc08" Sep 17 01:57:17.794541 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.794459 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f797392dd9512e92009817e090d3da114d41960544517cd7c99ddab706fc08"} err="failed to get container status \"08f797392dd9512e92009817e090d3da114d41960544517cd7c99ddab706fc08\": rpc error: code = NotFound desc = could not find container \"08f797392dd9512e92009817e090d3da114d41960544517cd7c99ddab706fc08\": container with ID starting with 08f797392dd9512e92009817e090d3da114d41960544517cd7c99ddab706fc08 not found: ID does not exist" Sep 17 01:57:17.803027 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.802266 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq"] Sep 17 01:57:17.810445 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:17.810402 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd402-68d85975b8-fmfvq"] Sep 17 01:57:19.212981 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:19.212924 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d1c705f-8f65-435f-9747-1ba29daee4fe" path="/var/lib/kubelet/pods/9d1c705f-8f65-435f-9747-1ba29daee4fe/volumes" Sep 17 01:57:34.203420 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.203383 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w"] Sep 17 01:57:34.204007 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.203637 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" podUID="02e4bd03-22eb-4bb1-9329-d5e787a8ecc3" containerName="maas-api" containerID="cri-o://45f1abd33e6ab43bc7cda3bd4f4ad37fdc95c3d6e4a770f48c649edff966040c" gracePeriod=30 Sep 17 01:57:34.457496 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.457417 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" Sep 17 01:57:34.634876 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.634828 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-openshift-service-ca\") pod \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\" (UID: \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\") " Sep 17 01:57:34.635083 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.634912 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-metrics-certs\") pod \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\" (UID: \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\") " Sep 17 01:57:34.635083 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.635052 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fc6tx\" (UniqueName: \"kubernetes.io/projected/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-kube-api-access-fc6tx\") pod \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\" (UID: \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\") " Sep 17 01:57:34.635180 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.635106 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-maas-api-tls\") pod \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\" (UID: \"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3\") " Sep 17 01:57:34.635323 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.635296 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "02e4bd03-22eb-4bb1-9329-d5e787a8ecc3" (UID: "02e4bd03-22eb-4bb1-9329-d5e787a8ecc3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:57:34.635458 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.635435 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-openshift-service-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:57:34.637160 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.637133 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "02e4bd03-22eb-4bb1-9329-d5e787a8ecc3" (UID: "02e4bd03-22eb-4bb1-9329-d5e787a8ecc3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:57:34.637246 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.637184 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "02e4bd03-22eb-4bb1-9329-d5e787a8ecc3" (UID: "02e4bd03-22eb-4bb1-9329-d5e787a8ecc3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:57:34.637246 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.637207 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-kube-api-access-fc6tx" (OuterVolumeSpecName: "kube-api-access-fc6tx") pod "02e4bd03-22eb-4bb1-9329-d5e787a8ecc3" (UID: "02e4bd03-22eb-4bb1-9329-d5e787a8ecc3"). InnerVolumeSpecName "kube-api-access-fc6tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:57:34.736314 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.736229 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-maas-api-tls\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:57:34.736314 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.736259 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-metrics-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:57:34.736314 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.736270 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc6tx\" (UniqueName: \"kubernetes.io/projected/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3-kube-api-access-fc6tx\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:57:34.896950 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.896905 2577 generic.go:356] "Generic (PLEG): container finished" podID="02e4bd03-22eb-4bb1-9329-d5e787a8ecc3" containerID="45f1abd33e6ab43bc7cda3bd4f4ad37fdc95c3d6e4a770f48c649edff966040c" exitCode=0 Sep 17 01:57:34.897137 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.896972 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" Sep 17 01:57:34.897137 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.896992 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" event={"ID":"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3","Type":"ContainerDied","Data":"45f1abd33e6ab43bc7cda3bd4f4ad37fdc95c3d6e4a770f48c649edff966040c"} Sep 17 01:57:34.897137 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.897047 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w" event={"ID":"02e4bd03-22eb-4bb1-9329-d5e787a8ecc3","Type":"ContainerDied","Data":"e793ad340c1e0b6321fbf14ac03c3a05c49df0cb4078043e2a84683b095f595e"} Sep 17 01:57:34.897137 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.897069 2577 scope.go:117] "RemoveContainer" containerID="45f1abd33e6ab43bc7cda3bd4f4ad37fdc95c3d6e4a770f48c649edff966040c" Sep 17 01:57:34.907627 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.907607 2577 scope.go:117] "RemoveContainer" containerID="45f1abd33e6ab43bc7cda3bd4f4ad37fdc95c3d6e4a770f48c649edff966040c" Sep 17 01:57:34.907992 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:57:34.907963 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45f1abd33e6ab43bc7cda3bd4f4ad37fdc95c3d6e4a770f48c649edff966040c\": container with ID starting with 45f1abd33e6ab43bc7cda3bd4f4ad37fdc95c3d6e4a770f48c649edff966040c not found: ID does not exist" containerID="45f1abd33e6ab43bc7cda3bd4f4ad37fdc95c3d6e4a770f48c649edff966040c" Sep 17 01:57:34.908070 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.907996 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45f1abd33e6ab43bc7cda3bd4f4ad37fdc95c3d6e4a770f48c649edff966040c"} err="failed to get container status \"45f1abd33e6ab43bc7cda3bd4f4ad37fdc95c3d6e4a770f48c649edff966040c\": rpc error: code = NotFound desc = could not find container \"45f1abd33e6ab43bc7cda3bd4f4ad37fdc95c3d6e4a770f48c649edff966040c\": container with ID starting with 45f1abd33e6ab43bc7cda3bd4f4ad37fdc95c3d6e4a770f48c649edff966040c not found: ID does not exist" Sep 17 01:57:34.919132 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.919079 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w"] Sep 17 01:57:34.922821 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:34.922797 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-361849-69d5dcdffc-szf8w"] Sep 17 01:57:35.214072 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:35.214037 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02e4bd03-22eb-4bb1-9329-d5e787a8ecc3" path="/var/lib/kubelet/pods/02e4bd03-22eb-4bb1-9329-d5e787a8ecc3/volumes" Sep 17 01:57:40.681701 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:40.681662 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb"] Sep 17 01:57:40.682224 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:40.681968 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" podUID="029db114-e409-4496-b364-f1ebac568192" containerName="istio-proxy" containerID="cri-o://42b6ec91979e81a9b8f5241068643c3310cc00cc56c0c69dc1a0a87bdaff19cd" gracePeriod=30 Sep 17 01:57:45.942834 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:45.942805 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:57:45.974578 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:45.974503 2577 generic.go:356] "Generic (PLEG): container finished" podID="029db114-e409-4496-b364-f1ebac568192" containerID="42b6ec91979e81a9b8f5241068643c3310cc00cc56c0c69dc1a0a87bdaff19cd" exitCode=0 Sep 17 01:57:45.974578 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:45.974569 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" Sep 17 01:57:45.974758 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:45.974581 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" event={"ID":"029db114-e409-4496-b364-f1ebac568192","Type":"ContainerDied","Data":"42b6ec91979e81a9b8f5241068643c3310cc00cc56c0c69dc1a0a87bdaff19cd"} Sep 17 01:57:45.974758 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:45.974635 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb" event={"ID":"029db114-e409-4496-b364-f1ebac568192","Type":"ContainerDied","Data":"5aed25fdb78531497655a75b1bcfe33b7334c75f91aeb04d4ee43b907c6fae88"} Sep 17 01:57:45.974758 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:45.974658 2577 scope.go:117] "RemoveContainer" containerID="42b6ec91979e81a9b8f5241068643c3310cc00cc56c0c69dc1a0a87bdaff19cd" Sep 17 01:57:45.984224 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:45.984200 2577 scope.go:117] "RemoveContainer" containerID="46ad107a814f5ffdcbfba14dee438ee474bde2f8052f80c7dd52ae44479d92ed" Sep 17 01:57:45.995333 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:45.995311 2577 scope.go:117] "RemoveContainer" containerID="42b6ec91979e81a9b8f5241068643c3310cc00cc56c0c69dc1a0a87bdaff19cd" Sep 17 01:57:45.995648 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:57:45.995629 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b6ec91979e81a9b8f5241068643c3310cc00cc56c0c69dc1a0a87bdaff19cd\": container with ID starting with 42b6ec91979e81a9b8f5241068643c3310cc00cc56c0c69dc1a0a87bdaff19cd not found: ID does not exist" containerID="42b6ec91979e81a9b8f5241068643c3310cc00cc56c0c69dc1a0a87bdaff19cd" Sep 17 01:57:45.995723 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:45.995660 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b6ec91979e81a9b8f5241068643c3310cc00cc56c0c69dc1a0a87bdaff19cd"} err="failed to get container status \"42b6ec91979e81a9b8f5241068643c3310cc00cc56c0c69dc1a0a87bdaff19cd\": rpc error: code = NotFound desc = could not find container \"42b6ec91979e81a9b8f5241068643c3310cc00cc56c0c69dc1a0a87bdaff19cd\": container with ID starting with 42b6ec91979e81a9b8f5241068643c3310cc00cc56c0c69dc1a0a87bdaff19cd not found: ID does not exist" Sep 17 01:57:45.995723 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:45.995686 2577 scope.go:117] "RemoveContainer" containerID="46ad107a814f5ffdcbfba14dee438ee474bde2f8052f80c7dd52ae44479d92ed" Sep 17 01:57:45.996011 ip-10-0-138-83 kubenswrapper[2577]: E0917 01:57:45.995988 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ad107a814f5ffdcbfba14dee438ee474bde2f8052f80c7dd52ae44479d92ed\": container with ID starting with 46ad107a814f5ffdcbfba14dee438ee474bde2f8052f80c7dd52ae44479d92ed not found: ID does not exist" containerID="46ad107a814f5ffdcbfba14dee438ee474bde2f8052f80c7dd52ae44479d92ed" Sep 17 01:57:45.996078 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:45.996019 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ad107a814f5ffdcbfba14dee438ee474bde2f8052f80c7dd52ae44479d92ed"} err="failed to get container status \"46ad107a814f5ffdcbfba14dee438ee474bde2f8052f80c7dd52ae44479d92ed\": rpc error: code = NotFound desc = could not find container \"46ad107a814f5ffdcbfba14dee438ee474bde2f8052f80c7dd52ae44479d92ed\": container with ID starting with 46ad107a814f5ffdcbfba14dee438ee474bde2f8052f80c7dd52ae44479d92ed not found: ID does not exist" Sep 17 01:57:46.043590 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.043556 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-credential-socket\") pod \"029db114-e409-4496-b364-f1ebac568192\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " Sep 17 01:57:46.043747 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.043617 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029db114-e409-4496-b364-f1ebac568192-istio-podinfo\") pod \"029db114-e409-4496-b364-f1ebac568192\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " Sep 17 01:57:46.043747 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.043662 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-workload-socket\") pod \"029db114-e409-4496-b364-f1ebac568192\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " Sep 17 01:57:46.043747 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.043707 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029db114-e409-4496-b364-f1ebac568192-istio-token\") pod \"029db114-e409-4496-b364-f1ebac568192\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " Sep 17 01:57:46.043918 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.043755 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-istio-data\") pod \"029db114-e409-4496-b364-f1ebac568192\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " Sep 17 01:57:46.043918 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.043780 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rglbt\" (UniqueName: \"kubernetes.io/projected/029db114-e409-4496-b364-f1ebac568192-kube-api-access-rglbt\") pod \"029db114-e409-4496-b364-f1ebac568192\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " Sep 17 01:57:46.043918 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.043811 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-istio-envoy\") pod \"029db114-e409-4496-b364-f1ebac568192\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " Sep 17 01:57:46.043918 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.043842 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "029db114-e409-4496-b364-f1ebac568192" (UID: "029db114-e409-4496-b364-f1ebac568192"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:57:46.043918 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.043859 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029db114-e409-4496-b364-f1ebac568192-istiod-ca-cert\") pod \"029db114-e409-4496-b364-f1ebac568192\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " Sep 17 01:57:46.043918 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.043901 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-workload-certs\") pod \"029db114-e409-4496-b364-f1ebac568192\" (UID: \"029db114-e409-4496-b364-f1ebac568192\") " Sep 17 01:57:46.044255 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.044212 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-credential-socket\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:57:46.044255 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.044235 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029db114-e409-4496-b364-f1ebac568192-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "029db114-e409-4496-b364-f1ebac568192" (UID: "029db114-e409-4496-b364-f1ebac568192"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:57:46.044352 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.044243 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "029db114-e409-4496-b364-f1ebac568192" (UID: "029db114-e409-4496-b364-f1ebac568192"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:57:46.044400 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.044357 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "029db114-e409-4496-b364-f1ebac568192" (UID: "029db114-e409-4496-b364-f1ebac568192"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:57:46.044400 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.044382 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-istio-data" (OuterVolumeSpecName: "istio-data") pod "029db114-e409-4496-b364-f1ebac568192" (UID: "029db114-e409-4496-b364-f1ebac568192"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:57:46.045969 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.045915 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/029db114-e409-4496-b364-f1ebac568192-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "029db114-e409-4496-b364-f1ebac568192" (UID: "029db114-e409-4496-b364-f1ebac568192"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 01:57:46.046109 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.046073 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029db114-e409-4496-b364-f1ebac568192-kube-api-access-rglbt" (OuterVolumeSpecName: "kube-api-access-rglbt") pod "029db114-e409-4496-b364-f1ebac568192" (UID: "029db114-e409-4496-b364-f1ebac568192"). InnerVolumeSpecName "kube-api-access-rglbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:57:46.046401 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.046377 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "029db114-e409-4496-b364-f1ebac568192" (UID: "029db114-e409-4496-b364-f1ebac568192"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:57:46.046401 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.046394 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029db114-e409-4496-b364-f1ebac568192-istio-token" (OuterVolumeSpecName: "istio-token") pod "029db114-e409-4496-b364-f1ebac568192" (UID: "029db114-e409-4496-b364-f1ebac568192"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:57:46.145026 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.144985 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-istio-data\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:57:46.145026 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.145018 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rglbt\" (UniqueName: \"kubernetes.io/projected/029db114-e409-4496-b364-f1ebac568192-kube-api-access-rglbt\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:57:46.145026 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.145028 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-istio-envoy\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:57:46.145026 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.145038 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029db114-e409-4496-b364-f1ebac568192-istiod-ca-cert\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:57:46.145299 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.145046 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-workload-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:57:46.145299 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.145054 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029db114-e409-4496-b364-f1ebac568192-istio-podinfo\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:57:46.145299 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.145061 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029db114-e409-4496-b364-f1ebac568192-workload-socket\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:57:46.145299 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.145069 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029db114-e409-4496-b364-f1ebac568192-istio-token\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 01:57:46.296743 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.296711 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb"] Sep 17 01:57:46.300670 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:46.300644 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-361849-openshift-default-69cfffbbc8-52ccb"] Sep 17 01:57:47.212552 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:57:47.212517 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="029db114-e409-4496-b364-f1ebac568192" path="/var/lib/kubelet/pods/029db114-e409-4496-b364-f1ebac568192/volumes" Sep 17 01:58:11.206775 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:58:11.206681 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:58:13.205804 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:58:13.205773 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:59:28.206043 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:59:28.206008 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 01:59:39.205830 ip-10-0-138-83 kubenswrapper[2577]: I0917 01:59:39.205794 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 02:00:00.129943 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.129904 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826840-xqgwg"] Sep 17 02:00:00.130422 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130342 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dfb40a0-707c-4618-b833-000bae90e50f" containerName="maas-api" Sep 17 02:00:00.130422 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130356 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dfb40a0-707c-4618-b833-000bae90e50f" containerName="maas-api" Sep 17 02:00:00.130422 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130377 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9" containerName="maas-api" Sep 17 02:00:00.130422 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130383 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9" containerName="maas-api" Sep 17 02:00:00.130422 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130389 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029db114-e409-4496-b364-f1ebac568192" containerName="istio-proxy" Sep 17 02:00:00.130422 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130395 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="029db114-e409-4496-b364-f1ebac568192" containerName="istio-proxy" Sep 17 02:00:00.130422 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130407 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d1c705f-8f65-435f-9747-1ba29daee4fe" containerName="maas-api" Sep 17 02:00:00.130422 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130413 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d1c705f-8f65-435f-9747-1ba29daee4fe" containerName="maas-api" Sep 17 02:00:00.130422 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130419 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e4bd03-22eb-4bb1-9329-d5e787a8ecc3" containerName="maas-api" Sep 17 02:00:00.130422 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130425 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e4bd03-22eb-4bb1-9329-d5e787a8ecc3" containerName="maas-api" Sep 17 02:00:00.130732 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130430 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029db114-e409-4496-b364-f1ebac568192" containerName="istio-proxy" Sep 17 02:00:00.130732 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130436 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="029db114-e409-4496-b364-f1ebac568192" containerName="istio-proxy" Sep 17 02:00:00.130732 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130445 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec2076b0-1172-42e9-b76a-1daa329d28ba" containerName="maas-api" Sep 17 02:00:00.130732 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130452 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec2076b0-1172-42e9-b76a-1daa329d28ba" containerName="maas-api" Sep 17 02:00:00.130732 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130513 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec2076b0-1172-42e9-b76a-1daa329d28ba" containerName="maas-api" Sep 17 02:00:00.130732 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130523 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="029db114-e409-4496-b364-f1ebac568192" containerName="istio-proxy" Sep 17 02:00:00.130732 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130529 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dfb40a0-707c-4618-b833-000bae90e50f" containerName="maas-api" Sep 17 02:00:00.130732 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130536 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e4bd03-22eb-4bb1-9329-d5e787a8ecc3" containerName="maas-api" Sep 17 02:00:00.130732 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130541 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="563df3a3-8ff9-4f1d-8a6d-2607b0b6e6c9" containerName="maas-api" Sep 17 02:00:00.130732 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.130549 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d1c705f-8f65-435f-9747-1ba29daee4fe" containerName="maas-api" Sep 17 02:00:00.133632 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.133614 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826840-xqgwg" Sep 17 02:00:00.136405 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.136354 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r9tdk\"" Sep 17 02:00:00.139213 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.139185 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826840-xqgwg"] Sep 17 02:00:00.146320 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.146296 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826840-xqgwg" Sep 17 02:00:00.417156 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.417128 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826840-xqgwg"] Sep 17 02:00:00.418139 ip-10-0-138-83 kubenswrapper[2577]: W0917 02:00:00.418111 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb160334_3249_4045_b22d_7abbba944ed4.slice/crio-832310c9cad3cc1e6e602f2e902fbdde323bcfe07cd401e95c3b5f6376146365 WatchSource:0}: Error finding container 832310c9cad3cc1e6e602f2e902fbdde323bcfe07cd401e95c3b5f6376146365: Status 404 returned error can't find the container with id 832310c9cad3cc1e6e602f2e902fbdde323bcfe07cd401e95c3b5f6376146365 Sep 17 02:00:00.773748 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.773654 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826840-xqgwg" event={"ID":"fb160334-3249-4045-b22d-7abbba944ed4","Type":"ContainerStarted","Data":"cb7ac25f24c12ca1a368b034d9a6b36ec7171522234d362a80ba4ab9b76c2c33"} Sep 17 02:00:00.773748 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.773713 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826840-xqgwg" event={"ID":"fb160334-3249-4045-b22d-7abbba944ed4","Type":"ContainerStarted","Data":"832310c9cad3cc1e6e602f2e902fbdde323bcfe07cd401e95c3b5f6376146365"} Sep 17 02:00:00.789740 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:00.789686 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826840-xqgwg" podStartSLOduration=0.78967182 podStartE2EDuration="789.67182ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:00:00.788067917 +0000 UTC m=+2266.023215106" watchObservedRunningTime="2026-09-17 02:00:00.78967182 +0000 UTC m=+2266.024819015" Sep 17 02:00:01.782096 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:01.782054 2577 generic.go:356] "Generic (PLEG): container finished" podID="fb160334-3249-4045-b22d-7abbba944ed4" containerID="cb7ac25f24c12ca1a368b034d9a6b36ec7171522234d362a80ba4ab9b76c2c33" exitCode=0 Sep 17 02:00:01.782471 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:01.782115 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826840-xqgwg" event={"ID":"fb160334-3249-4045-b22d-7abbba944ed4","Type":"ContainerDied","Data":"cb7ac25f24c12ca1a368b034d9a6b36ec7171522234d362a80ba4ab9b76c2c33"} Sep 17 02:00:02.923337 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:02.923307 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826840-xqgwg" Sep 17 02:00:03.795645 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:03.795610 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826840-xqgwg" event={"ID":"fb160334-3249-4045-b22d-7abbba944ed4","Type":"ContainerDied","Data":"832310c9cad3cc1e6e602f2e902fbdde323bcfe07cd401e95c3b5f6376146365"} Sep 17 02:00:03.795645 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:03.795643 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826840-xqgwg" Sep 17 02:00:03.795863 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:03.795663 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="832310c9cad3cc1e6e602f2e902fbdde323bcfe07cd401e95c3b5f6376146365" Sep 17 02:00:15.103711 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:15.103674 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6"] Sep 17 02:00:15.104189 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:15.103880 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" podUID="36a411dd-11a4-4abb-9ff1-677dfc09cac7" containerName="manager" containerID="cri-o://a1fd8160f6e45518e92d738340a41873976e3ffa64082c4041d9dc10fa7b394c" gracePeriod=10 Sep 17 02:00:15.872897 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:15.872863 2577 generic.go:356] "Generic (PLEG): container finished" podID="36a411dd-11a4-4abb-9ff1-677dfc09cac7" containerID="a1fd8160f6e45518e92d738340a41873976e3ffa64082c4041d9dc10fa7b394c" exitCode=0 Sep 17 02:00:15.873099 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:15.872949 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" event={"ID":"36a411dd-11a4-4abb-9ff1-677dfc09cac7","Type":"ContainerDied","Data":"a1fd8160f6e45518e92d738340a41873976e3ffa64082c4041d9dc10fa7b394c"} Sep 17 02:00:15.873099 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:15.873020 2577 scope.go:117] "RemoveContainer" containerID="c297fc2df366a5cc16cb060a7c457fde28758e19f602ca5a98349403c0bd9093" Sep 17 02:00:16.457893 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:16.457866 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" Sep 17 02:00:16.482564 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:16.482522 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbsgg\" (UniqueName: \"kubernetes.io/projected/36a411dd-11a4-4abb-9ff1-677dfc09cac7-kube-api-access-dbsgg\") pod \"36a411dd-11a4-4abb-9ff1-677dfc09cac7\" (UID: \"36a411dd-11a4-4abb-9ff1-677dfc09cac7\") " Sep 17 02:00:16.482753 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:16.482629 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/36a411dd-11a4-4abb-9ff1-677dfc09cac7-extensions-socket-volume\") pod \"36a411dd-11a4-4abb-9ff1-677dfc09cac7\" (UID: \"36a411dd-11a4-4abb-9ff1-677dfc09cac7\") " Sep 17 02:00:16.486043 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:16.485852 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a411dd-11a4-4abb-9ff1-677dfc09cac7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "36a411dd-11a4-4abb-9ff1-677dfc09cac7" (UID: "36a411dd-11a4-4abb-9ff1-677dfc09cac7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 02:00:16.489659 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:16.489626 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a411dd-11a4-4abb-9ff1-677dfc09cac7-kube-api-access-dbsgg" (OuterVolumeSpecName: "kube-api-access-dbsgg") pod "36a411dd-11a4-4abb-9ff1-677dfc09cac7" (UID: "36a411dd-11a4-4abb-9ff1-677dfc09cac7"). InnerVolumeSpecName "kube-api-access-dbsgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:00:16.584546 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:16.584458 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbsgg\" (UniqueName: \"kubernetes.io/projected/36a411dd-11a4-4abb-9ff1-677dfc09cac7-kube-api-access-dbsgg\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 02:00:16.584546 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:16.584492 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/36a411dd-11a4-4abb-9ff1-677dfc09cac7-extensions-socket-volume\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 02:00:16.879909 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:16.879877 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" event={"ID":"36a411dd-11a4-4abb-9ff1-677dfc09cac7","Type":"ContainerDied","Data":"58d72b79b7c70627c40103355e259179df84883a3b11ad73723df33637d3cf68"} Sep 17 02:00:16.879909 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:16.879900 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6" Sep 17 02:00:16.880159 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:16.879982 2577 scope.go:117] "RemoveContainer" containerID="a1fd8160f6e45518e92d738340a41873976e3ffa64082c4041d9dc10fa7b394c" Sep 17 02:00:16.900886 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:16.900856 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6"] Sep 17 02:00:16.904828 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:16.904797 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m6sw6"] Sep 17 02:00:17.212110 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:17.212022 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36a411dd-11a4-4abb-9ff1-677dfc09cac7" path="/var/lib/kubelet/pods/36a411dd-11a4-4abb-9ff1-677dfc09cac7/volumes" Sep 17 02:00:46.206110 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:46.206075 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 02:00:58.206160 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:00:58.206122 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 02:01:15.868842 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:15.868804 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7tlkp"] Sep 17 02:01:15.869301 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:15.869283 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36a411dd-11a4-4abb-9ff1-677dfc09cac7" containerName="manager" Sep 17 02:01:15.869350 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:15.869303 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="36a411dd-11a4-4abb-9ff1-677dfc09cac7" containerName="manager" Sep 17 02:01:15.869350 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:15.869328 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb160334-3249-4045-b22d-7abbba944ed4" containerName="cleanup" Sep 17 02:01:15.869350 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:15.869333 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb160334-3249-4045-b22d-7abbba944ed4" containerName="cleanup" Sep 17 02:01:15.869444 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:15.869400 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="36a411dd-11a4-4abb-9ff1-677dfc09cac7" containerName="manager" Sep 17 02:01:15.869444 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:15.869410 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="029db114-e409-4496-b364-f1ebac568192" containerName="istio-proxy" Sep 17 02:01:15.869444 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:15.869418 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="36a411dd-11a4-4abb-9ff1-677dfc09cac7" containerName="manager" Sep 17 02:01:15.869444 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:15.869424 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb160334-3249-4045-b22d-7abbba944ed4" containerName="cleanup" Sep 17 02:01:15.872886 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:15.872868 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7tlkp" Sep 17 02:01:15.875204 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:15.875181 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kl65s\"" Sep 17 02:01:15.882456 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:15.882428 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7tlkp"] Sep 17 02:01:15.913793 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:15.913702 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2bkb\" (UniqueName: \"kubernetes.io/projected/0cf779b9-82d2-441b-8100-c379d008c5b5-kube-api-access-h2bkb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7tlkp\" (UID: \"0cf779b9-82d2-441b-8100-c379d008c5b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7tlkp" Sep 17 02:01:15.913793 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:15.913780 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0cf779b9-82d2-441b-8100-c379d008c5b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7tlkp\" (UID: \"0cf779b9-82d2-441b-8100-c379d008c5b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7tlkp" Sep 17 02:01:16.015183 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:16.015140 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2bkb\" (UniqueName: \"kubernetes.io/projected/0cf779b9-82d2-441b-8100-c379d008c5b5-kube-api-access-h2bkb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7tlkp\" (UID: \"0cf779b9-82d2-441b-8100-c379d008c5b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7tlkp" Sep 17 02:01:16.015358 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:16.015194 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0cf779b9-82d2-441b-8100-c379d008c5b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7tlkp\" (UID: \"0cf779b9-82d2-441b-8100-c379d008c5b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7tlkp" Sep 17 02:01:16.015520 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:16.015504 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0cf779b9-82d2-441b-8100-c379d008c5b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7tlkp\" (UID: \"0cf779b9-82d2-441b-8100-c379d008c5b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7tlkp" Sep 17 02:01:16.023005 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:16.022975 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2bkb\" (UniqueName: \"kubernetes.io/projected/0cf779b9-82d2-441b-8100-c379d008c5b5-kube-api-access-h2bkb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7tlkp\" (UID: \"0cf779b9-82d2-441b-8100-c379d008c5b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7tlkp" Sep 17 02:01:16.185068 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:16.184970 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7tlkp" Sep 17 02:01:16.474378 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:16.474351 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7tlkp"] Sep 17 02:01:16.476460 ip-10-0-138-83 kubenswrapper[2577]: W0917 02:01:16.476427 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf779b9_82d2_441b_8100_c379d008c5b5.slice/crio-2776cdee74e3b6d52971db93c63e5f8ef81ae7d912e5667e678e4f45b9083376 WatchSource:0}: Error finding container 2776cdee74e3b6d52971db93c63e5f8ef81ae7d912e5667e678e4f45b9083376: Status 404 returned error can't find the container with id 2776cdee74e3b6d52971db93c63e5f8ef81ae7d912e5667e678e4f45b9083376 Sep 17 02:01:17.239684 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:17.239647 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7tlkp" event={"ID":"0cf779b9-82d2-441b-8100-c379d008c5b5","Type":"ContainerStarted","Data":"42cf44069eff0163f263a06252769ac9c3a39d3d31a7de8fc47a5e8f279792a2"} Sep 17 02:01:17.240168 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:17.239715 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7tlkp" Sep 17 02:01:17.240168 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:17.239732 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7tlkp" event={"ID":"0cf779b9-82d2-441b-8100-c379d008c5b5","Type":"ContainerStarted","Data":"2776cdee74e3b6d52971db93c63e5f8ef81ae7d912e5667e678e4f45b9083376"} Sep 17 02:01:17.261034 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:17.260976 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7tlkp" podStartSLOduration=2.260961386 podStartE2EDuration="2.260961386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:01:17.257381065 +0000 UTC m=+2342.492528256" watchObservedRunningTime="2026-09-17 02:01:17.260961386 +0000 UTC m=+2342.496108573" Sep 17 02:01:28.248237 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:28.248202 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7tlkp" Sep 17 02:01:59.205918 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:01:59.205874 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 02:02:05.209502 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:05.209468 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 02:02:15.420118 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:15.420088 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 02:02:15.431206 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:15.431184 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 02:02:55.884209 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:55.884161 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs"] Sep 17 02:02:55.884800 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:55.884779 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36a411dd-11a4-4abb-9ff1-677dfc09cac7" containerName="manager" Sep 17 02:02:55.884858 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:55.884803 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="36a411dd-11a4-4abb-9ff1-677dfc09cac7" containerName="manager" Sep 17 02:02:55.888128 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:55.888106 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" Sep 17 02:02:55.891172 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:55.891145 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 02:02:55.891314 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:55.891197 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 02:02:55.891314 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:55.891146 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 02:02:55.891436 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:55.891362 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s7lw7\"" Sep 17 02:02:55.891436 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:55.891429 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 17 02:02:55.898712 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:55.898689 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs"] Sep 17 02:02:55.905003 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:55.904976 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6"] Sep 17 02:02:55.909268 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:55.909246 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6" Sep 17 02:02:55.918181 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:55.918156 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6"] Sep 17 02:02:55.918862 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:55.918836 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc3d4c63-056e-403f-830f-28130028854e-maas-api-tls\") pod \"maas-api-59f6566bbc-4gnxs\" (UID: \"dc3d4c63-056e-403f-830f-28130028854e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" Sep 17 02:02:55.920921 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:55.920894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nrd2\" (UniqueName: \"kubernetes.io/projected/dc3d4c63-056e-403f-830f-28130028854e-kube-api-access-4nrd2\") pod \"maas-api-59f6566bbc-4gnxs\" (UID: \"dc3d4c63-056e-403f-830f-28130028854e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" Sep 17 02:02:55.921132 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:55.921096 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc3d4c63-056e-403f-830f-28130028854e-metrics-certs\") pod \"maas-api-59f6566bbc-4gnxs\" (UID: \"dc3d4c63-056e-403f-830f-28130028854e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" Sep 17 02:02:55.921253 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:55.921148 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3d4c63-056e-403f-830f-28130028854e-openshift-service-ca\") pod \"maas-api-59f6566bbc-4gnxs\" (UID: \"dc3d4c63-056e-403f-830f-28130028854e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" Sep 17 02:02:56.022013 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.021975 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc3d4c63-056e-403f-830f-28130028854e-maas-api-tls\") pod \"maas-api-59f6566bbc-4gnxs\" (UID: \"dc3d4c63-056e-403f-830f-28130028854e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" Sep 17 02:02:56.022013 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.022018 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nrd2\" (UniqueName: \"kubernetes.io/projected/dc3d4c63-056e-403f-830f-28130028854e-kube-api-access-4nrd2\") pod \"maas-api-59f6566bbc-4gnxs\" (UID: \"dc3d4c63-056e-403f-830f-28130028854e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" Sep 17 02:02:56.022271 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.022060 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58ae6928-692a-43cc-bdf5-be714a7a392d-maas-api-tls\") pod \"maas-api-86d9ddc875-cc5t6\" (UID: \"58ae6928-692a-43cc-bdf5-be714a7a392d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6" Sep 17 02:02:56.022271 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.022087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc3d4c63-056e-403f-830f-28130028854e-metrics-certs\") pod \"maas-api-59f6566bbc-4gnxs\" (UID: \"dc3d4c63-056e-403f-830f-28130028854e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" Sep 17 02:02:56.022271 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.022114 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58ae6928-692a-43cc-bdf5-be714a7a392d-metrics-certs\") pod \"maas-api-86d9ddc875-cc5t6\" (UID: \"58ae6928-692a-43cc-bdf5-be714a7a392d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6" Sep 17 02:02:56.022271 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.022139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3d4c63-056e-403f-830f-28130028854e-openshift-service-ca\") pod \"maas-api-59f6566bbc-4gnxs\" (UID: \"dc3d4c63-056e-403f-830f-28130028854e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" Sep 17 02:02:56.022271 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.022202 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmpj4\" (UniqueName: \"kubernetes.io/projected/58ae6928-692a-43cc-bdf5-be714a7a392d-kube-api-access-cmpj4\") pod \"maas-api-86d9ddc875-cc5t6\" (UID: \"58ae6928-692a-43cc-bdf5-be714a7a392d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6" Sep 17 02:02:56.022523 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.022398 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58ae6928-692a-43cc-bdf5-be714a7a392d-openshift-service-ca\") pod \"maas-api-86d9ddc875-cc5t6\" (UID: \"58ae6928-692a-43cc-bdf5-be714a7a392d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6" Sep 17 02:02:56.022843 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.022823 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3d4c63-056e-403f-830f-28130028854e-openshift-service-ca\") pod \"maas-api-59f6566bbc-4gnxs\" (UID: \"dc3d4c63-056e-403f-830f-28130028854e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" Sep 17 02:02:56.024527 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.024506 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc3d4c63-056e-403f-830f-28130028854e-maas-api-tls\") pod \"maas-api-59f6566bbc-4gnxs\" (UID: \"dc3d4c63-056e-403f-830f-28130028854e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" Sep 17 02:02:56.025037 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.025016 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc3d4c63-056e-403f-830f-28130028854e-metrics-certs\") pod \"maas-api-59f6566bbc-4gnxs\" (UID: \"dc3d4c63-056e-403f-830f-28130028854e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" Sep 17 02:02:56.030045 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.030018 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nrd2\" (UniqueName: \"kubernetes.io/projected/dc3d4c63-056e-403f-830f-28130028854e-kube-api-access-4nrd2\") pod \"maas-api-59f6566bbc-4gnxs\" (UID: \"dc3d4c63-056e-403f-830f-28130028854e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" Sep 17 02:02:56.123726 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.123681 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58ae6928-692a-43cc-bdf5-be714a7a392d-maas-api-tls\") pod \"maas-api-86d9ddc875-cc5t6\" (UID: \"58ae6928-692a-43cc-bdf5-be714a7a392d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6" Sep 17 02:02:56.123910 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.123748 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58ae6928-692a-43cc-bdf5-be714a7a392d-metrics-certs\") pod \"maas-api-86d9ddc875-cc5t6\" (UID: \"58ae6928-692a-43cc-bdf5-be714a7a392d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6" Sep 17 02:02:56.123910 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.123814 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmpj4\" (UniqueName: \"kubernetes.io/projected/58ae6928-692a-43cc-bdf5-be714a7a392d-kube-api-access-cmpj4\") pod \"maas-api-86d9ddc875-cc5t6\" (UID: \"58ae6928-692a-43cc-bdf5-be714a7a392d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6" Sep 17 02:02:56.123910 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.123879 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58ae6928-692a-43cc-bdf5-be714a7a392d-openshift-service-ca\") pod \"maas-api-86d9ddc875-cc5t6\" (UID: \"58ae6928-692a-43cc-bdf5-be714a7a392d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6" Sep 17 02:02:56.124698 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.124673 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58ae6928-692a-43cc-bdf5-be714a7a392d-openshift-service-ca\") pod \"maas-api-86d9ddc875-cc5t6\" (UID: \"58ae6928-692a-43cc-bdf5-be714a7a392d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6" Sep 17 02:02:56.126284 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.126254 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58ae6928-692a-43cc-bdf5-be714a7a392d-maas-api-tls\") pod \"maas-api-86d9ddc875-cc5t6\" (UID: \"58ae6928-692a-43cc-bdf5-be714a7a392d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6" Sep 17 02:02:56.126391 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.126313 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58ae6928-692a-43cc-bdf5-be714a7a392d-metrics-certs\") pod \"maas-api-86d9ddc875-cc5t6\" (UID: \"58ae6928-692a-43cc-bdf5-be714a7a392d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6" Sep 17 02:02:56.130509 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.130477 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmpj4\" (UniqueName: \"kubernetes.io/projected/58ae6928-692a-43cc-bdf5-be714a7a392d-kube-api-access-cmpj4\") pod \"maas-api-86d9ddc875-cc5t6\" (UID: \"58ae6928-692a-43cc-bdf5-be714a7a392d\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6" Sep 17 02:02:56.200909 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.200817 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" Sep 17 02:02:56.224429 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.224394 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6" Sep 17 02:02:56.476979 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.476947 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs"] Sep 17 02:02:56.481155 ip-10-0-138-83 kubenswrapper[2577]: W0917 02:02:56.481107 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3d4c63_056e_403f_830f_28130028854e.slice/crio-16d41dcf539563212ddc2a530f3daf20f9979c58ebb8857724761a209f0938f2 WatchSource:0}: Error finding container 16d41dcf539563212ddc2a530f3daf20f9979c58ebb8857724761a209f0938f2: Status 404 returned error can't find the container with id 16d41dcf539563212ddc2a530f3daf20f9979c58ebb8857724761a209f0938f2 Sep 17 02:02:56.482101 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.482075 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 02:02:56.503884 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.503859 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6"] Sep 17 02:02:56.505557 ip-10-0-138-83 kubenswrapper[2577]: W0917 02:02:56.505534 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ae6928_692a_43cc_bdf5_be714a7a392d.slice/crio-515d5f69578ba11b8f663ff7a1fb6379d9722a1d44c534f44929c94ed336132a WatchSource:0}: Error finding container 515d5f69578ba11b8f663ff7a1fb6379d9722a1d44c534f44929c94ed336132a: Status 404 returned error can't find the container with id 515d5f69578ba11b8f663ff7a1fb6379d9722a1d44c534f44929c94ed336132a Sep 17 02:02:56.840222 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.840193 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" event={"ID":"dc3d4c63-056e-403f-830f-28130028854e","Type":"ContainerStarted","Data":"16d41dcf539563212ddc2a530f3daf20f9979c58ebb8857724761a209f0938f2"} Sep 17 02:02:56.841280 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:56.841255 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6" event={"ID":"58ae6928-692a-43cc-bdf5-be714a7a392d","Type":"ContainerStarted","Data":"515d5f69578ba11b8f663ff7a1fb6379d9722a1d44c534f44929c94ed336132a"} Sep 17 02:02:57.850845 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:57.850814 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" event={"ID":"dc3d4c63-056e-403f-830f-28130028854e","Type":"ContainerStarted","Data":"60ef12e95eff1a26d4b473c72b47fe2578779f036dfeb094ee8b9338b87deaa5"} Sep 17 02:02:57.851355 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:57.850899 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" Sep 17 02:02:57.852568 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:57.852524 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6" event={"ID":"58ae6928-692a-43cc-bdf5-be714a7a392d","Type":"ContainerStarted","Data":"5f63d43bec6d3b94a0c31ba4c18426edb988a454850775e69b4bbed1dbdc21a7"} Sep 17 02:02:57.852813 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:57.852594 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6" Sep 17 02:02:57.867761 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:57.867702 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" podStartSLOduration=2.491115976 podStartE2EDuration="2.867684764s" totalImagesPullingTime="376.568788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:02:57.866248551 +0000 UTC m=+2443.101395736" watchObservedRunningTime="2026-09-17 02:02:57.867684764 +0000 UTC m=+2443.102831951" Sep 17 02:02:57.882891 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:02:57.882843 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6" podStartSLOduration=2.49779084 podStartE2EDuration="2.882824915s" totalImagesPullingTime="385.034075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:02:57.881919563 +0000 UTC m=+2443.117066752" watchObservedRunningTime="2026-09-17 02:02:57.882824915 +0000 UTC m=+2443.117972102" Sep 17 02:03:03.864458 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:03.864425 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-cc5t6" Sep 17 02:03:03.864986 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:03.864749 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" Sep 17 02:03:03.922628 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:03.921942 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs"] Sep 17 02:03:03.922628 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:03.922170 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" podUID="dc3d4c63-056e-403f-830f-28130028854e" containerName="maas-api" containerID="cri-o://60ef12e95eff1a26d4b473c72b47fe2578779f036dfeb094ee8b9338b87deaa5" gracePeriod=30 Sep 17 02:03:04.175988 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.175960 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" Sep 17 02:03:04.302644 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.302605 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3d4c63-056e-403f-830f-28130028854e-openshift-service-ca\") pod \"dc3d4c63-056e-403f-830f-28130028854e\" (UID: \"dc3d4c63-056e-403f-830f-28130028854e\") " Sep 17 02:03:04.302841 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.302696 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc3d4c63-056e-403f-830f-28130028854e-maas-api-tls\") pod \"dc3d4c63-056e-403f-830f-28130028854e\" (UID: \"dc3d4c63-056e-403f-830f-28130028854e\") " Sep 17 02:03:04.302918 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.302865 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nrd2\" (UniqueName: \"kubernetes.io/projected/dc3d4c63-056e-403f-830f-28130028854e-kube-api-access-4nrd2\") pod \"dc3d4c63-056e-403f-830f-28130028854e\" (UID: \"dc3d4c63-056e-403f-830f-28130028854e\") " Sep 17 02:03:04.303065 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.303041 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3d4c63-056e-403f-830f-28130028854e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dc3d4c63-056e-403f-830f-28130028854e" (UID: "dc3d4c63-056e-403f-830f-28130028854e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 02:03:04.303065 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.303059 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc3d4c63-056e-403f-830f-28130028854e-metrics-certs\") pod \"dc3d4c63-056e-403f-830f-28130028854e\" (UID: \"dc3d4c63-056e-403f-830f-28130028854e\") " Sep 17 02:03:04.303417 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.303394 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3d4c63-056e-403f-830f-28130028854e-openshift-service-ca\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 02:03:04.304856 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.304824 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc3d4c63-056e-403f-830f-28130028854e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dc3d4c63-056e-403f-830f-28130028854e" (UID: "dc3d4c63-056e-403f-830f-28130028854e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:03:04.305048 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.304958 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc3d4c63-056e-403f-830f-28130028854e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dc3d4c63-056e-403f-830f-28130028854e" (UID: "dc3d4c63-056e-403f-830f-28130028854e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 02:03:04.305048 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.304994 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3d4c63-056e-403f-830f-28130028854e-kube-api-access-4nrd2" (OuterVolumeSpecName: "kube-api-access-4nrd2") pod "dc3d4c63-056e-403f-830f-28130028854e" (UID: "dc3d4c63-056e-403f-830f-28130028854e"). InnerVolumeSpecName "kube-api-access-4nrd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 02:03:04.404668 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.404574 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc3d4c63-056e-403f-830f-28130028854e-maas-api-tls\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 02:03:04.404668 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.404609 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nrd2\" (UniqueName: \"kubernetes.io/projected/dc3d4c63-056e-403f-830f-28130028854e-kube-api-access-4nrd2\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 02:03:04.404668 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.404619 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc3d4c63-056e-403f-830f-28130028854e-metrics-certs\") on node \"ip-10-0-138-83.ec2.internal\" DevicePath \"\"" Sep 17 02:03:04.900507 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.900473 2577 generic.go:356] "Generic (PLEG): container finished" podID="dc3d4c63-056e-403f-830f-28130028854e" containerID="60ef12e95eff1a26d4b473c72b47fe2578779f036dfeb094ee8b9338b87deaa5" exitCode=0 Sep 17 02:03:04.901002 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.900531 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" Sep 17 02:03:04.901002 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.900539 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" event={"ID":"dc3d4c63-056e-403f-830f-28130028854e","Type":"ContainerDied","Data":"60ef12e95eff1a26d4b473c72b47fe2578779f036dfeb094ee8b9338b87deaa5"} Sep 17 02:03:04.901002 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.900604 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs" event={"ID":"dc3d4c63-056e-403f-830f-28130028854e","Type":"ContainerDied","Data":"16d41dcf539563212ddc2a530f3daf20f9979c58ebb8857724761a209f0938f2"} Sep 17 02:03:04.901002 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.900630 2577 scope.go:117] "RemoveContainer" containerID="60ef12e95eff1a26d4b473c72b47fe2578779f036dfeb094ee8b9338b87deaa5" Sep 17 02:03:04.910838 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.910821 2577 scope.go:117] "RemoveContainer" containerID="60ef12e95eff1a26d4b473c72b47fe2578779f036dfeb094ee8b9338b87deaa5" Sep 17 02:03:04.911138 ip-10-0-138-83 kubenswrapper[2577]: E0917 02:03:04.911115 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60ef12e95eff1a26d4b473c72b47fe2578779f036dfeb094ee8b9338b87deaa5\": container with ID starting with 60ef12e95eff1a26d4b473c72b47fe2578779f036dfeb094ee8b9338b87deaa5 not found: ID does not exist" containerID="60ef12e95eff1a26d4b473c72b47fe2578779f036dfeb094ee8b9338b87deaa5" Sep 17 02:03:04.911193 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.911146 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ef12e95eff1a26d4b473c72b47fe2578779f036dfeb094ee8b9338b87deaa5"} err="failed to get container status \"60ef12e95eff1a26d4b473c72b47fe2578779f036dfeb094ee8b9338b87deaa5\": rpc error: code = NotFound desc = could not find container \"60ef12e95eff1a26d4b473c72b47fe2578779f036dfeb094ee8b9338b87deaa5\": container with ID starting with 60ef12e95eff1a26d4b473c72b47fe2578779f036dfeb094ee8b9338b87deaa5 not found: ID does not exist" Sep 17 02:03:04.922300 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.922274 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs"] Sep 17 02:03:04.924924 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:04.924898 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-4gnxs"] Sep 17 02:03:05.220969 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:05.220874 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc3d4c63-056e-403f-830f-28130028854e" path="/var/lib/kubelet/pods/dc3d4c63-056e-403f-830f-28130028854e/volumes" Sep 17 02:03:26.294790 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:26.294754 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r7tqq/must-gather-c6wpb"] Sep 17 02:03:26.295306 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:26.295286 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc3d4c63-056e-403f-830f-28130028854e" containerName="maas-api" Sep 17 02:03:26.295374 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:26.295309 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc3d4c63-056e-403f-830f-28130028854e" containerName="maas-api" Sep 17 02:03:26.295412 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:26.295381 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc3d4c63-056e-403f-830f-28130028854e" containerName="maas-api" Sep 17 02:03:26.298698 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:26.298681 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r7tqq/must-gather-c6wpb" Sep 17 02:03:26.300861 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:26.300821 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r7tqq\"/\"default-dockercfg-t8545\"" Sep 17 02:03:26.301136 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:26.301119 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r7tqq\"/\"kube-root-ca.crt\"" Sep 17 02:03:26.301226 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:26.301148 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r7tqq\"/\"openshift-service-ca.crt\"" Sep 17 02:03:26.307941 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:26.307905 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r7tqq/must-gather-c6wpb"] Sep 17 02:03:26.316767 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:26.316743 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b2910c9d-e772-411a-a891-aee0de150c00-must-gather-output\") pod \"must-gather-c6wpb\" (UID: \"b2910c9d-e772-411a-a891-aee0de150c00\") " pod="openshift-must-gather-r7tqq/must-gather-c6wpb" Sep 17 02:03:26.316893 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:26.316809 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfdtp\" (UniqueName: \"kubernetes.io/projected/b2910c9d-e772-411a-a891-aee0de150c00-kube-api-access-hfdtp\") pod \"must-gather-c6wpb\" (UID: \"b2910c9d-e772-411a-a891-aee0de150c00\") " pod="openshift-must-gather-r7tqq/must-gather-c6wpb" Sep 17 02:03:26.417999 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:26.417955 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b2910c9d-e772-411a-a891-aee0de150c00-must-gather-output\") pod \"must-gather-c6wpb\" (UID: \"b2910c9d-e772-411a-a891-aee0de150c00\") " pod="openshift-must-gather-r7tqq/must-gather-c6wpb" Sep 17 02:03:26.418198 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:26.418050 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfdtp\" (UniqueName: \"kubernetes.io/projected/b2910c9d-e772-411a-a891-aee0de150c00-kube-api-access-hfdtp\") pod \"must-gather-c6wpb\" (UID: \"b2910c9d-e772-411a-a891-aee0de150c00\") " pod="openshift-must-gather-r7tqq/must-gather-c6wpb" Sep 17 02:03:26.418386 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:26.418358 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b2910c9d-e772-411a-a891-aee0de150c00-must-gather-output\") pod \"must-gather-c6wpb\" (UID: \"b2910c9d-e772-411a-a891-aee0de150c00\") " pod="openshift-must-gather-r7tqq/must-gather-c6wpb" Sep 17 02:03:26.427105 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:26.427079 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfdtp\" (UniqueName: \"kubernetes.io/projected/b2910c9d-e772-411a-a891-aee0de150c00-kube-api-access-hfdtp\") pod \"must-gather-c6wpb\" (UID: \"b2910c9d-e772-411a-a891-aee0de150c00\") " pod="openshift-must-gather-r7tqq/must-gather-c6wpb" Sep 17 02:03:26.608900 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:26.608861 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r7tqq/must-gather-c6wpb" Sep 17 02:03:26.885848 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:26.885815 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r7tqq/must-gather-c6wpb"] Sep 17 02:03:26.886709 ip-10-0-138-83 kubenswrapper[2577]: W0917 02:03:26.886683 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2910c9d_e772_411a_a891_aee0de150c00.slice/crio-00b0601e3b3add2e26c12e4997a65e9f5bca8a2906b91781af9dee1f8da2ff99 WatchSource:0}: Error finding container 00b0601e3b3add2e26c12e4997a65e9f5bca8a2906b91781af9dee1f8da2ff99: Status 404 returned error can't find the container with id 00b0601e3b3add2e26c12e4997a65e9f5bca8a2906b91781af9dee1f8da2ff99 Sep 17 02:03:27.041081 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:27.041040 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r7tqq/must-gather-c6wpb" event={"ID":"b2910c9d-e772-411a-a891-aee0de150c00","Type":"ContainerStarted","Data":"00b0601e3b3add2e26c12e4997a65e9f5bca8a2906b91781af9dee1f8da2ff99"} Sep 17 02:03:27.206546 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:27.206455 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-85fd65c94c-wh5fn" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 02:03:27.206748 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:27.206576 2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-76f657cdc4-p72fm" secret="" err="secret \"payload-processing-dockercfg-qfwsb\" not found" Sep 17 02:03:28.049864 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:28.049822 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r7tqq/must-gather-c6wpb" event={"ID":"b2910c9d-e772-411a-a891-aee0de150c00","Type":"ContainerStarted","Data":"52b588ba4571d2f35e7e331d4c00069f8603aad1727d95bed853c0b1d70f2e64"} Sep 17 02:03:28.050274 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:28.049880 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r7tqq/must-gather-c6wpb" event={"ID":"b2910c9d-e772-411a-a891-aee0de150c00","Type":"ContainerStarted","Data":"9811138a985f0e90332d8700b762b7dcf4824a5fe482aa8303897ae0e1761750"} Sep 17 02:03:28.065292 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:28.065231 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r7tqq/must-gather-c6wpb" podStartSLOduration=1.3846958360000001 podStartE2EDuration="2.065214617s" totalImagesPullingTime="680.518781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 02:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:03:28.065164786 +0000 UTC m=+2473.300311976" watchObservedRunningTime="2026-09-17 02:03:28.065214617 +0000 UTC m=+2473.300361805" Sep 17 02:03:29.176454 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:29.176412 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lkbfd_8a366399-229b-40ae-b916-4323338cb805/global-pull-secret-syncer/0.log" Sep 17 02:03:29.306138 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:29.306098 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-p9475_a32cfdee-7fb5-4b6f-bd3a-ce1441f806d4/konnectivity-agent/0.log" Sep 17 02:03:29.399110 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:29.399071 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-83.ec2.internal_5a1ee08f24cd676f4521086b970d2efd/haproxy/0.log" Sep 17 02:03:34.697086 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:34.697052 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-8wt2k_81042ac3-f417-42ca-b6ab-f14933e3c927/manager/0.log" Sep 17 02:03:34.720343 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:34.720315 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-bf8sz_6ab4c687-a263-4ad4-a8be-ae0eb5ecddb1/kuadrant-console-plugin/0.log" Sep 17 02:03:34.761714 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:34.761681 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-7tlkp_0cf779b9-82d2-441b-8100-c379d008c5b5/manager/0.log" Sep 17 02:03:34.849515 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:34.849475 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-7xlfz_63872aa9-d5cd-4c35-8ebd-3e7c5b18d0ec/manager/0.log" Sep 17 02:03:36.410337 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:36.410300 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60997e32-a723-4ea7-9716-66971576bb36/alertmanager/0.log" Sep 17 02:03:36.428716 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:36.428684 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60997e32-a723-4ea7-9716-66971576bb36/config-reloader/0.log" Sep 17 02:03:36.450523 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:36.450487 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60997e32-a723-4ea7-9716-66971576bb36/kube-rbac-proxy-web/0.log" Sep 17 02:03:36.469843 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:36.469815 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60997e32-a723-4ea7-9716-66971576bb36/kube-rbac-proxy/0.log" Sep 17 02:03:36.490950 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:36.490911 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60997e32-a723-4ea7-9716-66971576bb36/kube-rbac-proxy-metric/0.log" Sep 17 02:03:36.509156 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:36.509117 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60997e32-a723-4ea7-9716-66971576bb36/prom-label-proxy/0.log" Sep 17 02:03:36.530922 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:36.530882 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60997e32-a723-4ea7-9716-66971576bb36/init-config-reloader/0.log" Sep 17 02:03:36.948216 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:36.948173 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-stc9t_f809c0db-e228-416b-b96c-bdd10e842c22/node-exporter/0.log" Sep 17 02:03:36.967619 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:36.967542 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-stc9t_f809c0db-e228-416b-b96c-bdd10e842c22/kube-rbac-proxy/0.log" Sep 17 02:03:36.985508 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:36.985476 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-stc9t_f809c0db-e228-416b-b96c-bdd10e842c22/init-textfile/0.log" Sep 17 02:03:37.016136 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:37.016102 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-9jqgz_ba8140a0-85da-41de-a192-93694fa5b116/kube-rbac-proxy-main/0.log" Sep 17 02:03:37.041159 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:37.041110 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-9jqgz_ba8140a0-85da-41de-a192-93694fa5b116/kube-rbac-proxy-self/0.log" Sep 17 02:03:37.064080 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:37.064046 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-9jqgz_ba8140a0-85da-41de-a192-93694fa5b116/openshift-state-metrics/0.log" Sep 17 02:03:37.407079 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:37.407037 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-h9n89_8139b531-69ac-4848-9fd9-92e36ef2a87a/prometheus-operator-admission-webhook/0.log" Sep 17 02:03:37.437865 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:37.437830 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6445df8557-7hkg9_a2e8bdcb-05af-4b85-99cb-bf9de2cc2901/telemeter-client/0.log" Sep 17 02:03:37.461869 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:37.461834 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6445df8557-7hkg9_a2e8bdcb-05af-4b85-99cb-bf9de2cc2901/reload/0.log" Sep 17 02:03:37.486638 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:37.486604 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6445df8557-7hkg9_a2e8bdcb-05af-4b85-99cb-bf9de2cc2901/kube-rbac-proxy/0.log" Sep 17 02:03:37.512844 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:37.512806 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-559ccbc595-79zsl_bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49/thanos-query/0.log" Sep 17 02:03:37.531236 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:37.531206 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-559ccbc595-79zsl_bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49/kube-rbac-proxy-web/0.log" Sep 17 02:03:37.551023 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:37.550983 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-559ccbc595-79zsl_bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49/kube-rbac-proxy/0.log" Sep 17 02:03:37.568803 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:37.568774 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-559ccbc595-79zsl_bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49/prom-label-proxy/0.log" Sep 17 02:03:37.587141 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:37.587109 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-559ccbc595-79zsl_bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49/kube-rbac-proxy-rules/0.log" Sep 17 02:03:37.608339 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:37.608304 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-559ccbc595-79zsl_bd9ed4e1-6b8d-4c51-9b9a-d48d7df71b49/kube-rbac-proxy-metrics/0.log" Sep 17 02:03:39.537293 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.537249 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct"] Sep 17 02:03:39.549764 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.549728 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct"] Sep 17 02:03:39.549966 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.549860 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" Sep 17 02:03:39.588859 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.588810 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/1.log" Sep 17 02:03:39.592745 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.592704 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9zzb8_086884dc-5574-41f1-952f-f57a355d5294/console-operator/2.log" Sep 17 02:03:39.672005 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.671963 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dd0dd6e-0418-4a57-9d7e-36831be4ddf6-sys\") pod \"perf-node-gather-daemonset-jtlct\" (UID: \"2dd0dd6e-0418-4a57-9d7e-36831be4ddf6\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" Sep 17 02:03:39.672215 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.672017 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2dd0dd6e-0418-4a57-9d7e-36831be4ddf6-proc\") pod \"perf-node-gather-daemonset-jtlct\" (UID: \"2dd0dd6e-0418-4a57-9d7e-36831be4ddf6\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" Sep 17 02:03:39.672215 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.672070 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2dd0dd6e-0418-4a57-9d7e-36831be4ddf6-podres\") pod \"perf-node-gather-daemonset-jtlct\" (UID: \"2dd0dd6e-0418-4a57-9d7e-36831be4ddf6\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" Sep 17 02:03:39.672215 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.672130 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrckr\" (UniqueName: \"kubernetes.io/projected/2dd0dd6e-0418-4a57-9d7e-36831be4ddf6-kube-api-access-qrckr\") pod \"perf-node-gather-daemonset-jtlct\" (UID: \"2dd0dd6e-0418-4a57-9d7e-36831be4ddf6\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" Sep 17 02:03:39.672215 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.672206 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2dd0dd6e-0418-4a57-9d7e-36831be4ddf6-lib-modules\") pod \"perf-node-gather-daemonset-jtlct\" (UID: \"2dd0dd6e-0418-4a57-9d7e-36831be4ddf6\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" Sep 17 02:03:39.773088 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.773043 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2dd0dd6e-0418-4a57-9d7e-36831be4ddf6-lib-modules\") pod \"perf-node-gather-daemonset-jtlct\" (UID: \"2dd0dd6e-0418-4a57-9d7e-36831be4ddf6\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" Sep 17 02:03:39.773397 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.773358 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2dd0dd6e-0418-4a57-9d7e-36831be4ddf6-lib-modules\") pod \"perf-node-gather-daemonset-jtlct\" (UID: \"2dd0dd6e-0418-4a57-9d7e-36831be4ddf6\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" Sep 17 02:03:39.773517 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.773384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dd0dd6e-0418-4a57-9d7e-36831be4ddf6-sys\") pod \"perf-node-gather-daemonset-jtlct\" (UID: \"2dd0dd6e-0418-4a57-9d7e-36831be4ddf6\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" Sep 17 02:03:39.773650 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.773632 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2dd0dd6e-0418-4a57-9d7e-36831be4ddf6-proc\") pod \"perf-node-gather-daemonset-jtlct\" (UID: \"2dd0dd6e-0418-4a57-9d7e-36831be4ddf6\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" Sep 17 02:03:39.773786 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.773771 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2dd0dd6e-0418-4a57-9d7e-36831be4ddf6-podres\") pod \"perf-node-gather-daemonset-jtlct\" (UID: \"2dd0dd6e-0418-4a57-9d7e-36831be4ddf6\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" Sep 17 02:03:39.773922 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.773895 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2dd0dd6e-0418-4a57-9d7e-36831be4ddf6-podres\") pod \"perf-node-gather-daemonset-jtlct\" (UID: \"2dd0dd6e-0418-4a57-9d7e-36831be4ddf6\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" Sep 17 02:03:39.774106 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.773525 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dd0dd6e-0418-4a57-9d7e-36831be4ddf6-sys\") pod \"perf-node-gather-daemonset-jtlct\" (UID: \"2dd0dd6e-0418-4a57-9d7e-36831be4ddf6\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" Sep 17 02:03:39.774208 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.773944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrckr\" (UniqueName: \"kubernetes.io/projected/2dd0dd6e-0418-4a57-9d7e-36831be4ddf6-kube-api-access-qrckr\") pod \"perf-node-gather-daemonset-jtlct\" (UID: \"2dd0dd6e-0418-4a57-9d7e-36831be4ddf6\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" Sep 17 02:03:39.774333 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.773713 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2dd0dd6e-0418-4a57-9d7e-36831be4ddf6-proc\") pod \"perf-node-gather-daemonset-jtlct\" (UID: \"2dd0dd6e-0418-4a57-9d7e-36831be4ddf6\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" Sep 17 02:03:39.781311 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.781280 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrckr\" (UniqueName: \"kubernetes.io/projected/2dd0dd6e-0418-4a57-9d7e-36831be4ddf6-kube-api-access-qrckr\") pod \"perf-node-gather-daemonset-jtlct\" (UID: \"2dd0dd6e-0418-4a57-9d7e-36831be4ddf6\") " pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" Sep 17 02:03:39.865239 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:39.865197 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" Sep 17 02:03:40.137461 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:40.137372 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6c95c4f55d-sg92x_dd98a571-4bfe-44d5-973b-74a3758758f7/console/0.log" Sep 17 02:03:40.162983 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:40.162954 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct"] Sep 17 02:03:40.165551 ip-10-0-138-83 kubenswrapper[2577]: W0917 02:03:40.164794 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2dd0dd6e_0418_4a57_9d7e_36831be4ddf6.slice/crio-5cb208124f3ff20d29e18908760200ca1269c29673b7ad8902da830cd13df0ff WatchSource:0}: Error finding container 5cb208124f3ff20d29e18908760200ca1269c29673b7ad8902da830cd13df0ff: Status 404 returned error can't find the container with id 5cb208124f3ff20d29e18908760200ca1269c29673b7ad8902da830cd13df0ff Sep 17 02:03:40.176446 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:40.176425 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-lwhdx_51861ecb-5643-4b47-9c98-c083d6abec0d/download-server/0.log" Sep 17 02:03:40.840828 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:40.840795 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-b87fj_535cb079-9f6c-4cab-ace5-1664b0dd0523/volume-data-source-validator/0.log" Sep 17 02:03:41.182192 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:41.182095 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" event={"ID":"2dd0dd6e-0418-4a57-9d7e-36831be4ddf6","Type":"ContainerStarted","Data":"4ae1c70d48426d99b42977d23c7458535d7ff45933b7da6a8c1c82b21490e033"} Sep 17 02:03:41.182497 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:41.182466 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" Sep 17 02:03:41.182629 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:41.182512 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" event={"ID":"2dd0dd6e-0418-4a57-9d7e-36831be4ddf6","Type":"ContainerStarted","Data":"5cb208124f3ff20d29e18908760200ca1269c29673b7ad8902da830cd13df0ff"} Sep 17 02:03:41.196971 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:41.196884 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" podStartSLOduration=2.196865207 podStartE2EDuration="2.196865207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 02:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 02:03:41.195016343 +0000 UTC m=+2486.430163532" watchObservedRunningTime="2026-09-17 02:03:41.196865207 +0000 UTC m=+2486.432012421" Sep 17 02:03:41.912763 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:41.912729 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m4hzm_942e7bc8-54bc-4e0f-939b-814164a4ed3d/dns/0.log" Sep 17 02:03:41.933563 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:41.933529 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m4hzm_942e7bc8-54bc-4e0f-939b-814164a4ed3d/kube-rbac-proxy/0.log" Sep 17 02:03:42.045214 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:42.045175 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ttq94_78186fc7-1ae0-4bf9-89ec-38c47218f909/dns-node-resolver/0.log" Sep 17 02:03:42.578617 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:42.578555 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-km877_7b6b4978-53de-4fa4-83cc-21d3de40ea24/node-ca/0.log" Sep 17 02:03:43.515596 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:43.515561 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn_ad0996a8-d893-486b-a3ee-4ee366bc558b/istio-proxy/0.log" Sep 17 02:03:43.949867 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:43.949834 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76f657cdc4-p72fm_d3faa76e-121f-491e-a616-1ac261e85583/payload-pre-processing/0.log" Sep 17 02:03:43.972182 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:43.972124 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-wh5fn_6a26ca5a-c33d-42f7-96c2-cfa79451016c/payload-processing/0.log" Sep 17 02:03:43.999311 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:43.999253 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-585fc78676-kqsfm_4d1b91db-e40c-48f8-8008-f01fd5801ec0/router/0.log" Sep 17 02:03:44.823121 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:44.823088 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6p5q7_2a717347-6484-44b8-bdb8-19ca455b2905/serve-healthcheck-canary/0.log" Sep 17 02:03:45.335113 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:45.335080 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6drdk_f7e40a74-41cf-4f58-a203-0450fc55391e/kube-rbac-proxy/0.log" Sep 17 02:03:45.356042 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:45.355997 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6drdk_f7e40a74-41cf-4f58-a203-0450fc55391e/exporter/0.log" Sep 17 02:03:45.376831 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:45.376792 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6drdk_f7e40a74-41cf-4f58-a203-0450fc55391e/extractor/0.log" Sep 17 02:03:47.201201 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:47.201144 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r7tqq/perf-node-gather-daemonset-jtlct" Sep 17 02:03:47.558451 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:47.558355 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6bbfbbc97b-krnlb_e13e08b9-5f27-4c88-9dbb-b8ec40a0ff44/manager/0.log" Sep 17 02:03:47.871379 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:47.871349 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-69vh5_5ea401cd-94c4-4dba-94a7-b08cd781569b/server/0.log" Sep 17 02:03:48.001460 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:48.001323 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9x5vs_6d0d0ca1-0f07-4b70-b26c-57821ec3e927/manager/0.log" Sep 17 02:03:48.293618 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:48.293586 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57c6978585-vxh8t_24d8f922-bd35-4b68-b549-e801ac074c1a/manager/0.log" Sep 17 02:03:48.311413 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:48.311382 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57c6978585-vxh8t_24d8f922-bd35-4b68-b549-e801ac074c1a/copy-manifests/0.log" Sep 17 02:03:49.849067 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:49.849038 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5749dbf97f-f95v9_635e80dd-2525-4e5b-9a0c-e8ff2db390fa/manager/0.log" Sep 17 02:03:49.910724 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:49.910695 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-b7z29_0bdc6fe7-6652-4190-bfc0-b1030e58e632/openshift-lws-operator/0.log" Sep 17 02:03:50.421503 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:50.421470 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn_ad0996a8-d893-486b-a3ee-4ee366bc558b/istio-proxy/0.log" Sep 17 02:03:50.833698 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:50.833612 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76f657cdc4-p72fm_d3faa76e-121f-491e-a616-1ac261e85583/payload-pre-processing/0.log" Sep 17 02:03:50.849052 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:50.849025 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-wh5fn_6a26ca5a-c33d-42f7-96c2-cfa79451016c/payload-processing/0.log" Sep 17 02:03:50.870245 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:50.870210 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-585fc78676-kqsfm_4d1b91db-e40c-48f8-8008-f01fd5801ec0/router/0.log" Sep 17 02:03:52.313644 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:52.313612 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-57kmn_ad0996a8-d893-486b-a3ee-4ee366bc558b/istio-proxy/0.log" Sep 17 02:03:52.766182 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:52.766151 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-76f657cdc4-p72fm_d3faa76e-121f-491e-a616-1ac261e85583/payload-pre-processing/0.log" Sep 17 02:03:52.780206 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:52.780175 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-85fd65c94c-wh5fn_6a26ca5a-c33d-42f7-96c2-cfa79451016c/payload-processing/0.log" Sep 17 02:03:52.798774 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:52.798743 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-585fc78676-kqsfm_4d1b91db-e40c-48f8-8008-f01fd5801ec0/router/0.log" Sep 17 02:03:54.906942 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:54.906909 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-g227w_69f841a5-fa85-4f3d-bbe6-0f86fec7af12/migrator/0.log" Sep 17 02:03:54.925230 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:54.925182 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-g227w_69f841a5-fa85-4f3d-bbe6-0f86fec7af12/graceful-termination/0.log" Sep 17 02:03:55.331555 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:55.331459 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-gtwjd_24dd28d0-d328-41a2-83bc-d392d123272b/kube-storage-version-migrator-operator/1.log" Sep 17 02:03:55.333503 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:55.333480 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-gtwjd_24dd28d0-d328-41a2-83bc-d392d123272b/kube-storage-version-migrator-operator/0.log" Sep 17 02:03:56.487317 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:56.487285 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m9ptd_766b09df-bf87-4ace-8430-628323cd59c9/kube-multus-additional-cni-plugins/0.log" Sep 17 02:03:56.505429 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:56.505403 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m9ptd_766b09df-bf87-4ace-8430-628323cd59c9/egress-router-binary-copy/0.log" Sep 17 02:03:56.523632 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:56.523603 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m9ptd_766b09df-bf87-4ace-8430-628323cd59c9/cni-plugins/0.log" Sep 17 02:03:56.542685 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:56.542655 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m9ptd_766b09df-bf87-4ace-8430-628323cd59c9/bond-cni-plugin/0.log" Sep 17 02:03:56.560802 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:56.560774 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m9ptd_766b09df-bf87-4ace-8430-628323cd59c9/routeoverride-cni/0.log" Sep 17 02:03:56.578034 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:56.577997 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m9ptd_766b09df-bf87-4ace-8430-628323cd59c9/whereabouts-cni-bincopy/0.log" Sep 17 02:03:56.596303 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:56.596273 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m9ptd_766b09df-bf87-4ace-8430-628323cd59c9/whereabouts-cni/0.log" Sep 17 02:03:56.811772 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:56.811679 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cbqj4_282d394d-c1cd-4368-bbb5-f8a67ab9c07b/kube-multus/0.log" Sep 17 02:03:56.952806 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:56.952776 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rdhqv_32c97ab8-07c6-4f64-9065-2441c8604a5a/network-metrics-daemon/0.log" Sep 17 02:03:56.967535 ip-10-0-138-83 kubenswrapper[2577]: I0917 02:03:56.967509 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rdhqv_32c97ab8-07c6-4f64-9065-2441c8604a5a/kube-rbac-proxy/0.log"