Sep 17 23:05:59.609901 ip-10-0-143-82 systemd[1]: Starting Kubernetes Kubelet... Sep 17 23:06:03.065094 ip-10-0-143-82 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 23:06:03.065094 ip-10-0-143-82 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 17 23:06:03.065094 ip-10-0-143-82 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 23:06:03.065094 ip-10-0-143-82 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 17 23:06:03.065094 ip-10-0-143-82 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 23:06:03.065928 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.065164 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 17 23:06:03.067905 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067889 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 23:06:03.067905 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067904 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 23:06:03.067962 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067908 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 23:06:03.067962 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067911 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 23:06:03.067962 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067914 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 23:06:03.067962 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067918 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 23:06:03.067962 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067922 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 23:06:03.067962 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067925 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 23:06:03.067962 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067927 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 23:06:03.067962 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067930 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 23:06:03.067962 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067933 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 23:06:03.067962 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067936 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 23:06:03.067962 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067938 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 23:06:03.067962 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067941 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 23:06:03.067962 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067950 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 23:06:03.067962 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067953 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 23:06:03.067962 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067956 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 23:06:03.067962 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067959 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 23:06:03.067962 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067961 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 23:06:03.067962 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067964 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 23:06:03.067962 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067967 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 23:06:03.068401 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067970 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 23:06:03.068401 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067972 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 23:06:03.068401 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067977 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 23:06:03.068401 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067981 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 23:06:03.068401 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067984 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 23:06:03.068401 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067987 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 23:06:03.068401 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067990 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 23:06:03.068401 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067993 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 23:06:03.068401 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067995 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 23:06:03.068401 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.067998 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 23:06:03.068401 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068001 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 23:06:03.068401 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068004 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 23:06:03.068401 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068006 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 23:06:03.068401 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068009 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 23:06:03.068401 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068014 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 23:06:03.068401 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068017 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 23:06:03.068401 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068020 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 23:06:03.068401 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068023 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 23:06:03.068401 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068025 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068028 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068030 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068033 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068035 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068038 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068040 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068043 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068045 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068048 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068050 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068053 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068055 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068057 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068060 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068064 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068066 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068069 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068071 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068074 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 23:06:03.068892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068076 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068079 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068082 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068085 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068087 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068090 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068092 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068094 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068097 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068100 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068102 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068105 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068107 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068109 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068112 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068114 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068117 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068119 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068122 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068125 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 23:06:03.069362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068128 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068130 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068133 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068135 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068138 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068140 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068143 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068146 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068148 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068151 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068153 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068156 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068158 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068161 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068163 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068166 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068168 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068170 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068173 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068175 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 23:06:03.069883 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068852 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 23:06:03.070345 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068858 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 23:06:03.070345 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068861 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 23:06:03.070345 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068866 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 23:06:03.070345 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068870 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 23:06:03.070345 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068873 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 23:06:03.070345 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068876 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 23:06:03.070345 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068879 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 23:06:03.070345 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068883 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 23:06:03.070345 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068886 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 23:06:03.070345 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068888 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 23:06:03.070345 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068892 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 23:06:03.070345 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068895 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 23:06:03.070345 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068898 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 23:06:03.070345 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068901 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 23:06:03.070345 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068904 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 23:06:03.070345 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068906 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 23:06:03.070345 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068909 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 23:06:03.070345 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068912 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 23:06:03.070345 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068915 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 23:06:03.070847 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068918 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 23:06:03.070847 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068921 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 23:06:03.070847 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068923 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 23:06:03.070847 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068926 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 23:06:03.070847 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068928 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 23:06:03.070847 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068931 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 23:06:03.070847 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068933 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 23:06:03.070847 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068936 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 23:06:03.070847 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068938 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 23:06:03.070847 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068941 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 23:06:03.070847 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068943 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 23:06:03.070847 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068946 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 23:06:03.070847 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068949 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 23:06:03.070847 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068951 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 23:06:03.070847 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068954 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 23:06:03.070847 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068956 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 23:06:03.070847 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068959 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 23:06:03.070847 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068961 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 23:06:03.070847 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068964 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 23:06:03.071340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068966 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 23:06:03.071340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068969 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 23:06:03.071340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068971 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 23:06:03.071340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068974 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 23:06:03.071340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068977 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 23:06:03.071340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068982 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 23:06:03.071340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068985 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 23:06:03.071340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068988 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 23:06:03.071340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068991 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 23:06:03.071340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068995 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 23:06:03.071340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.068998 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 23:06:03.071340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069001 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 23:06:03.071340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069004 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 23:06:03.071340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069006 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 23:06:03.071340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069009 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 23:06:03.071340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069011 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 23:06:03.071340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069014 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 23:06:03.071340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069016 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 23:06:03.071340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069019 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069021 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069024 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069035 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069038 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069040 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069044 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069047 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069049 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069052 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069055 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069058 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069060 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069063 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069065 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069068 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069070 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069073 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069076 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069078 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 23:06:03.071808 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069082 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069085 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069087 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069090 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069092 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069095 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069098 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069101 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069103 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069106 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069108 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069111 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069114 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069116 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069119 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069121 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069124 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069126 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069129 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069132 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 23:06:03.072289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069134 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 23:06:03.072771 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069137 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 23:06:03.072771 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069522 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 23:06:03.072771 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069528 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 23:06:03.072771 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069531 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 23:06:03.072771 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069534 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 23:06:03.072771 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069536 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 23:06:03.072771 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069539 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 23:06:03.072771 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069542 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 23:06:03.072771 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069544 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 23:06:03.072771 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069547 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 23:06:03.072771 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069549 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 23:06:03.072771 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069552 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 23:06:03.072771 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069555 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 23:06:03.072771 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069560 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 23:06:03.072771 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069563 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 23:06:03.072771 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069566 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 23:06:03.072771 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069569 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 23:06:03.072771 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069572 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 23:06:03.072771 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069575 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 23:06:03.073226 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069578 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 23:06:03.073226 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069581 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 23:06:03.073226 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069584 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 23:06:03.073226 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069587 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 23:06:03.073226 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069589 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 23:06:03.073226 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069592 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 23:06:03.073226 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069595 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 23:06:03.073226 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069597 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 23:06:03.073226 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069600 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 23:06:03.073226 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069602 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 23:06:03.073226 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069605 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 23:06:03.073226 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069607 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 23:06:03.073226 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069610 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 23:06:03.073226 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069613 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 23:06:03.073226 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069616 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 23:06:03.073226 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069619 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 23:06:03.073226 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069622 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 23:06:03.073226 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069624 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 23:06:03.073226 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069627 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069630 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069634 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069637 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069640 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069642 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069645 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069648 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069650 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069653 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069655 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069658 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069660 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069663 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069666 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069668 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069671 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069673 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069676 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069680 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 23:06:03.073690 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069683 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 23:06:03.074156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069685 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 23:06:03.074156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069688 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 23:06:03.074156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069690 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 23:06:03.074156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069693 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 23:06:03.074156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069695 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 23:06:03.074156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069698 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 23:06:03.074156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069700 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 23:06:03.074156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069703 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 23:06:03.074156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069705 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 23:06:03.074156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069707 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 23:06:03.074156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069710 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 23:06:03.074156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069714 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 23:06:03.074156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069717 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 23:06:03.074156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069720 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 23:06:03.074156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069723 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 23:06:03.074156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069725 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 23:06:03.074156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069728 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 23:06:03.074156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069730 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 23:06:03.074156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069733 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069738 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069741 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069743 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069746 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069749 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069751 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069754 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069757 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069759 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069762 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069765 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069769 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069771 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069774 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069777 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069779 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069781 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069784 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069786 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 23:06:03.074611 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069789 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069791 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069794 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.069796 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069861 2582 flags.go:64] FLAG: --address="0.0.0.0" Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069893 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069899 2582 flags.go:64] FLAG: --anonymous-auth="true" Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069904 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069908 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069911 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069915 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069919 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069922 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069925 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069928 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069932 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069935 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069938 2582 flags.go:64] FLAG: --cgroup-root="" Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069941 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069944 2582 flags.go:64] FLAG: --client-ca-file="" Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069947 2582 flags.go:64] FLAG: --cloud-provider="external" Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069950 2582 flags.go:64] FLAG: --cluster-dns="[]" Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069954 2582 flags.go:64] FLAG: --cluster-domain="" Sep 17 23:06:03.075079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069957 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069963 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069966 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069969 2582 flags.go:64] FLAG: --container-log-max-files="5" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069973 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069976 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069979 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069982 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069985 2582 flags.go:64] FLAG: --contention-profiling="false" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069988 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069990 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069993 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.069996 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070000 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070003 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070005 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070008 2582 flags.go:64] FLAG: --enable-load-reader="false" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070011 2582 flags.go:64] FLAG: --enable-server="true" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070014 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070019 2582 flags.go:64] FLAG: --event-burst="100" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070022 2582 flags.go:64] FLAG: --event-qps="50" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070025 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070028 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070031 2582 flags.go:64] FLAG: --eviction-hard="" Sep 17 23:06:03.075624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070035 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070038 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070042 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070045 2582 flags.go:64] FLAG: --eviction-soft="" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070048 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070051 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070054 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070057 2582 flags.go:64] FLAG: --experimental-mounter-path="" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070060 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070064 2582 flags.go:64] FLAG: --fail-swap-on="true" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070066 2582 flags.go:64] FLAG: --feature-gates="" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070070 2582 flags.go:64] FLAG: --file-check-frequency="20s" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070073 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070076 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070079 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070082 2582 flags.go:64] FLAG: --healthz-port="10248" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070085 2582 flags.go:64] FLAG: --help="false" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070088 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-143-82.ec2.internal" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070091 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070094 2582 flags.go:64] FLAG: --http-check-frequency="20s" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070097 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070101 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070104 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070107 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 17 23:06:03.076220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070110 2582 flags.go:64] FLAG: --image-service-endpoint="" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070112 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070115 2582 flags.go:64] FLAG: --kube-api-burst="100" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070118 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070121 2582 flags.go:64] FLAG: --kube-api-qps="50" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070124 2582 flags.go:64] FLAG: --kube-reserved="" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070127 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070129 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070132 2582 flags.go:64] FLAG: --kubelet-cgroups="" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070136 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070140 2582 flags.go:64] FLAG: --lock-file="" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070143 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070146 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070149 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070154 2582 flags.go:64] FLAG: --log-json-split-stream="false" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070156 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070159 2582 flags.go:64] FLAG: --log-text-split-stream="false" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070163 2582 flags.go:64] FLAG: --logging-format="text" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070166 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070169 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070172 2582 flags.go:64] FLAG: --manifest-url="" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070175 2582 flags.go:64] FLAG: --manifest-url-header="" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070179 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070182 2582 flags.go:64] FLAG: --max-open-files="1000000" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070186 2582 flags.go:64] FLAG: --max-pods="110" Sep 17 23:06:03.076817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070189 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070192 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070195 2582 flags.go:64] FLAG: --memory-manager-policy="None" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070198 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070201 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070203 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070206 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070213 2582 flags.go:64] FLAG: --node-status-max-images="50" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070216 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070219 2582 flags.go:64] FLAG: --oom-score-adj="-999" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070222 2582 flags.go:64] FLAG: --pod-cidr="" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070224 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070230 2582 flags.go:64] FLAG: --pod-manifest-path="" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070233 2582 flags.go:64] FLAG: --pod-max-pids="-1" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070236 2582 flags.go:64] FLAG: --pods-per-core="0" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070239 2582 flags.go:64] FLAG: --port="10250" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070241 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070245 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01ff131ebcbf1cece" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070249 2582 flags.go:64] FLAG: --qos-reserved="" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070252 2582 flags.go:64] FLAG: --read-only-port="10255" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070255 2582 flags.go:64] FLAG: --register-node="true" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070258 2582 flags.go:64] FLAG: --register-with-taints="" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070261 2582 flags.go:64] FLAG: --registry-burst="10" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070264 2582 flags.go:64] FLAG: --registry-qps="5" Sep 17 23:06:03.077405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070269 2582 flags.go:64] FLAG: --reserved-cpus="" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070272 2582 flags.go:64] FLAG: --reserved-memory="" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070276 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070279 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070282 2582 flags.go:64] FLAG: --rotate-certificates="false" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070285 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070288 2582 flags.go:64] FLAG: --runonce="false" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070290 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070294 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070297 2582 flags.go:64] FLAG: --seccomp-default="false" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070300 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070303 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070306 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070309 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070312 2582 flags.go:64] FLAG: --storage-driver-password="root" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070315 2582 flags.go:64] FLAG: --storage-driver-secure="false" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070318 2582 flags.go:64] FLAG: --storage-driver-table="stats" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070321 2582 flags.go:64] FLAG: --storage-driver-user="root" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070324 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070327 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070330 2582 flags.go:64] FLAG: --system-cgroups="" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070333 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070338 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070341 2582 flags.go:64] FLAG: --tls-cert-file="" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070343 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 17 23:06:03.077982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070347 2582 flags.go:64] FLAG: --tls-min-version="" Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070352 2582 flags.go:64] FLAG: --tls-private-key-file="" Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070355 2582 flags.go:64] FLAG: --topology-manager-policy="none" Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070358 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070361 2582 flags.go:64] FLAG: --topology-manager-scope="container" Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070364 2582 flags.go:64] FLAG: --v="2" Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070368 2582 flags.go:64] FLAG: --version="false" Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070373 2582 flags.go:64] FLAG: --vmodule="" Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070377 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.070380 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070484 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070488 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070491 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070494 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070509 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070512 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070515 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070518 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070521 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070523 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070526 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070529 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 23:06:03.078659 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070531 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070534 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070537 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070539 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070542 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070544 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070547 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070550 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070552 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070555 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070557 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070562 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070565 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070568 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070570 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070573 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070576 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070579 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070581 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070584 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 23:06:03.079367 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070586 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070589 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070592 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070594 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070597 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070599 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070602 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070604 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070607 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070609 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070612 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070614 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070617 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070620 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070622 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070625 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070627 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070630 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070633 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070635 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 23:06:03.079960 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070638 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070640 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070643 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070646 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070649 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070652 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070654 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070658 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070662 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070665 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070667 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070670 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070672 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070675 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070678 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070680 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070682 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070686 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070690 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070693 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 23:06:03.080483 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070696 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 23:06:03.080985 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070699 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 23:06:03.080985 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070702 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 23:06:03.080985 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070704 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 23:06:03.080985 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070707 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 23:06:03.080985 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070710 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 23:06:03.080985 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070713 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 23:06:03.080985 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070715 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 23:06:03.080985 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070718 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 23:06:03.080985 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070721 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 23:06:03.080985 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070723 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 23:06:03.080985 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070726 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 23:06:03.080985 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070728 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 23:06:03.080985 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070731 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 23:06:03.080985 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070733 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 23:06:03.080985 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070738 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 23:06:03.080985 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070742 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 23:06:03.080985 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070745 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 23:06:03.080985 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070748 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 23:06:03.080985 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070751 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 23:06:03.081448 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070754 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 23:06:03.081448 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070756 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 23:06:03.081448 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070759 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 23:06:03.081448 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070761 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 23:06:03.081448 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070764 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 23:06:03.081448 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070767 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 23:06:03.081448 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.070769 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 23:06:03.081448 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.071611 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 23:06:03.081448 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.080168 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 17 23:06:03.081448 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.080182 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 17 23:06:03.081448 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.080202 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 17 23:06:03.081448 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.080211 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 17 23:06:03.081448 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080264 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 23:06:03.081448 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080269 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 23:06:03.081448 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080272 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080275 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080278 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080281 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080284 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080287 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080290 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080293 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080296 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080298 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080301 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080303 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080306 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080309 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080311 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080314 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080316 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080319 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080322 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080325 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 23:06:03.081828 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080327 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 23:06:03.082300 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080330 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 23:06:03.082300 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080333 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 23:06:03.082300 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080335 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 23:06:03.082300 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080338 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 23:06:03.082300 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080341 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 23:06:03.082300 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080343 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 23:06:03.082300 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080347 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 23:06:03.082300 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080349 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 23:06:03.082300 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080352 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 23:06:03.082300 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080355 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 23:06:03.082300 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080357 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 23:06:03.082300 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080360 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 23:06:03.082300 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080362 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 23:06:03.082300 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080365 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 23:06:03.082300 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080368 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 23:06:03.082300 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080370 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 23:06:03.082300 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080374 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 23:06:03.082300 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080378 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 23:06:03.082300 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080381 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080384 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080387 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080389 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080392 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080395 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080398 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080400 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080403 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080406 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080408 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080411 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080413 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080416 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080419 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080421 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080424 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080426 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080429 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080431 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 23:06:03.082783 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080434 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080437 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080440 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080442 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080445 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080447 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080450 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080452 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080455 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080458 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080460 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080463 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080466 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080468 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080471 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080473 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080476 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080479 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080482 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080485 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 23:06:03.083247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080487 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 23:06:03.083721 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080490 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 23:06:03.083721 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080493 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 23:06:03.083721 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080513 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 23:06:03.083721 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080517 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 23:06:03.083721 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080521 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 23:06:03.083721 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080525 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 23:06:03.083721 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080528 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 23:06:03.083721 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080531 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 23:06:03.083721 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080534 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 23:06:03.083721 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080536 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 23:06:03.083721 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080539 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 23:06:03.083721 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080542 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 23:06:03.083721 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080561 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 23:06:03.083721 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080565 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 23:06:03.083721 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080568 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 23:06:03.083721 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080571 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 23:06:03.083721 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080574 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 23:06:03.084141 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.080579 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 23:06:03.084141 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080694 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 23:06:03.084141 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080699 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 23:06:03.084141 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080702 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 23:06:03.084141 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080705 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 23:06:03.084141 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080708 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 23:06:03.084141 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080711 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 23:06:03.084141 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080713 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 23:06:03.084141 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080716 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 23:06:03.084141 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080718 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 23:06:03.084141 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080721 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 23:06:03.084141 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080724 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 23:06:03.084141 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080726 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 23:06:03.084141 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080729 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 23:06:03.084141 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080732 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080734 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080737 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080739 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080742 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080744 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080747 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080749 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080752 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080755 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080757 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080760 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080762 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080765 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080768 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080770 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080773 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080776 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080778 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080780 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 23:06:03.084490 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080783 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080786 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080788 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080791 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080793 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080796 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080798 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080801 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080803 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080806 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080808 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080810 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080813 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080816 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080818 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080821 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080823 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080826 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080829 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080831 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 23:06:03.085053 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080834 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 23:06:03.086025 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080836 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 23:06:03.086025 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080838 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 23:06:03.086025 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080841 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 23:06:03.086025 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080843 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 23:06:03.086025 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080846 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 23:06:03.086025 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080850 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 23:06:03.086025 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080853 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 23:06:03.086025 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080856 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 23:06:03.086025 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080859 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 23:06:03.086025 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080863 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 23:06:03.086025 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080866 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 23:06:03.086025 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080869 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 23:06:03.086025 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080872 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 23:06:03.086025 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080875 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 23:06:03.086025 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080878 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 23:06:03.086025 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080881 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 23:06:03.086025 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080883 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 23:06:03.086025 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080886 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 23:06:03.086025 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080889 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080891 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080894 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080896 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080898 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080901 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080904 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080906 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080909 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080912 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080914 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080917 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080919 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080921 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080924 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080926 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080929 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080931 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080936 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080939 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 23:06:03.086520 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080941 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 23:06:03.087044 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080944 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 23:06:03.087044 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080946 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 23:06:03.087044 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080949 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 23:06:03.087044 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080951 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 23:06:03.087044 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080954 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 23:06:03.087044 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:03.080957 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 23:06:03.087044 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.080962 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 23:06:03.087044 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.081068 2582 server.go:962] "Client rotation is on, will bootstrap in background" Sep 17 23:06:03.087044 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.083729 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 17 23:06:03.087044 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.084843 2582 server.go:1019] "Starting client certificate rotation" Sep 17 23:06:03.087044 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.084960 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 23:06:03.087044 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.085001 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 23:06:03.087398 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.087315 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 23:06:03.088424 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.088405 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 23:06:03.093411 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.093396 2582 log.go:25] "Validated CRI v1 runtime API" Sep 17 23:06:03.094995 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.094981 2582 log.go:25] "Validated CRI v1 image API" Sep 17 23:06:03.095827 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.095814 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 17 23:06:03.097842 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.097821 2582 fs.go:135] Filesystem UUIDs: map[3822c8c2-39ec-4672-af64-2c6db63edc12:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9a7a0e30-a785-4ef3-99a6-c94dee63e4d7:/dev/nvme0n1p3] Sep 17 23:06:03.097899 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.097843 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 17 23:06:03.103553 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.103417 2582 manager.go:216] Machine: {Timestamp:2026-09-17 23:06:03.101969257 +0000 UTC m=+2.035876383 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199476 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a614ffb9b2a5edccadc8618af62f4 SystemUUID:ec2a614f-fb9b-2a5e-dcca-dc8618af62f4 BootID:dd936bb2-5bee-44b7-8d76-a9c0d817e1e7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c6:19:33:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c6:19:33:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:f5:7f:7c:7d:fa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 17 23:06:03.103553 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.103552 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 17 23:06:03.103654 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.103640 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 17 23:06:03.105200 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.105175 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 17 23:06:03.105344 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.105211 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-82.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 17 23:06:03.106474 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.106465 2582 topology_manager.go:138] "Creating topology manager with none policy" Sep 17 23:06:03.106520 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.106477 2582 container_manager_linux.go:309] "Creating device plugin manager" Sep 17 23:06:03.106520 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.106485 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 23:06:03.106520 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.106506 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 23:06:03.106605 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.106573 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 17 23:06:03.109848 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.109833 2582 state_mem.go:36] "Initialized new in-memory state store" Sep 17 23:06:03.109967 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.109957 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 17 23:06:03.110031 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.109993 2582 kubelet.go:486] "Attempting to sync node with API server" Sep 17 23:06:03.110031 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.110005 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 17 23:06:03.110031 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.110022 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 17 23:06:03.110156 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.110034 2582 kubelet.go:398] "Adding apiserver pod source" Sep 17 23:06:03.110156 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.110043 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 17 23:06:03.110441 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.110416 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 23:06:03.110526 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.110453 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 23:06:03.112670 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.112653 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 17 23:06:03.113466 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.113448 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 23:06:03.115033 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.115015 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 17 23:06:03.115088 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.115049 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 17 23:06:03.116285 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.116274 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 17 23:06:03.116335 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.116289 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 17 23:06:03.116335 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.116295 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 17 23:06:03.116335 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.116301 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 17 23:06:03.116335 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.116309 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 17 23:06:03.116335 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.116315 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 17 23:06:03.116335 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.116328 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 17 23:06:03.116335 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.116334 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 17 23:06:03.116335 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.116340 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 17 23:06:03.116584 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.116346 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 17 23:06:03.116584 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.116355 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 17 23:06:03.116584 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.116359 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 17 23:06:03.116584 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.116382 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 17 23:06:03.116584 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.116387 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 17 23:06:03.117931 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.117912 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 17 23:06:03.118034 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.118015 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 17 23:06:03.127903 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.127887 2582 server.go:1268] "Started kubelet" Sep 17 23:06:03.128019 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.127991 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 17 23:06:03.128372 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.127994 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 17 23:06:03.128372 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.128169 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 17 23:06:03.128786 ip-10-0-143-82 systemd[1]: Started Kubernetes Kubelet. Sep 17 23:06:03.129257 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.129168 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 17 23:06:03.129995 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.129791 2582 server.go:310] "Adding debug handlers to kubelet server" Sep 17 23:06:03.132011 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.131995 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-82.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 17 23:06:03.133509 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.133470 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 17 23:06:03.135698 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.135680 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 17 23:06:03.135698 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.135689 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 17 23:06:03.135837 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.135757 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-82.ec2.internal\" not found" Sep 17 23:06:03.136311 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.136294 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 17 23:06:03.136311 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.136297 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 17 23:06:03.136455 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.136325 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 17 23:06:03.136455 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.136413 2582 reconstruct.go:97] "Volume reconstruction finished" Sep 17 23:06:03.136455 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.136422 2582 reconciler.go:29] "Reconciler: start to sync state" Sep 17 23:06:03.136648 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.136617 2582 factory.go:55] Registering systemd factory Sep 17 23:06:03.136648 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.136640 2582 factory.go:223] Registration of the systemd container factory successfully Sep 17 23:06:03.136880 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.136860 2582 factory.go:153] Registering CRI-O factory Sep 17 23:06:03.136880 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.136879 2582 factory.go:223] Registration of the crio container factory successfully Sep 17 23:06:03.136965 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.136942 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 17 23:06:03.136965 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.136963 2582 factory.go:103] Registering Raw factory Sep 17 23:06:03.137066 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.136977 2582 manager.go:1196] Started watching for new ooms in manager Sep 17 23:06:03.137509 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.137483 2582 manager.go:349] Starting recovery of all containers Sep 17 23:06:03.138676 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.138658 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-swvx7" Sep 17 23:06:03.164908 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.164888 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-swvx7" Sep 17 23:06:03.165033 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.164886 2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 17 23:06:03.165033 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.164982 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 17 23:06:03.171477 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.170196 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-82.ec2.internal.18d63e43dd98c581 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-82.ec2.internal,UID:ip-10-0-143-82.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-82.ec2.internal,},FirstTimestamp:2026-09-17 23:06:03.127858561 +0000 UTC m=+2.061765687,LastTimestamp:2026-09-17 23:06:03.127858561 +0000 UTC m=+2.061765687,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-82.ec2.internal,}" Sep 17 23:06:03.173299 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.173278 2582 manager.go:354] Recovery completed Sep 17 23:06:03.177110 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.177095 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 23:06:03.180047 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.180031 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasSufficientMemory" Sep 17 23:06:03.180116 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.180061 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 23:06:03.180116 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.180071 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasSufficientPID" Sep 17 23:06:03.180606 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.180592 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 17 23:06:03.180606 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.180604 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 17 23:06:03.180684 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.180610 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 23:06:03.180684 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.180618 2582 state_mem.go:36] "Initialized new in-memory state store" Sep 17 23:06:03.181724 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.181706 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasSufficientMemory" Sep 17 23:06:03.181801 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.181738 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 23:06:03.181801 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.181753 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasSufficientPID" Sep 17 23:06:03.181801 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.181760 2582 policy_none.go:49] "None policy: Start" Sep 17 23:06:03.181801 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.181773 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 17 23:06:03.181801 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.181785 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 17 23:06:03.182794 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.182784 2582 policy_none.go:47] "Start" Sep 17 23:06:03.218052 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.218038 2582 manager.go:337] "Starting Device Plugin manager" Sep 17 23:06:03.229649 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.218072 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 17 23:06:03.229649 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.218082 2582 server.go:87] "Starting device plugin registration server" Sep 17 23:06:03.229649 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.218329 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 17 23:06:03.229649 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.218351 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 17 23:06:03.229649 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.218473 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 17 23:06:03.229649 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.218561 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 17 23:06:03.229649 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.218570 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 17 23:06:03.229649 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.218941 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 17 23:06:03.229649 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.218986 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-82.ec2.internal\" not found" Sep 17 23:06:03.285674 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.285641 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 17 23:06:03.286746 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.286729 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 17 23:06:03.286834 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.286784 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 17 23:06:03.286834 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.286811 2582 kubelet.go:2448] "Starting kubelet main sync loop" Sep 17 23:06:03.286920 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.286857 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 17 23:06:03.289071 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.289048 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 23:06:03.319370 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.319304 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 23:06:03.320197 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.320182 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasSufficientMemory" Sep 17 23:06:03.320283 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.320208 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 23:06:03.320283 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.320219 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasSufficientPID" Sep 17 23:06:03.320283 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.320246 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-82.ec2.internal" Sep 17 23:06:03.327726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.327709 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-82.ec2.internal" Sep 17 23:06:03.327811 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.327734 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-82.ec2.internal\": node \"ip-10-0-143-82.ec2.internal\" not found" Sep 17 23:06:03.343832 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.343815 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-82.ec2.internal\" not found" Sep 17 23:06:03.387660 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.387634 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-82.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal"] Sep 17 23:06:03.387770 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.387707 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 23:06:03.388626 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.388609 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasSufficientMemory" Sep 17 23:06:03.388710 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.388638 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 23:06:03.388710 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.388648 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasSufficientPID" Sep 17 23:06:03.390062 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.390049 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 23:06:03.390178 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.390165 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-82.ec2.internal" Sep 17 23:06:03.390233 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.390198 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 23:06:03.390791 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.390776 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasSufficientMemory" Sep 17 23:06:03.390885 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.390798 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 23:06:03.390885 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.390807 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasSufficientPID" Sep 17 23:06:03.390885 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.390781 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasSufficientMemory" Sep 17 23:06:03.390885 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.390868 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 23:06:03.390885 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.390877 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasSufficientPID" Sep 17 23:06:03.391956 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.391938 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal" Sep 17 23:06:03.392038 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.391967 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 23:06:03.392636 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.392622 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasSufficientMemory" Sep 17 23:06:03.392694 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.392652 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 23:06:03.392694 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.392666 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeHasSufficientPID" Sep 17 23:06:03.423144 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.423120 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-82.ec2.internal\" not found" node="ip-10-0-143-82.ec2.internal" Sep 17 23:06:03.426334 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.426316 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-82.ec2.internal\" not found" node="ip-10-0-143-82.ec2.internal" Sep 17 23:06:03.437969 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.437955 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9b0067a4916205e46911412e53db8082-config\") pod \"kube-apiserver-proxy-ip-10-0-143-82.ec2.internal\" (UID: \"9b0067a4916205e46911412e53db8082\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-82.ec2.internal" Sep 17 23:06:03.438041 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.437977 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c42993043d7f11ec0cbc6c6de37a429-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal\" (UID: \"7c42993043d7f11ec0cbc6c6de37a429\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal" Sep 17 23:06:03.438041 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.437993 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c42993043d7f11ec0cbc6c6de37a429-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal\" (UID: \"7c42993043d7f11ec0cbc6c6de37a429\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal" Sep 17 23:06:03.444015 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.444000 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-82.ec2.internal\" not found" Sep 17 23:06:03.538588 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.538566 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9b0067a4916205e46911412e53db8082-config\") pod \"kube-apiserver-proxy-ip-10-0-143-82.ec2.internal\" (UID: \"9b0067a4916205e46911412e53db8082\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-82.ec2.internal" Sep 17 23:06:03.538672 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.538592 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c42993043d7f11ec0cbc6c6de37a429-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal\" (UID: \"7c42993043d7f11ec0cbc6c6de37a429\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal" Sep 17 23:06:03.538672 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.538608 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c42993043d7f11ec0cbc6c6de37a429-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal\" (UID: \"7c42993043d7f11ec0cbc6c6de37a429\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal" Sep 17 23:06:03.538672 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.538633 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c42993043d7f11ec0cbc6c6de37a429-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal\" (UID: \"7c42993043d7f11ec0cbc6c6de37a429\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal" Sep 17 23:06:03.538672 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.538659 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9b0067a4916205e46911412e53db8082-config\") pod \"kube-apiserver-proxy-ip-10-0-143-82.ec2.internal\" (UID: \"9b0067a4916205e46911412e53db8082\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-82.ec2.internal" Sep 17 23:06:03.538793 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.538659 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c42993043d7f11ec0cbc6c6de37a429-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal\" (UID: \"7c42993043d7f11ec0cbc6c6de37a429\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal" Sep 17 23:06:03.544494 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.544474 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-82.ec2.internal\" not found" Sep 17 23:06:03.645368 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.645289 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-82.ec2.internal\" not found" Sep 17 23:06:03.727531 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.727511 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-82.ec2.internal" Sep 17 23:06:03.730239 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.730214 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal" Sep 17 23:06:03.745615 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.745593 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-82.ec2.internal\" not found" Sep 17 23:06:03.846097 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.846056 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-82.ec2.internal\" not found" Sep 17 23:06:03.946653 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:03.946588 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-82.ec2.internal\" not found" Sep 17 23:06:03.966172 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:03.966156 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 23:06:04.024736 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.024716 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 23:06:04.036348 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.036322 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-82.ec2.internal" Sep 17 23:06:04.045680 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.045665 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 23:06:04.047366 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.047347 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal" Sep 17 23:06:04.057091 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.057077 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 23:06:04.084254 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.084230 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 17 23:06:04.084657 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.084350 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 23:06:04.084657 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.084394 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 23:06:04.084657 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.084394 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 23:06:04.110533 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.110513 2582 apiserver.go:52] "Watching apiserver" Sep 17 23:06:04.127090 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.127065 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 17 23:06:04.127405 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.127384 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-js95g","kube-system/global-pull-secret-syncer-drknm","kube-system/kube-apiserver-proxy-ip-10-0-143-82.ec2.internal","openshift-cluster-node-tuning-operator/tuned-5s5pl","openshift-dns/node-resolver-k2bg2","openshift-image-registry/node-ca-trs6m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal","openshift-multus/multus-additional-cni-plugins-jqrzv","openshift-multus/multus-zzcb8","kube-system/konnectivity-agent-svq9k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t","openshift-multus/network-metrics-daemon-xg2sz","openshift-network-diagnostics/network-check-target-q8sw2","openshift-network-operator/iptables-alerter-d8d5w"] Sep 17 23:06:04.128732 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.128717 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-svq9k" Sep 17 23:06:04.129772 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.129755 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:04.129855 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.129834 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8sw2" podUID="62ca42a6-c013-4fce-b108-acefb7dab500" Sep 17 23:06:04.130895 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.130878 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d8d5w" Sep 17 23:06:04.130972 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.130958 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m7zl9\"" Sep 17 23:06:04.131025 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.130884 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 17 23:06:04.131025 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.131015 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 17 23:06:04.132065 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.132050 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.133122 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.133104 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ltsbr\"" Sep 17 23:06:04.133341 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.133325 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 17 23:06:04.133400 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.133331 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k2bg2" Sep 17 23:06:04.133475 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.133454 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 17 23:06:04.133692 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.133677 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 17 23:06:04.134302 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.134280 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sjvpx\"" Sep 17 23:06:04.134393 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.134318 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 17 23:06:04.134393 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.134335 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 17 23:06:04.134540 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.134528 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-trs6m" Sep 17 23:06:04.135270 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.135257 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 17 23:06:04.135623 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.135608 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-czpdx\"" Sep 17 23:06:04.135676 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.135622 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 17 23:06:04.135676 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.135646 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.135779 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.135764 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 17 23:06:04.136535 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.136514 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 17 23:06:04.136630 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.136536 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c2mkq\"" Sep 17 23:06:04.136677 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.136663 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 17 23:06:04.136715 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.136706 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 17 23:06:04.136910 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.136894 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.138036 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.138022 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:04.138100 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.138076 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drknm" podUID="66bdcd7b-3b69-4bdd-9579-9e035e9d19b0" Sep 17 23:06:04.138172 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.138156 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 17 23:06:04.138550 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.138537 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 17 23:06:04.138603 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.138591 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 17 23:06:04.138775 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.138762 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 17 23:06:04.138910 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.138897 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-blzvj\"" Sep 17 23:06:04.139214 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.139200 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 17 23:06:04.139248 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.139215 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 17 23:06:04.139294 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.139216 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bht7f\"" Sep 17 23:06:04.139337 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.139327 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.140575 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.140560 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.141683 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.141667 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-cnibin\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.141683 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.141689 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-host-run-k8s-cni-cncf-io\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.141861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.141706 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fc91b818-8121-4735-81d4-2da07c40b110-iptables-alerter-script\") pod \"iptables-alerter-d8d5w\" (UID: \"fc91b818-8121-4735-81d4-2da07c40b110\") " pod="openshift-network-operator/iptables-alerter-d8d5w" Sep 17 23:06:04.141861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.141722 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc91b818-8121-4735-81d4-2da07c40b110-host-slash\") pod \"iptables-alerter-d8d5w\" (UID: \"fc91b818-8121-4735-81d4-2da07c40b110\") " pod="openshift-network-operator/iptables-alerter-d8d5w" Sep 17 23:06:04.141861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.141739 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/938149bc-e432-42d5-8ff3-0230b863718f-etc-tuned\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.141861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.141762 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 17 23:06:04.141861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.141799 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bdbecc0a-d250-434e-8478-2f6f57c094d3-serviceca\") pod \"node-ca-trs6m\" (UID: \"bdbecc0a-d250-434e-8478-2f6f57c094d3\") " pod="openshift-image-registry/node-ca-trs6m" Sep 17 23:06:04.141861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.141842 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl4z7\" (UniqueName: \"kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7\") pod \"network-check-target-q8sw2\" (UID: \"62ca42a6-c013-4fce-b108-acefb7dab500\") " pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:04.141861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.141854 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:04.142169 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.141868 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-multus-cni-dir\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.142169 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.141893 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-etc-kubernetes\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.142169 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.141917 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2sz" podUID="3752b568-d53f-49ac-9095-0f61cee96767" Sep 17 23:06:04.142169 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.141925 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttp2h\" (UniqueName: \"kubernetes.io/projected/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-kube-api-access-ttp2h\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.142169 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.141961 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bdbecc0a-d250-434e-8478-2f6f57c094d3-host\") pod \"node-ca-trs6m\" (UID: \"bdbecc0a-d250-434e-8478-2f6f57c094d3\") " pod="openshift-image-registry/node-ca-trs6m" Sep 17 23:06:04.142169 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.141980 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 17 23:06:04.142169 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.141992 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-etc-systemd\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.142169 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.141995 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 17 23:06:04.142169 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142024 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-host\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.142169 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142065 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-cni-binary-copy\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.142169 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142091 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-host-var-lib-cni-bin\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.142169 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142117 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4c08617-40c6-47d5-bcc8-e4765ca7a477-cnibin\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.142169 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142143 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-var-lib-kubelet\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.142798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142184 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbda2aea-1bb6-4a24-a74f-fc8d79b07586-tmp-dir\") pod \"node-resolver-k2bg2\" (UID: \"bbda2aea-1bb6-4a24-a74f-fc8d79b07586\") " pod="openshift-dns/node-resolver-k2bg2" Sep 17 23:06:04.142798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142230 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zktw\" (UniqueName: \"kubernetes.io/projected/bbda2aea-1bb6-4a24-a74f-fc8d79b07586-kube-api-access-8zktw\") pod \"node-resolver-k2bg2\" (UID: \"bbda2aea-1bb6-4a24-a74f-fc8d79b07586\") " pod="openshift-dns/node-resolver-k2bg2" Sep 17 23:06:04.142798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142266 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-host-var-lib-cni-multus\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.142798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142287 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4c08617-40c6-47d5-bcc8-e4765ca7a477-system-cni-dir\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.142798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142305 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4c08617-40c6-47d5-bcc8-e4765ca7a477-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.142798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142336 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4c08617-40c6-47d5-bcc8-e4765ca7a477-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.142798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142356 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4c08617-40c6-47d5-bcc8-e4765ca7a477-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.142798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142396 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4518bc1-148d-4005-a3ec-2b36297214fa-agent-certs\") pod \"konnectivity-agent-svq9k\" (UID: \"d4518bc1-148d-4005-a3ec-2b36297214fa\") " pod="kube-system/konnectivity-agent-svq9k" Sep 17 23:06:04.142798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142418 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4518bc1-148d-4005-a3ec-2b36297214fa-konnectivity-ca\") pod \"konnectivity-agent-svq9k\" (UID: \"d4518bc1-148d-4005-a3ec-2b36297214fa\") " pod="kube-system/konnectivity-agent-svq9k" Sep 17 23:06:04.142798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142433 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65tvm\" (UniqueName: \"kubernetes.io/projected/938149bc-e432-42d5-8ff3-0230b863718f-kube-api-access-65tvm\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.142798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142462 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-multus-socket-dir-parent\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.142798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142526 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-host-run-netns\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.142798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142538 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vqvrf\"" Sep 17 23:06:04.142798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142554 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-hostroot\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.142798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142577 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-multus-conf-dir\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.142798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142600 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-host-run-multus-certs\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.142798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142619 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 17 23:06:04.143641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142625 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-etc-modprobe-d\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.143641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142647 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-etc-kubernetes\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.143641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142666 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-etc-sysctl-conf\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.143641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142692 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-run\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.143641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142712 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-lib-modules\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.143641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142734 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-host-var-lib-kubelet\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.143641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142760 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4c08617-40c6-47d5-bcc8-e4765ca7a477-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.143641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142783 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-etc-sysconfig\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.143641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142803 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-etc-sysctl-d\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.143641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142846 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/938149bc-e432-42d5-8ff3-0230b863718f-tmp\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.143641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142892 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-multus-daemon-config\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.143641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142921 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdh4v\" (UniqueName: \"kubernetes.io/projected/d4c08617-40c6-47d5-bcc8-e4765ca7a477-kube-api-access-sdh4v\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.143641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142952 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9qdf\" (UniqueName: \"kubernetes.io/projected/bdbecc0a-d250-434e-8478-2f6f57c094d3-kube-api-access-r9qdf\") pod \"node-ca-trs6m\" (UID: \"bdbecc0a-d250-434e-8478-2f6f57c094d3\") " pod="openshift-image-registry/node-ca-trs6m" Sep 17 23:06:04.143641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142975 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg7df\" (UniqueName: \"kubernetes.io/projected/fc91b818-8121-4735-81d4-2da07c40b110-kube-api-access-hg7df\") pod \"iptables-alerter-d8d5w\" (UID: \"fc91b818-8121-4735-81d4-2da07c40b110\") " pod="openshift-network-operator/iptables-alerter-d8d5w" Sep 17 23:06:04.143641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.142995 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-sys\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.143641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.143019 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbda2aea-1bb6-4a24-a74f-fc8d79b07586-hosts-file\") pod \"node-resolver-k2bg2\" (UID: \"bbda2aea-1bb6-4a24-a74f-fc8d79b07586\") " pod="openshift-dns/node-resolver-k2bg2" Sep 17 23:06:04.144119 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.143041 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-system-cni-dir\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.144119 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.143294 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 17 23:06:04.144119 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.143464 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 17 23:06:04.144119 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.143478 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 17 23:06:04.144119 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.143523 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 17 23:06:04.144119 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.143537 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 17 23:06:04.144119 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.143567 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v4xbp\"" Sep 17 23:06:04.152983 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.152965 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 23:06:04.169885 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.169841 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 23:01:03 +0000 UTC" deadline="2028-03-21 07:27:44.917418539 +0000 UTC" Sep 17 23:06:04.169885 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.169881 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13208h21m40.747540061s" Sep 17 23:06:04.172744 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.172727 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kks94" Sep 17 23:06:04.178824 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.178804 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kks94" Sep 17 23:06:04.237920 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.237753 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 17 23:06:04.244107 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244086 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-multus-socket-dir-parent\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.244220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244112 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-multus-conf-dir\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.244220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244133 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/24a660d0-e7cf-402b-8b09-9fb5141b67ba-socket-dir\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.244220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244148 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-log-socket\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.244220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244164 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-env-overrides\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.244220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244186 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-etc-kubernetes\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.244220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244212 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-etc-sysctl-conf\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.244471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244184 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-multus-conf-dir\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.244471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244230 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-etc-kubernetes\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.244471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244228 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-lib-modules\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.244471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244249 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-multus-socket-dir-parent\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.244471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244298 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-lib-modules\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.244471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244295 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-host-var-lib-kubelet\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.244471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244333 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-etc-sysconfig\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.244471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244338 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-etc-sysctl-conf\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.244471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244331 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-host-var-lib-kubelet\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.244471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244354 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-etc-sysctl-d\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.244471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244383 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-etc-sysconfig\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.244471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244392 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/938149bc-e432-42d5-8ff3-0230b863718f-tmp\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.244471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244418 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-etc-sysctl-d\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.244471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244423 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/24a660d0-e7cf-402b-8b09-9fb5141b67ba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.244471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244445 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-cni-bin\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.244471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244476 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-ovnkube-config\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.245309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244492 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg7df\" (UniqueName: \"kubernetes.io/projected/fc91b818-8121-4735-81d4-2da07c40b110-kube-api-access-hg7df\") pod \"iptables-alerter-d8d5w\" (UID: \"fc91b818-8121-4735-81d4-2da07c40b110\") " pod="openshift-network-operator/iptables-alerter-d8d5w" Sep 17 23:06:04.245309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244530 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-sys\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.245309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244624 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbda2aea-1bb6-4a24-a74f-fc8d79b07586-hosts-file\") pod \"node-resolver-k2bg2\" (UID: \"bbda2aea-1bb6-4a24-a74f-fc8d79b07586\") " pod="openshift-dns/node-resolver-k2bg2" Sep 17 23:06:04.245309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244649 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-cnibin\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.245309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244665 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/24a660d0-e7cf-402b-8b09-9fb5141b67ba-etc-selinux\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.245309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244686 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbda2aea-1bb6-4a24-a74f-fc8d79b07586-hosts-file\") pod \"node-resolver-k2bg2\" (UID: \"bbda2aea-1bb6-4a24-a74f-fc8d79b07586\") " pod="openshift-dns/node-resolver-k2bg2" Sep 17 23:06:04.245309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244698 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-cnibin\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.245309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244733 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-run-systemd\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.245309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244762 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-cni-netd\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.245309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244780 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-ovnkube-script-lib\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.245309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244734 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-sys\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.245309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244806 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fc91b818-8121-4735-81d4-2da07c40b110-iptables-alerter-script\") pod \"iptables-alerter-d8d5w\" (UID: \"fc91b818-8121-4735-81d4-2da07c40b110\") " pod="openshift-network-operator/iptables-alerter-d8d5w" Sep 17 23:06:04.245309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244886 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc91b818-8121-4735-81d4-2da07c40b110-host-slash\") pod \"iptables-alerter-d8d5w\" (UID: \"fc91b818-8121-4735-81d4-2da07c40b110\") " pod="openshift-network-operator/iptables-alerter-d8d5w" Sep 17 23:06:04.245309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244913 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bdbecc0a-d250-434e-8478-2f6f57c094d3-serviceca\") pod \"node-ca-trs6m\" (UID: \"bdbecc0a-d250-434e-8478-2f6f57c094d3\") " pod="openshift-image-registry/node-ca-trs6m" Sep 17 23:06:04.245309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244937 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-kubelet\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.245309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244960 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-run-netns\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.245309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.244982 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-host\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.246106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245030 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-host\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.246106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245035 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4c08617-40c6-47d5-bcc8-e4765ca7a477-cnibin\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.246106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245013 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc91b818-8121-4735-81d4-2da07c40b110-host-slash\") pod \"iptables-alerter-d8d5w\" (UID: \"fc91b818-8121-4735-81d4-2da07c40b110\") " pod="openshift-network-operator/iptables-alerter-d8d5w" Sep 17 23:06:04.246106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245069 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4c08617-40c6-47d5-bcc8-e4765ca7a477-cnibin\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.246106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245070 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/24a660d0-e7cf-402b-8b09-9fb5141b67ba-device-dir\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.246106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245099 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/24a660d0-e7cf-402b-8b09-9fb5141b67ba-registration-dir\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.246106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245130 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-var-lib-kubelet\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.246106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245154 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbda2aea-1bb6-4a24-a74f-fc8d79b07586-tmp-dir\") pod \"node-resolver-k2bg2\" (UID: \"bbda2aea-1bb6-4a24-a74f-fc8d79b07586\") " pod="openshift-dns/node-resolver-k2bg2" Sep 17 23:06:04.246106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245177 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4c08617-40c6-47d5-bcc8-e4765ca7a477-system-cni-dir\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.246106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245194 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-var-lib-kubelet\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.246106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245201 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4c08617-40c6-47d5-bcc8-e4765ca7a477-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.246106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245232 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9qdf\" (UniqueName: \"kubernetes.io/projected/bdbecc0a-d250-434e-8478-2f6f57c094d3-kube-api-access-r9qdf\") pod \"node-ca-trs6m\" (UID: \"bdbecc0a-d250-434e-8478-2f6f57c094d3\") " pod="openshift-image-registry/node-ca-trs6m" Sep 17 23:06:04.246106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245269 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4c08617-40c6-47d5-bcc8-e4765ca7a477-system-cni-dir\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.246106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245284 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-systemd-units\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.246106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245542 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-run-ovn\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.246106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245541 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 17 23:06:04.246106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245562 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4518bc1-148d-4005-a3ec-2b36297214fa-konnectivity-ca\") pod \"konnectivity-agent-svq9k\" (UID: \"d4518bc1-148d-4005-a3ec-2b36297214fa\") " pod="kube-system/konnectivity-agent-svq9k" Sep 17 23:06:04.246859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245579 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-host-run-netns\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.246859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245605 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fc91b818-8121-4735-81d4-2da07c40b110-iptables-alerter-script\") pod \"iptables-alerter-d8d5w\" (UID: \"fc91b818-8121-4735-81d4-2da07c40b110\") " pod="openshift-network-operator/iptables-alerter-d8d5w" Sep 17 23:06:04.246859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245587 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bdbecc0a-d250-434e-8478-2f6f57c094d3-serviceca\") pod \"node-ca-trs6m\" (UID: \"bdbecc0a-d250-434e-8478-2f6f57c094d3\") " pod="openshift-image-registry/node-ca-trs6m" Sep 17 23:06:04.246859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245621 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-hostroot\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.246859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245335 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4c08617-40c6-47d5-bcc8-e4765ca7a477-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.246859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245482 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbda2aea-1bb6-4a24-a74f-fc8d79b07586-tmp-dir\") pod \"node-resolver-k2bg2\" (UID: \"bbda2aea-1bb6-4a24-a74f-fc8d79b07586\") " pod="openshift-dns/node-resolver-k2bg2" Sep 17 23:06:04.246859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245655 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-hostroot\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.246859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245658 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-host-run-netns\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.246859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245671 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-host-run-multus-certs\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.246859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245699 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4c08617-40c6-47d5-bcc8-e4765ca7a477-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.246859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245724 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-host-run-multus-certs\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.246859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245733 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-etc-openvswitch\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.246859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245757 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9j6j\" (UniqueName: \"kubernetes.io/projected/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-kube-api-access-l9j6j\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.246859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245792 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-etc-modprobe-d\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.246859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245872 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-run\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.246859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245885 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-etc-modprobe-d\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.246859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.245931 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-run\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.246859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246070 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4518bc1-148d-4005-a3ec-2b36297214fa-konnectivity-ca\") pod \"konnectivity-agent-svq9k\" (UID: \"d4518bc1-148d-4005-a3ec-2b36297214fa\") " pod="kube-system/konnectivity-agent-svq9k" Sep 17 23:06:04.247782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246212 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4c08617-40c6-47d5-bcc8-e4765ca7a477-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.247782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246353 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b75vg\" (UniqueName: \"kubernetes.io/projected/24a660d0-e7cf-402b-8b09-9fb5141b67ba-kube-api-access-b75vg\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.247782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246378 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret\") pod \"global-pull-secret-syncer-drknm\" (UID: \"66bdcd7b-3b69-4bdd-9579-9e035e9d19b0\") " pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:04.247782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246399 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-node-log\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.247782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246421 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.247782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246477 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-multus-daemon-config\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.247782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246549 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdh4v\" (UniqueName: \"kubernetes.io/projected/d4c08617-40c6-47d5-bcc8-e4765ca7a477-kube-api-access-sdh4v\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.247782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246585 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-system-cni-dir\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.247782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246614 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-host-run-k8s-cni-cncf-io\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.247782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246658 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-host-run-k8s-cni-cncf-io\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.247782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246688 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-dbus\") pod \"global-pull-secret-syncer-drknm\" (UID: \"66bdcd7b-3b69-4bdd-9579-9e035e9d19b0\") " pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:04.247782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246716 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-system-cni-dir\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.247782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246758 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/938149bc-e432-42d5-8ff3-0230b863718f-etc-tuned\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.247782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246787 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs\") pod \"network-metrics-daemon-xg2sz\" (UID: \"3752b568-d53f-49ac-9095-0f61cee96767\") " pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:04.247782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246812 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-run-openvswitch\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.247782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246835 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.248435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246858 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-ovn-node-metrics-cert\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.248435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246887 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl4z7\" (UniqueName: \"kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7\") pod \"network-check-target-q8sw2\" (UID: \"62ca42a6-c013-4fce-b108-acefb7dab500\") " pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:04.248435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246910 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-multus-cni-dir\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.248435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246933 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-etc-kubernetes\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.248435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246957 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttp2h\" (UniqueName: \"kubernetes.io/projected/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-kube-api-access-ttp2h\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.248435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.246981 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bdbecc0a-d250-434e-8478-2f6f57c094d3-host\") pod \"node-ca-trs6m\" (UID: \"bdbecc0a-d250-434e-8478-2f6f57c094d3\") " pod="openshift-image-registry/node-ca-trs6m" Sep 17 23:06:04.248435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247000 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-multus-daemon-config\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.248435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247020 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-etc-kubernetes\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.248435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247025 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-multus-cni-dir\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.248435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247002 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-kubelet-config\") pod \"global-pull-secret-syncer-drknm\" (UID: \"66bdcd7b-3b69-4bdd-9579-9e035e9d19b0\") " pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:04.248435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247093 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bdbecc0a-d250-434e-8478-2f6f57c094d3-host\") pod \"node-ca-trs6m\" (UID: \"bdbecc0a-d250-434e-8478-2f6f57c094d3\") " pod="openshift-image-registry/node-ca-trs6m" Sep 17 23:06:04.248435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247139 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-slash\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.248435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247166 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-etc-systemd\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.248435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247191 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-cni-binary-copy\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.248435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247215 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-host-var-lib-cni-bin\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.248435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247234 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/938149bc-e432-42d5-8ff3-0230b863718f-etc-systemd\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.248435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247237 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-var-lib-openvswitch\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.248435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247273 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zktw\" (UniqueName: \"kubernetes.io/projected/bbda2aea-1bb6-4a24-a74f-fc8d79b07586-kube-api-access-8zktw\") pod \"node-resolver-k2bg2\" (UID: \"bbda2aea-1bb6-4a24-a74f-fc8d79b07586\") " pod="openshift-dns/node-resolver-k2bg2" Sep 17 23:06:04.249049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247298 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-host-var-lib-cni-multus\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.249049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247323 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4c08617-40c6-47d5-bcc8-e4765ca7a477-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.249049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247330 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-host-var-lib-cni-multus\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.249049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247298 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-host-var-lib-cni-bin\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.249049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247348 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4c08617-40c6-47d5-bcc8-e4765ca7a477-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.249049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247373 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/24a660d0-e7cf-402b-8b09-9fb5141b67ba-sys-fs\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.249049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247397 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvgbr\" (UniqueName: \"kubernetes.io/projected/3752b568-d53f-49ac-9095-0f61cee96767-kube-api-access-tvgbr\") pod \"network-metrics-daemon-xg2sz\" (UID: \"3752b568-d53f-49ac-9095-0f61cee96767\") " pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:04.249049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247419 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4518bc1-148d-4005-a3ec-2b36297214fa-agent-certs\") pod \"konnectivity-agent-svq9k\" (UID: \"d4518bc1-148d-4005-a3ec-2b36297214fa\") " pod="kube-system/konnectivity-agent-svq9k" Sep 17 23:06:04.249049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247443 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65tvm\" (UniqueName: \"kubernetes.io/projected/938149bc-e432-42d5-8ff3-0230b863718f-kube-api-access-65tvm\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.249049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247690 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-cni-binary-copy\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.249049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247897 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4c08617-40c6-47d5-bcc8-e4765ca7a477-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.249049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.247982 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4c08617-40c6-47d5-bcc8-e4765ca7a477-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.249444 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.249227 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/938149bc-e432-42d5-8ff3-0230b863718f-etc-tuned\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.249444 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.249239 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/938149bc-e432-42d5-8ff3-0230b863718f-tmp\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.250693 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.250647 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4518bc1-148d-4005-a3ec-2b36297214fa-agent-certs\") pod \"konnectivity-agent-svq9k\" (UID: \"d4518bc1-148d-4005-a3ec-2b36297214fa\") " pod="kube-system/konnectivity-agent-svq9k" Sep 17 23:06:04.251913 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.251894 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg7df\" (UniqueName: \"kubernetes.io/projected/fc91b818-8121-4735-81d4-2da07c40b110-kube-api-access-hg7df\") pod \"iptables-alerter-d8d5w\" (UID: \"fc91b818-8121-4735-81d4-2da07c40b110\") " pod="openshift-network-operator/iptables-alerter-d8d5w" Sep 17 23:06:04.252335 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.252320 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9qdf\" (UniqueName: \"kubernetes.io/projected/bdbecc0a-d250-434e-8478-2f6f57c094d3-kube-api-access-r9qdf\") pod \"node-ca-trs6m\" (UID: \"bdbecc0a-d250-434e-8478-2f6f57c094d3\") " pod="openshift-image-registry/node-ca-trs6m" Sep 17 23:06:04.256367 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.256353 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 23:06:04.256440 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.256370 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 23:06:04.256440 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.256390 2582 projected.go:196] Error preparing data for projected volume kube-api-access-gl4z7 for pod openshift-network-diagnostics/network-check-target-q8sw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:04.256565 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.256446 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7 podName:62ca42a6-c013-4fce-b108-acefb7dab500 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:04.756422655 +0000 UTC m=+3.690329768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gl4z7" (UniqueName: "kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7") pod "network-check-target-q8sw2" (UID: "62ca42a6-c013-4fce-b108-acefb7dab500") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:04.258959 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.258901 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdh4v\" (UniqueName: \"kubernetes.io/projected/d4c08617-40c6-47d5-bcc8-e4765ca7a477-kube-api-access-sdh4v\") pod \"multus-additional-cni-plugins-jqrzv\" (UID: \"d4c08617-40c6-47d5-bcc8-e4765ca7a477\") " pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.260747 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.258992 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttp2h\" (UniqueName: \"kubernetes.io/projected/dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6-kube-api-access-ttp2h\") pod \"multus-zzcb8\" (UID: \"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6\") " pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.261206 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.261178 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zktw\" (UniqueName: \"kubernetes.io/projected/bbda2aea-1bb6-4a24-a74f-fc8d79b07586-kube-api-access-8zktw\") pod \"node-resolver-k2bg2\" (UID: \"bbda2aea-1bb6-4a24-a74f-fc8d79b07586\") " pod="openshift-dns/node-resolver-k2bg2" Sep 17 23:06:04.262294 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.262276 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65tvm\" (UniqueName: \"kubernetes.io/projected/938149bc-e432-42d5-8ff3-0230b863718f-kube-api-access-65tvm\") pod \"tuned-5s5pl\" (UID: \"938149bc-e432-42d5-8ff3-0230b863718f\") " pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.348679 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348647 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/24a660d0-e7cf-402b-8b09-9fb5141b67ba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.348679 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348678 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-cni-bin\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.348909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348694 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-ovnkube-config\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.348909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348709 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/24a660d0-e7cf-402b-8b09-9fb5141b67ba-etc-selinux\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.348909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348749 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-run-systemd\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.348909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348792 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-cni-bin\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.348909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348794 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/24a660d0-e7cf-402b-8b09-9fb5141b67ba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.348909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348822 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-cni-netd\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.348909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348817 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/24a660d0-e7cf-402b-8b09-9fb5141b67ba-etc-selinux\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.348909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348845 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-ovnkube-script-lib\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.348909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348847 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-run-systemd\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.348909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348865 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-cni-netd\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.348909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348870 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-kubelet\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.348909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348902 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-run-netns\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.348909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348906 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-kubelet\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.349481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348928 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/24a660d0-e7cf-402b-8b09-9fb5141b67ba-device-dir\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.349481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348955 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/24a660d0-e7cf-402b-8b09-9fb5141b67ba-registration-dir\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.349481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348983 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-systemd-units\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.349481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348957 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-run-netns\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.349481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349007 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-run-ovn\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.349481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349009 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/24a660d0-e7cf-402b-8b09-9fb5141b67ba-registration-dir\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.349481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349032 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-systemd-units\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.349481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.348996 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/24a660d0-e7cf-402b-8b09-9fb5141b67ba-device-dir\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.349481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349047 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-run-ovn\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.349481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349058 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-etc-openvswitch\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.349481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349081 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9j6j\" (UniqueName: \"kubernetes.io/projected/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-kube-api-access-l9j6j\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.349481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349107 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b75vg\" (UniqueName: \"kubernetes.io/projected/24a660d0-e7cf-402b-8b09-9fb5141b67ba-kube-api-access-b75vg\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.349481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349127 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-etc-openvswitch\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.349481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349130 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret\") pod \"global-pull-secret-syncer-drknm\" (UID: \"66bdcd7b-3b69-4bdd-9579-9e035e9d19b0\") " pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:04.349481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349194 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-node-log\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.349481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349219 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.349481 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.349263 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:04.350236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349274 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-dbus\") pod \"global-pull-secret-syncer-drknm\" (UID: \"66bdcd7b-3b69-4bdd-9579-9e035e9d19b0\") " pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:04.350236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349271 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.350236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349298 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs\") pod \"network-metrics-daemon-xg2sz\" (UID: \"3752b568-d53f-49ac-9095-0f61cee96767\") " pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:04.350236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349323 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-run-openvswitch\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.350236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349331 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-node-log\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.350236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349349 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.350236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349374 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-ovnkube-script-lib\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.350236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349377 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-ovn-node-metrics-cert\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.350236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349382 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-run-openvswitch\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.350236 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.349403 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret podName:66bdcd7b-3b69-4bdd-9579-9e035e9d19b0 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:04.849384742 +0000 UTC m=+3.783291865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret") pod "global-pull-secret-syncer-drknm" (UID: "66bdcd7b-3b69-4bdd-9579-9e035e9d19b0") : object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:04.350236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349418 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-dbus\") pod \"global-pull-secret-syncer-drknm\" (UID: \"66bdcd7b-3b69-4bdd-9579-9e035e9d19b0\") " pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:04.350236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349436 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.350236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349452 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-kubelet-config\") pod \"global-pull-secret-syncer-drknm\" (UID: \"66bdcd7b-3b69-4bdd-9579-9e035e9d19b0\") " pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:04.350236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349471 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-slash\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.350236 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.349455 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:04.350236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349518 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-kubelet-config\") pod \"global-pull-secret-syncer-drknm\" (UID: \"66bdcd7b-3b69-4bdd-9579-9e035e9d19b0\") " pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:04.350236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349535 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-var-lib-openvswitch\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.350726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349544 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-host-slash\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.350726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349491 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-var-lib-openvswitch\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.350726 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.349572 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs podName:3752b568-d53f-49ac-9095-0f61cee96767 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:04.84955444 +0000 UTC m=+3.783461570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs") pod "network-metrics-daemon-xg2sz" (UID: "3752b568-d53f-49ac-9095-0f61cee96767") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:04.350726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349650 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/24a660d0-e7cf-402b-8b09-9fb5141b67ba-sys-fs\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.350726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349677 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvgbr\" (UniqueName: \"kubernetes.io/projected/3752b568-d53f-49ac-9095-0f61cee96767-kube-api-access-tvgbr\") pod \"network-metrics-daemon-xg2sz\" (UID: \"3752b568-d53f-49ac-9095-0f61cee96767\") " pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:04.350726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349705 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/24a660d0-e7cf-402b-8b09-9fb5141b67ba-socket-dir\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.350726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349729 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-log-socket\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.350726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349743 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/24a660d0-e7cf-402b-8b09-9fb5141b67ba-sys-fs\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.350726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349754 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-env-overrides\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.350726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349786 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-ovnkube-config\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.350726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349833 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-log-socket\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.350726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.349881 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/24a660d0-e7cf-402b-8b09-9fb5141b67ba-socket-dir\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.350726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.350126 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-env-overrides\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.351623 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.351600 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-ovn-node-metrics-cert\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.363946 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.363923 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b75vg\" (UniqueName: \"kubernetes.io/projected/24a660d0-e7cf-402b-8b09-9fb5141b67ba-kube-api-access-b75vg\") pod \"aws-ebs-csi-driver-node-jkp5t\" (UID: \"24a660d0-e7cf-402b-8b09-9fb5141b67ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.364044 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.363955 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9j6j\" (UniqueName: \"kubernetes.io/projected/c6f43b5a-9f32-4789-88f4-1f81e8d764c9-kube-api-access-l9j6j\") pod \"ovnkube-node-js95g\" (UID: \"c6f43b5a-9f32-4789-88f4-1f81e8d764c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.364044 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.363997 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvgbr\" (UniqueName: \"kubernetes.io/projected/3752b568-d53f-49ac-9095-0f61cee96767-kube-api-access-tvgbr\") pod \"network-metrics-daemon-xg2sz\" (UID: \"3752b568-d53f-49ac-9095-0f61cee96767\") " pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:04.411919 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:04.411879 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c42993043d7f11ec0cbc6c6de37a429.slice/crio-3edc68b8f70876bad28b6e56fbc007775b85249f84154901c412fe1df0547d3b WatchSource:0}: Error finding container 3edc68b8f70876bad28b6e56fbc007775b85249f84154901c412fe1df0547d3b: Status 404 returned error can't find the container with id 3edc68b8f70876bad28b6e56fbc007775b85249f84154901c412fe1df0547d3b Sep 17 23:06:04.412466 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:04.412445 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0067a4916205e46911412e53db8082.slice/crio-45bfd2496d0f1a3a634751c1280588855a1ef6ee97cfdbd72e5acb77e2ad303a WatchSource:0}: Error finding container 45bfd2496d0f1a3a634751c1280588855a1ef6ee97cfdbd72e5acb77e2ad303a: Status 404 returned error can't find the container with id 45bfd2496d0f1a3a634751c1280588855a1ef6ee97cfdbd72e5acb77e2ad303a Sep 17 23:06:04.417198 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.417145 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 23:06:04.461116 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.461014 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-svq9k" Sep 17 23:06:04.469160 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:04.469128 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4518bc1_148d_4005_a3ec_2b36297214fa.slice/crio-6d3a4e3a99ca327b4b6095499d4f23a6bf96977794316793c5783d935dd4f2c9 WatchSource:0}: Error finding container 6d3a4e3a99ca327b4b6095499d4f23a6bf96977794316793c5783d935dd4f2c9: Status 404 returned error can't find the container with id 6d3a4e3a99ca327b4b6095499d4f23a6bf96977794316793c5783d935dd4f2c9 Sep 17 23:06:04.473035 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.473011 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d8d5w" Sep 17 23:06:04.479582 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.479558 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" Sep 17 23:06:04.479892 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:04.479875 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc91b818_8121_4735_81d4_2da07c40b110.slice/crio-3cf7cbf0ef5fdd6c337356f0746e1b235c624bd77d39f322b3cd1ecd61157e9b WatchSource:0}: Error finding container 3cf7cbf0ef5fdd6c337356f0746e1b235c624bd77d39f322b3cd1ecd61157e9b: Status 404 returned error can't find the container with id 3cf7cbf0ef5fdd6c337356f0746e1b235c624bd77d39f322b3cd1ecd61157e9b Sep 17 23:06:04.486247 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:04.486228 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938149bc_e432_42d5_8ff3_0230b863718f.slice/crio-968b80f8fc9b1af9d7a5c20e68d014da3a77bde15ffd72522dfa52524843993e WatchSource:0}: Error finding container 968b80f8fc9b1af9d7a5c20e68d014da3a77bde15ffd72522dfa52524843993e: Status 404 returned error can't find the container with id 968b80f8fc9b1af9d7a5c20e68d014da3a77bde15ffd72522dfa52524843993e Sep 17 23:06:04.501839 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.501818 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k2bg2" Sep 17 23:06:04.508712 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:04.508692 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbda2aea_1bb6_4a24_a74f_fc8d79b07586.slice/crio-049a3288b9873ef387931818c429b83f069f2ab32284fb56c483645a7f917dfb WatchSource:0}: Error finding container 049a3288b9873ef387931818c429b83f069f2ab32284fb56c483645a7f917dfb: Status 404 returned error can't find the container with id 049a3288b9873ef387931818c429b83f069f2ab32284fb56c483645a7f917dfb Sep 17 23:06:04.520864 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.520846 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-trs6m" Sep 17 23:06:04.525932 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.525915 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zzcb8" Sep 17 23:06:04.527916 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:04.527895 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbecc0a_d250_434e_8478_2f6f57c094d3.slice/crio-397290944fe1d3268beb9dd94e4e81de8c6bd9df8a0524c21798ab663419275e WatchSource:0}: Error finding container 397290944fe1d3268beb9dd94e4e81de8c6bd9df8a0524c21798ab663419275e: Status 404 returned error can't find the container with id 397290944fe1d3268beb9dd94e4e81de8c6bd9df8a0524c21798ab663419275e Sep 17 23:06:04.532239 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:04.532218 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5f6c2c_5d7d_4863_aa8d_ab6c172114f6.slice/crio-b8ee274282f42f13a9594fc7779a94e05c0330a764c180be5471eeddbf7fbe13 WatchSource:0}: Error finding container b8ee274282f42f13a9594fc7779a94e05c0330a764c180be5471eeddbf7fbe13: Status 404 returned error can't find the container with id b8ee274282f42f13a9594fc7779a94e05c0330a764c180be5471eeddbf7fbe13 Sep 17 23:06:04.551024 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.551005 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jqrzv" Sep 17 23:06:04.558289 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:04.558269 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c08617_40c6_47d5_bcc8_e4765ca7a477.slice/crio-d57fd79195d1cb9b330d16a34f20cf0cb0725a1637563e17bd0cef30e1e6ead1 WatchSource:0}: Error finding container d57fd79195d1cb9b330d16a34f20cf0cb0725a1637563e17bd0cef30e1e6ead1: Status 404 returned error can't find the container with id d57fd79195d1cb9b330d16a34f20cf0cb0725a1637563e17bd0cef30e1e6ead1 Sep 17 23:06:04.586268 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.586241 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:04.591509 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.591389 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" Sep 17 23:06:04.594340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:04.594319 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f43b5a_9f32_4789_88f4_1f81e8d764c9.slice/crio-0b43123fc892a14b3e16704b63f52bb98b8609625de672fc2133bc0815cae292 WatchSource:0}: Error finding container 0b43123fc892a14b3e16704b63f52bb98b8609625de672fc2133bc0815cae292: Status 404 returned error can't find the container with id 0b43123fc892a14b3e16704b63f52bb98b8609625de672fc2133bc0815cae292 Sep 17 23:06:04.598908 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:04.598888 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a660d0_e7cf_402b_8b09_9fb5141b67ba.slice/crio-7bf86f12e37925c45d3a8604a4ee61f2dc1e48a9192d8b0b94258a725ef3d13a WatchSource:0}: Error finding container 7bf86f12e37925c45d3a8604a4ee61f2dc1e48a9192d8b0b94258a725ef3d13a: Status 404 returned error can't find the container with id 7bf86f12e37925c45d3a8604a4ee61f2dc1e48a9192d8b0b94258a725ef3d13a Sep 17 23:06:04.600406 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.600389 2582 kuberuntime_manager.go:1459] "Unhandled Error" err="container csi-driver start failed in pod aws-ebs-csi-driver-node-jkp5t_openshift-cluster-csi-drivers(24a660d0-e7cf-402b-8b09-9fb5141b67ba): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Sep 17 23:06:04.601137 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.601115 2582 kuberuntime_manager.go:1459] "Unhandled Error" err="container csi-node-driver-registrar start failed in pod aws-ebs-csi-driver-node-jkp5t_openshift-cluster-csi-drivers(24a660d0-e7cf-402b-8b09-9fb5141b67ba): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Sep 17 23:06:04.602825 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.602809 2582 kuberuntime_manager.go:1459] "Unhandled Error" err="container csi-liveness-probe start failed in pod aws-ebs-csi-driver-node-jkp5t_openshift-cluster-csi-drivers(24a660d0-e7cf-402b-8b09-9fb5141b67ba): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Sep 17 23:06:04.602868 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.602851 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-driver\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"csi-liveness-probe\" with ErrImagePull: \"pull QPS exceeded\"]" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" podUID="24a660d0-e7cf-402b-8b09-9fb5141b67ba" Sep 17 23:06:04.712736 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.712656 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 23:06:04.853981 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.853949 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret\") pod \"global-pull-secret-syncer-drknm\" (UID: \"66bdcd7b-3b69-4bdd-9579-9e035e9d19b0\") " pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:04.854139 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.854000 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs\") pod \"network-metrics-daemon-xg2sz\" (UID: \"3752b568-d53f-49ac-9095-0f61cee96767\") " pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:04.854139 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.854023 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl4z7\" (UniqueName: \"kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7\") pod \"network-check-target-q8sw2\" (UID: \"62ca42a6-c013-4fce-b108-acefb7dab500\") " pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:04.854139 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.854134 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 23:06:04.854305 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.854160 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 23:06:04.854305 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.854159 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:04.854305 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.854133 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:04.854305 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.854174 2582 projected.go:196] Error preparing data for projected volume kube-api-access-gl4z7 for pod openshift-network-diagnostics/network-check-target-q8sw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:04.854305 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.854227 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret podName:66bdcd7b-3b69-4bdd-9579-9e035e9d19b0 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:05.854213088 +0000 UTC m=+4.788120201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret") pod "global-pull-secret-syncer-drknm" (UID: "66bdcd7b-3b69-4bdd-9579-9e035e9d19b0") : object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:04.854305 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.854241 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs podName:3752b568-d53f-49ac-9095-0f61cee96767 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:05.854234982 +0000 UTC m=+4.788142095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs") pod "network-metrics-daemon-xg2sz" (UID: "3752b568-d53f-49ac-9095-0f61cee96767") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:04.854305 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:04.854252 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7 podName:62ca42a6-c013-4fce-b108-acefb7dab500 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:05.854246359 +0000 UTC m=+4.788153472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gl4z7" (UniqueName: "kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7") pod "network-check-target-q8sw2" (UID: "62ca42a6-c013-4fce-b108-acefb7dab500") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:04.964782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:04.964710 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 23:06:05.180970 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:05.180873 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:01:04 +0000 UTC" deadline="2028-06-14 01:24:53.364750464 +0000 UTC" Sep 17 23:06:05.180970 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:05.180925 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15242h18m48.18382892s" Sep 17 23:06:05.309515 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:05.308574 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" event={"ID":"24a660d0-e7cf-402b-8b09-9fb5141b67ba","Type":"ContainerStarted","Data":"7bf86f12e37925c45d3a8604a4ee61f2dc1e48a9192d8b0b94258a725ef3d13a"} Sep 17 23:06:05.330042 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:05.329936 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-driver\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3d3869c161a073dfbaf036364ff0996f20e4d9ee240af9f0daa929f78c7ea46e\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:54fc81b7047fca86ea91318cae087e3f654cc8d54725953ea6f0ddfcaf515545\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"csi-liveness-probe\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c43eb8f4e3024c90d819ecdc6c4101d91bd5c368d5b50945756ddf6fa4b3b9cf\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" podUID="24a660d0-e7cf-402b-8b09-9fb5141b67ba" Sep 17 23:06:05.336971 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:05.336938 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js95g" event={"ID":"c6f43b5a-9f32-4789-88f4-1f81e8d764c9","Type":"ContainerStarted","Data":"0b43123fc892a14b3e16704b63f52bb98b8609625de672fc2133bc0815cae292"} Sep 17 23:06:05.341847 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:05.341733 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqrzv" event={"ID":"d4c08617-40c6-47d5-bcc8-e4765ca7a477","Type":"ContainerStarted","Data":"d57fd79195d1cb9b330d16a34f20cf0cb0725a1637563e17bd0cef30e1e6ead1"} Sep 17 23:06:05.352412 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:05.352385 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k2bg2" event={"ID":"bbda2aea-1bb6-4a24-a74f-fc8d79b07586","Type":"ContainerStarted","Data":"049a3288b9873ef387931818c429b83f069f2ab32284fb56c483645a7f917dfb"} Sep 17 23:06:05.377062 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:05.377032 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" event={"ID":"938149bc-e432-42d5-8ff3-0230b863718f","Type":"ContainerStarted","Data":"968b80f8fc9b1af9d7a5c20e68d014da3a77bde15ffd72522dfa52524843993e"} Sep 17 23:06:05.398785 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:05.398754 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-svq9k" event={"ID":"d4518bc1-148d-4005-a3ec-2b36297214fa","Type":"ContainerStarted","Data":"6d3a4e3a99ca327b4b6095499d4f23a6bf96977794316793c5783d935dd4f2c9"} Sep 17 23:06:05.414336 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:05.414303 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-82.ec2.internal" event={"ID":"9b0067a4916205e46911412e53db8082","Type":"ContainerStarted","Data":"45bfd2496d0f1a3a634751c1280588855a1ef6ee97cfdbd72e5acb77e2ad303a"} Sep 17 23:06:05.428454 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:05.428424 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zzcb8" event={"ID":"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6","Type":"ContainerStarted","Data":"b8ee274282f42f13a9594fc7779a94e05c0330a764c180be5471eeddbf7fbe13"} Sep 17 23:06:05.446057 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:05.446022 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-trs6m" event={"ID":"bdbecc0a-d250-434e-8478-2f6f57c094d3","Type":"ContainerStarted","Data":"397290944fe1d3268beb9dd94e4e81de8c6bd9df8a0524c21798ab663419275e"} Sep 17 23:06:05.481131 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:05.481096 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d8d5w" event={"ID":"fc91b818-8121-4735-81d4-2da07c40b110","Type":"ContainerStarted","Data":"3cf7cbf0ef5fdd6c337356f0746e1b235c624bd77d39f322b3cd1ecd61157e9b"} Sep 17 23:06:05.502153 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:05.502122 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal" event={"ID":"7c42993043d7f11ec0cbc6c6de37a429","Type":"ContainerStarted","Data":"3edc68b8f70876bad28b6e56fbc007775b85249f84154901c412fe1df0547d3b"} Sep 17 23:06:05.860880 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:05.860612 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret\") pod \"global-pull-secret-syncer-drknm\" (UID: \"66bdcd7b-3b69-4bdd-9579-9e035e9d19b0\") " pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:05.860880 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:05.860667 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs\") pod \"network-metrics-daemon-xg2sz\" (UID: \"3752b568-d53f-49ac-9095-0f61cee96767\") " pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:05.860880 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:05.860696 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl4z7\" (UniqueName: \"kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7\") pod \"network-check-target-q8sw2\" (UID: \"62ca42a6-c013-4fce-b108-acefb7dab500\") " pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:05.860880 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:05.860803 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:05.860880 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:05.860843 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 23:06:05.860880 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:05.860860 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 23:06:05.860880 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:05.860879 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret podName:66bdcd7b-3b69-4bdd-9579-9e035e9d19b0 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:07.860858259 +0000 UTC m=+6.794765382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret") pod "global-pull-secret-syncer-drknm" (UID: "66bdcd7b-3b69-4bdd-9579-9e035e9d19b0") : object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:05.860880 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:05.860882 2582 projected.go:196] Error preparing data for projected volume kube-api-access-gl4z7 for pod openshift-network-diagnostics/network-check-target-q8sw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:05.861410 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:05.860933 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7 podName:62ca42a6-c013-4fce-b108-acefb7dab500 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:07.860918293 +0000 UTC m=+6.794825410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gl4z7" (UniqueName: "kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7") pod "network-check-target-q8sw2" (UID: "62ca42a6-c013-4fce-b108-acefb7dab500") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:05.861410 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:05.860806 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:05.861410 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:05.860969 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs podName:3752b568-d53f-49ac-9095-0f61cee96767 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:07.860961285 +0000 UTC m=+6.794868398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs") pod "network-metrics-daemon-xg2sz" (UID: "3752b568-d53f-49ac-9095-0f61cee96767") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:06.181432 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:06.181324 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:01:04 +0000 UTC" deadline="2028-06-02 10:22:56.046502528 +0000 UTC" Sep 17 23:06:06.181432 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:06.181362 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14963h16m49.865144624s" Sep 17 23:06:06.287587 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:06.287425 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:06.287587 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:06.287556 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8sw2" podUID="62ca42a6-c013-4fce-b108-acefb7dab500" Sep 17 23:06:06.287953 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:06.287934 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:06.288059 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:06.288030 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drknm" podUID="66bdcd7b-3b69-4bdd-9579-9e035e9d19b0" Sep 17 23:06:06.288116 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:06.288102 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:06.288378 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:06.288183 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2sz" podUID="3752b568-d53f-49ac-9095-0f61cee96767" Sep 17 23:06:06.946032 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:06.946005 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 23:06:07.084878 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:07.084678 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 23:06:07.879317 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:07.878412 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret\") pod \"global-pull-secret-syncer-drknm\" (UID: \"66bdcd7b-3b69-4bdd-9579-9e035e9d19b0\") " pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:07.879317 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:07.878466 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs\") pod \"network-metrics-daemon-xg2sz\" (UID: \"3752b568-d53f-49ac-9095-0f61cee96767\") " pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:07.879317 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:07.878509 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl4z7\" (UniqueName: \"kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7\") pod \"network-check-target-q8sw2\" (UID: \"62ca42a6-c013-4fce-b108-acefb7dab500\") " pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:07.879317 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:07.878672 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 23:06:07.879317 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:07.878692 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 23:06:07.879317 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:07.878706 2582 projected.go:196] Error preparing data for projected volume kube-api-access-gl4z7 for pod openshift-network-diagnostics/network-check-target-q8sw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:07.879317 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:07.878758 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7 podName:62ca42a6-c013-4fce-b108-acefb7dab500 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:11.878741705 +0000 UTC m=+10.812648823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gl4z7" (UniqueName: "kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7") pod "network-check-target-q8sw2" (UID: "62ca42a6-c013-4fce-b108-acefb7dab500") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:07.879317 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:07.879132 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:07.879317 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:07.879181 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret podName:66bdcd7b-3b69-4bdd-9579-9e035e9d19b0 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:11.87916708 +0000 UTC m=+10.813074200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret") pod "global-pull-secret-syncer-drknm" (UID: "66bdcd7b-3b69-4bdd-9579-9e035e9d19b0") : object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:07.879317 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:07.879253 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:07.879317 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:07.879285 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs podName:3752b568-d53f-49ac-9095-0f61cee96767 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:11.879274367 +0000 UTC m=+10.813181487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs") pod "network-metrics-daemon-xg2sz" (UID: "3752b568-d53f-49ac-9095-0f61cee96767") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:08.288182 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:08.287704 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:08.288182 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:08.287820 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8sw2" podUID="62ca42a6-c013-4fce-b108-acefb7dab500" Sep 17 23:06:08.288182 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:08.287886 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:08.288182 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:08.288056 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drknm" podUID="66bdcd7b-3b69-4bdd-9579-9e035e9d19b0" Sep 17 23:06:08.288182 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:08.288135 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:08.288617 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:08.288227 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2sz" podUID="3752b568-d53f-49ac-9095-0f61cee96767" Sep 17 23:06:10.287340 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:10.287293 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:10.287340 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:10.287324 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:10.287880 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:10.287329 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:10.287880 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:10.287426 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8sw2" podUID="62ca42a6-c013-4fce-b108-acefb7dab500" Sep 17 23:06:10.287880 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:10.287525 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drknm" podUID="66bdcd7b-3b69-4bdd-9579-9e035e9d19b0" Sep 17 23:06:10.287880 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:10.287619 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2sz" podUID="3752b568-d53f-49ac-9095-0f61cee96767" Sep 17 23:06:11.912313 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:11.912268 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl4z7\" (UniqueName: \"kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7\") pod \"network-check-target-q8sw2\" (UID: \"62ca42a6-c013-4fce-b108-acefb7dab500\") " pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:11.912776 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:11.912363 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret\") pod \"global-pull-secret-syncer-drknm\" (UID: \"66bdcd7b-3b69-4bdd-9579-9e035e9d19b0\") " pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:11.912776 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:11.912392 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs\") pod \"network-metrics-daemon-xg2sz\" (UID: \"3752b568-d53f-49ac-9095-0f61cee96767\") " pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:11.912776 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:11.912421 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 23:06:11.912776 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:11.912437 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 23:06:11.912776 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:11.912450 2582 projected.go:196] Error preparing data for projected volume kube-api-access-gl4z7 for pod openshift-network-diagnostics/network-check-target-q8sw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:11.912776 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:11.912471 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:11.912776 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:11.912528 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7 podName:62ca42a6-c013-4fce-b108-acefb7dab500 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:19.912511068 +0000 UTC m=+18.846418195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gl4z7" (UniqueName: "kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7") pod "network-check-target-q8sw2" (UID: "62ca42a6-c013-4fce-b108-acefb7dab500") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:11.912776 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:11.912532 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:11.912776 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:11.912548 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs podName:3752b568-d53f-49ac-9095-0f61cee96767 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:19.912538633 +0000 UTC m=+18.846445750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs") pod "network-metrics-daemon-xg2sz" (UID: "3752b568-d53f-49ac-9095-0f61cee96767") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:11.912776 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:11.912586 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret podName:66bdcd7b-3b69-4bdd-9579-9e035e9d19b0 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:19.912574336 +0000 UTC m=+18.846481454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret") pod "global-pull-secret-syncer-drknm" (UID: "66bdcd7b-3b69-4bdd-9579-9e035e9d19b0") : object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:12.288047 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:12.287303 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:12.288047 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:12.287443 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2sz" podUID="3752b568-d53f-49ac-9095-0f61cee96767" Sep 17 23:06:12.288047 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:12.287812 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:12.288047 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:12.287896 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8sw2" podUID="62ca42a6-c013-4fce-b108-acefb7dab500" Sep 17 23:06:12.288047 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:12.287937 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:12.288047 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:12.288006 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drknm" podUID="66bdcd7b-3b69-4bdd-9579-9e035e9d19b0" Sep 17 23:06:14.287903 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:14.287857 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:14.288402 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:14.287870 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:14.288402 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:14.287981 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:14.288402 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:14.287987 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2sz" podUID="3752b568-d53f-49ac-9095-0f61cee96767" Sep 17 23:06:14.288402 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:14.288044 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8sw2" podUID="62ca42a6-c013-4fce-b108-acefb7dab500" Sep 17 23:06:14.288402 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:14.288132 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drknm" podUID="66bdcd7b-3b69-4bdd-9579-9e035e9d19b0" Sep 17 23:06:16.287453 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:16.287417 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:16.287453 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:16.287436 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:16.288096 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:16.287438 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:16.288096 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:16.287563 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2sz" podUID="3752b568-d53f-49ac-9095-0f61cee96767" Sep 17 23:06:16.288096 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:16.287672 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8sw2" podUID="62ca42a6-c013-4fce-b108-acefb7dab500" Sep 17 23:06:16.288096 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:16.287750 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drknm" podUID="66bdcd7b-3b69-4bdd-9579-9e035e9d19b0" Sep 17 23:06:18.288007 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:18.287966 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:18.288448 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:18.287969 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:18.288448 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:18.288094 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drknm" podUID="66bdcd7b-3b69-4bdd-9579-9e035e9d19b0" Sep 17 23:06:18.288448 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:18.288184 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8sw2" podUID="62ca42a6-c013-4fce-b108-acefb7dab500" Sep 17 23:06:18.288448 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:18.287969 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:18.288448 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:18.288296 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2sz" podUID="3752b568-d53f-49ac-9095-0f61cee96767" Sep 17 23:06:19.970655 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:19.970620 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret\") pod \"global-pull-secret-syncer-drknm\" (UID: \"66bdcd7b-3b69-4bdd-9579-9e035e9d19b0\") " pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:19.971085 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:19.970671 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs\") pod \"network-metrics-daemon-xg2sz\" (UID: \"3752b568-d53f-49ac-9095-0f61cee96767\") " pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:19.971085 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:19.970772 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:19.971085 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:19.970798 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl4z7\" (UniqueName: \"kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7\") pod \"network-check-target-q8sw2\" (UID: \"62ca42a6-c013-4fce-b108-acefb7dab500\") " pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:19.971085 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:19.970830 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret podName:66bdcd7b-3b69-4bdd-9579-9e035e9d19b0 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:35.970815976 +0000 UTC m=+34.904723090 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret") pod "global-pull-secret-syncer-drknm" (UID: "66bdcd7b-3b69-4bdd-9579-9e035e9d19b0") : object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:19.971085 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:19.970773 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:19.971085 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:19.970901 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs podName:3752b568-d53f-49ac-9095-0f61cee96767 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:35.970887881 +0000 UTC m=+34.904794995 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs") pod "network-metrics-daemon-xg2sz" (UID: "3752b568-d53f-49ac-9095-0f61cee96767") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:19.971085 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:19.970952 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 23:06:19.971085 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:19.970967 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 23:06:19.971085 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:19.970980 2582 projected.go:196] Error preparing data for projected volume kube-api-access-gl4z7 for pod openshift-network-diagnostics/network-check-target-q8sw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:19.971085 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:19.971018 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7 podName:62ca42a6-c013-4fce-b108-acefb7dab500 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:35.971007327 +0000 UTC m=+34.904914466 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gl4z7" (UniqueName: "kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7") pod "network-check-target-q8sw2" (UID: "62ca42a6-c013-4fce-b108-acefb7dab500") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:20.287049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:20.286966 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:20.287049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:20.286989 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:20.287334 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:20.287075 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:20.287334 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:20.287084 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8sw2" podUID="62ca42a6-c013-4fce-b108-acefb7dab500" Sep 17 23:06:20.287334 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:20.287202 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drknm" podUID="66bdcd7b-3b69-4bdd-9579-9e035e9d19b0" Sep 17 23:06:20.287334 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:20.287295 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2sz" podUID="3752b568-d53f-49ac-9095-0f61cee96767" Sep 17 23:06:22.287896 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.287864 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:22.288276 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:22.287993 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8sw2" podUID="62ca42a6-c013-4fce-b108-acefb7dab500" Sep 17 23:06:22.288276 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.288067 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:22.288276 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:22.288136 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drknm" podUID="66bdcd7b-3b69-4bdd-9579-9e035e9d19b0" Sep 17 23:06:22.288276 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.288186 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:22.288276 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:22.288260 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2sz" podUID="3752b568-d53f-49ac-9095-0f61cee96767" Sep 17 23:06:22.540758 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.540539 2582 generic.go:356] "Generic (PLEG): container finished" podID="7c42993043d7f11ec0cbc6c6de37a429" containerID="73e01cfd99f32aabd00b20416c9a4e4084a744eabe2ac3cc8aaeaeec50e55d5f" exitCode=0 Sep 17 23:06:22.540882 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.540592 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal" event={"ID":"7c42993043d7f11ec0cbc6c6de37a429","Type":"ContainerDied","Data":"73e01cfd99f32aabd00b20416c9a4e4084a744eabe2ac3cc8aaeaeec50e55d5f"} Sep 17 23:06:22.546641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.546615 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js95g" event={"ID":"c6f43b5a-9f32-4789-88f4-1f81e8d764c9","Type":"ContainerStarted","Data":"7ebf029c63994641642305de4092fb311f011e531a0c3ca644ccc33f9f4738c4"} Sep 17 23:06:22.546756 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.546681 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js95g" event={"ID":"c6f43b5a-9f32-4789-88f4-1f81e8d764c9","Type":"ContainerStarted","Data":"2ef62d7d961f3ecae351818d004ac9d1c88c9da2ebfd3b1cde063863a29de938"} Sep 17 23:06:22.546756 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.546717 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js95g" event={"ID":"c6f43b5a-9f32-4789-88f4-1f81e8d764c9","Type":"ContainerStarted","Data":"1d7e234b551e21e908caf97c993960f738e5f32e2cfe330b2da8fd3773d53e15"} Sep 17 23:06:22.546862 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.546806 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js95g" event={"ID":"c6f43b5a-9f32-4789-88f4-1f81e8d764c9","Type":"ContainerStarted","Data":"a7678efa90f0719110ea1ff9f1ba7bc2a56dbad23c5e21adc74131cce3d9d0f4"} Sep 17 23:06:22.548457 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.548430 2582 generic.go:356] "Generic (PLEG): container finished" podID="d4c08617-40c6-47d5-bcc8-e4765ca7a477" containerID="1333ce9b8b4a1c2e2a3926c74426b3b28a13006f3e903b6354326c59513742a2" exitCode=0 Sep 17 23:06:22.548578 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.548536 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqrzv" event={"ID":"d4c08617-40c6-47d5-bcc8-e4765ca7a477","Type":"ContainerDied","Data":"1333ce9b8b4a1c2e2a3926c74426b3b28a13006f3e903b6354326c59513742a2"} Sep 17 23:06:22.550660 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.550279 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k2bg2" event={"ID":"bbda2aea-1bb6-4a24-a74f-fc8d79b07586","Type":"ContainerStarted","Data":"a366edb1712cb9898e7fbbe71e3a45eadd513f776c46bea7a37b46cf0445a8b2"} Sep 17 23:06:22.552115 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.552095 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" event={"ID":"938149bc-e432-42d5-8ff3-0230b863718f","Type":"ContainerStarted","Data":"b0a353166948e4ee5ed28601a7f9ac89143d7b204f3b9fd913a58410aece3577"} Sep 17 23:06:22.554462 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.554418 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-svq9k" event={"ID":"d4518bc1-148d-4005-a3ec-2b36297214fa","Type":"ContainerStarted","Data":"934644cd9581952bd6a80859b38241fc5a7b210c59dc2b126ec18e2826b20cce"} Sep 17 23:06:22.556771 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.556739 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-82.ec2.internal" event={"ID":"9b0067a4916205e46911412e53db8082","Type":"ContainerStarted","Data":"d258cadf0b4fca71d13ab37c202ecebeeae74e5411305981e5f4a5a6d8aee7a6"} Sep 17 23:06:22.559642 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.559621 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zzcb8" event={"ID":"dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6","Type":"ContainerStarted","Data":"4277ae3305a1f4f80ba2787ca6c7e7bb975f7bd8efb222cd9cc311fd7d9aa06a"} Sep 17 23:06:22.561740 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.561715 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-trs6m" event={"ID":"bdbecc0a-d250-434e-8478-2f6f57c094d3","Type":"ContainerStarted","Data":"415a1a594f01dd86a8d907c144b8f0d8e84933e857045051180155ea3edef13e"} Sep 17 23:06:22.588644 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.586539 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-82.ec2.internal" podStartSLOduration=18.586525934 podStartE2EDuration="18.586525934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:22.585760058 +0000 UTC m=+21.519667195" watchObservedRunningTime="2026-09-17 23:06:22.586525934 +0000 UTC m=+21.520433071" Sep 17 23:06:22.603565 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.603518 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5s5pl" podStartSLOduration=2.105334943 podStartE2EDuration="19.603483251s" totalImagesPullingTime="17.498148308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:22.602944196 +0000 UTC m=+21.536851341" watchObservedRunningTime="2026-09-17 23:06:22.603483251 +0000 UTC m=+21.537390385" Sep 17 23:06:22.616119 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.616071 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-trs6m" podStartSLOduration=2.161259143 podStartE2EDuration="19.616054687s" totalImagesPullingTime="17.454795544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:22.615800348 +0000 UTC m=+21.549707483" watchObservedRunningTime="2026-09-17 23:06:22.616054687 +0000 UTC m=+21.549961822" Sep 17 23:06:22.631533 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.631454 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zzcb8" podStartSLOduration=2.135472132 podStartE2EDuration="19.631439124s" totalImagesPullingTime="17.495966992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:22.63084013 +0000 UTC m=+21.564747267" watchObservedRunningTime="2026-09-17 23:06:22.631439124 +0000 UTC m=+21.565346260" Sep 17 23:06:22.644693 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.644653 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-svq9k" podStartSLOduration=2.131463737 podStartE2EDuration="19.644641666s" totalImagesPullingTime="17.513177929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:22.644585483 +0000 UTC m=+21.578492608" watchObservedRunningTime="2026-09-17 23:06:22.644641666 +0000 UTC m=+21.578548798" Sep 17 23:06:22.658870 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:22.658830 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-k2bg2" podStartSLOduration=2.185342817 podStartE2EDuration="19.658820094s" totalImagesPullingTime="17.473477277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:22.658754642 +0000 UTC m=+21.592661778" watchObservedRunningTime="2026-09-17 23:06:22.658820094 +0000 UTC m=+21.592727228" Sep 17 23:06:23.565324 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:23.565289 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d8d5w" event={"ID":"fc91b818-8121-4735-81d4-2da07c40b110","Type":"ContainerStarted","Data":"2d6be46a8855ad3b2a1be390fbecb623a18373a2fc824266cb106ed84731536f"} Sep 17 23:06:23.567384 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:23.567357 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" event={"ID":"24a660d0-e7cf-402b-8b09-9fb5141b67ba","Type":"ContainerStarted","Data":"1104abcce06181117242206a4ed10f1ef9d111cef5b5210aa0466476e011012b"} Sep 17 23:06:23.571005 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:23.570969 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js95g" event={"ID":"c6f43b5a-9f32-4789-88f4-1f81e8d764c9","Type":"ContainerStarted","Data":"34ee7ef22a85a5192570c64af5882b2f17df6132006a47e416d4b6bd7956de90"} Sep 17 23:06:23.571114 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:23.571054 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js95g" event={"ID":"c6f43b5a-9f32-4789-88f4-1f81e8d764c9","Type":"ContainerStarted","Data":"8d601ca91d56646f75279c37bc0d52971b4c89759f26ad9af46cc4c9d13e149d"} Sep 17 23:06:23.578884 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:23.578846 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-d8d5w" podStartSLOduration=3.073755758 podStartE2EDuration="20.578831364s" totalImagesPullingTime="17.505075606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:23.578652271 +0000 UTC m=+22.512559407" watchObservedRunningTime="2026-09-17 23:06:23.578831364 +0000 UTC m=+22.512738499" Sep 17 23:06:24.287581 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:24.287540 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:24.288046 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:24.287584 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:24.288046 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:24.287584 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:24.288046 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:24.287667 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8sw2" podUID="62ca42a6-c013-4fce-b108-acefb7dab500" Sep 17 23:06:24.288046 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:24.287791 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2sz" podUID="3752b568-d53f-49ac-9095-0f61cee96767" Sep 17 23:06:24.288046 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:24.287894 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drknm" podUID="66bdcd7b-3b69-4bdd-9579-9e035e9d19b0" Sep 17 23:06:24.467475 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:24.467449 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 17 23:06:24.575148 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:24.575068 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal" event={"ID":"7c42993043d7f11ec0cbc6c6de37a429","Type":"ContainerStarted","Data":"6c1b85d134199a922db6830e354affd47e121def68efca0f6bf48fa13bfe530a"} Sep 17 23:06:24.577152 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:24.577118 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" event={"ID":"24a660d0-e7cf-402b-8b09-9fb5141b67ba","Type":"ContainerStarted","Data":"2cf0cf772c129329b5c71a466268bb7c4ace52794a6d242299e54b0a732cb50d"} Sep 17 23:06:24.590224 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:24.590184 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-82.ec2.internal" podStartSLOduration=20.590171439 podStartE2EDuration="20.590171439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:24.58967933 +0000 UTC m=+23.523586465" watchObservedRunningTime="2026-09-17 23:06:24.590171439 +0000 UTC m=+23.524078573" Sep 17 23:06:25.228961 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:25.228849 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T23:06:24.467472117Z","UUID":"07f48bb0-f5db-49f4-9f77-491f7a5d3c90","Handler":null,"Name":"","Endpoint":""} Sep 17 23:06:25.231089 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:25.231059 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 17 23:06:25.231225 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:25.231100 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 17 23:06:25.582272 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:25.582204 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" event={"ID":"24a660d0-e7cf-402b-8b09-9fb5141b67ba","Type":"ContainerStarted","Data":"0e919c6aa6a0c931f3e2f813f1522f74a9742f29b2437813691d162e93a67bfa"} Sep 17 23:06:25.586091 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:25.586065 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js95g" event={"ID":"c6f43b5a-9f32-4789-88f4-1f81e8d764c9","Type":"ContainerStarted","Data":"88b5760df3b94d00ac0cb8eb69440d9763cc4580bd7dc6b173258629dbc251ab"} Sep 17 23:06:26.223459 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:26.223430 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-svq9k" Sep 17 23:06:26.224002 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:26.223984 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-svq9k" Sep 17 23:06:26.238459 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:26.238406 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jkp5t" podStartSLOduration=2.5180583370000003 podStartE2EDuration="23.238387789s" totalImagesPullingTime="20.720329452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=3 observedRunningTime="2026-09-17 23:06:25.598594136 +0000 UTC m=+24.532501271" watchObservedRunningTime="2026-09-17 23:06:26.238387789 +0000 UTC m=+25.172294927" Sep 17 23:06:26.287130 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:26.287100 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:26.287290 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:26.287206 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:26.287354 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:26.287211 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drknm" podUID="66bdcd7b-3b69-4bdd-9579-9e035e9d19b0" Sep 17 23:06:26.287354 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:26.287336 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2sz" podUID="3752b568-d53f-49ac-9095-0f61cee96767" Sep 17 23:06:26.287437 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:26.287361 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:26.287470 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:26.287443 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8sw2" podUID="62ca42a6-c013-4fce-b108-acefb7dab500" Sep 17 23:06:26.588763 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:26.588696 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-svq9k" Sep 17 23:06:26.589248 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:26.589182 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-svq9k" Sep 17 23:06:27.593847 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:27.593616 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js95g" event={"ID":"c6f43b5a-9f32-4789-88f4-1f81e8d764c9","Type":"ContainerStarted","Data":"71ce6736067b116b97eccc596028297068fd10d925ecf2b939b2b6ed62e3845d"} Sep 17 23:06:27.594313 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:27.593898 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:27.595615 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:27.595595 2582 generic.go:356] "Generic (PLEG): container finished" podID="d4c08617-40c6-47d5-bcc8-e4765ca7a477" containerID="84920154bcb81898b6f54332dac558a262e94b4743ce3d1613857dcd3a1dbeac" exitCode=0 Sep 17 23:06:27.595728 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:27.595662 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqrzv" event={"ID":"d4c08617-40c6-47d5-bcc8-e4765ca7a477","Type":"ContainerDied","Data":"84920154bcb81898b6f54332dac558a262e94b4743ce3d1613857dcd3a1dbeac"} Sep 17 23:06:27.608534 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:27.608514 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:27.620448 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:27.620413 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-js95g" podStartSLOduration=7.111723413 podStartE2EDuration="24.620401613s" totalImagesPullingTime="17.5086782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:27.620160965 +0000 UTC m=+26.554068100" watchObservedRunningTime="2026-09-17 23:06:27.620401613 +0000 UTC m=+26.554308747" Sep 17 23:06:27.835033 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:27.835006 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:28.287922 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:28.287890 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:28.288085 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:28.288002 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:28.288085 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:28.288030 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:28.288085 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:28.288018 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drknm" podUID="66bdcd7b-3b69-4bdd-9579-9e035e9d19b0" Sep 17 23:06:28.288244 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:28.288106 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8sw2" podUID="62ca42a6-c013-4fce-b108-acefb7dab500" Sep 17 23:06:28.288244 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:28.288162 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2sz" podUID="3752b568-d53f-49ac-9095-0f61cee96767" Sep 17 23:06:28.600994 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:28.600791 2582 generic.go:356] "Generic (PLEG): container finished" podID="d4c08617-40c6-47d5-bcc8-e4765ca7a477" containerID="cd9f3bf34544962beb79ae39bac5783632e9734e439f5b0250472fae28f60a2f" exitCode=0 Sep 17 23:06:28.600994 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:28.600860 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqrzv" event={"ID":"d4c08617-40c6-47d5-bcc8-e4765ca7a477","Type":"ContainerDied","Data":"cd9f3bf34544962beb79ae39bac5783632e9734e439f5b0250472fae28f60a2f"} Sep 17 23:06:28.601563 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:28.601546 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:28.615483 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:28.615465 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:06:28.819123 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:28.819031 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xg2sz"] Sep 17 23:06:28.819252 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:28.819126 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:28.819252 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:28.819233 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2sz" podUID="3752b568-d53f-49ac-9095-0f61cee96767" Sep 17 23:06:28.822212 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:28.822178 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-drknm"] Sep 17 23:06:28.822345 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:28.822262 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:28.822402 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:28.822352 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drknm" podUID="66bdcd7b-3b69-4bdd-9579-9e035e9d19b0" Sep 17 23:06:28.822738 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:28.822719 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q8sw2"] Sep 17 23:06:28.822798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:28.822787 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:28.822871 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:28.822850 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8sw2" podUID="62ca42a6-c013-4fce-b108-acefb7dab500" Sep 17 23:06:29.605145 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:29.605112 2582 generic.go:356] "Generic (PLEG): container finished" podID="d4c08617-40c6-47d5-bcc8-e4765ca7a477" containerID="c870c1d9f860e981c9ad48796e0ab9d1a650f7eb7987ca88f746c0f8303f7a6e" exitCode=0 Sep 17 23:06:29.605594 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:29.605191 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqrzv" event={"ID":"d4c08617-40c6-47d5-bcc8-e4765ca7a477","Type":"ContainerDied","Data":"c870c1d9f860e981c9ad48796e0ab9d1a650f7eb7987ca88f746c0f8303f7a6e"} Sep 17 23:06:30.287864 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:30.287824 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:30.288029 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:30.287867 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:30.288029 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:30.287871 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:30.288029 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:30.287940 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8sw2" podUID="62ca42a6-c013-4fce-b108-acefb7dab500" Sep 17 23:06:30.288191 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:30.288035 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2sz" podUID="3752b568-d53f-49ac-9095-0f61cee96767" Sep 17 23:06:30.288191 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:30.288122 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drknm" podUID="66bdcd7b-3b69-4bdd-9579-9e035e9d19b0" Sep 17 23:06:32.287469 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:32.287434 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:32.288006 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:32.287551 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:32.288006 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:32.287554 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8sw2" podUID="62ca42a6-c013-4fce-b108-acefb7dab500" Sep 17 23:06:32.288006 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:32.287667 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drknm" podUID="66bdcd7b-3b69-4bdd-9579-9e035e9d19b0" Sep 17 23:06:32.288006 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:32.287558 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:32.288006 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:32.287817 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2sz" podUID="3752b568-d53f-49ac-9095-0f61cee96767" Sep 17 23:06:34.287367 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:34.287311 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:34.287724 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:34.287398 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:34.287724 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:34.287403 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drknm" podUID="66bdcd7b-3b69-4bdd-9579-9e035e9d19b0" Sep 17 23:06:34.287724 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:34.287431 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:34.287724 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:34.287548 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xg2sz" podUID="3752b568-d53f-49ac-9095-0f61cee96767" Sep 17 23:06:34.287724 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:34.287598 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8sw2" podUID="62ca42a6-c013-4fce-b108-acefb7dab500" Sep 17 23:06:34.620781 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:34.620750 2582 generic.go:356] "Generic (PLEG): container finished" podID="d4c08617-40c6-47d5-bcc8-e4765ca7a477" containerID="b9476a4e50536f3f0767a4a67c4b1548b665193891385a54ff34a6629dbcadd4" exitCode=0 Sep 17 23:06:34.620878 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:34.620799 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqrzv" event={"ID":"d4c08617-40c6-47d5-bcc8-e4765ca7a477","Type":"ContainerDied","Data":"b9476a4e50536f3f0767a4a67c4b1548b665193891385a54ff34a6629dbcadd4"} Sep 17 23:06:34.991936 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:34.991755 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-82.ec2.internal" event="NodeReady" Sep 17 23:06:34.992070 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:34.992022 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 17 23:06:35.033806 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.033775 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pkh9l"] Sep 17 23:06:35.047871 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.047847 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kk5m4"] Sep 17 23:06:35.048017 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.047975 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pkh9l" Sep 17 23:06:35.050484 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.050425 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 17 23:06:35.050643 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.050512 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 17 23:06:35.050904 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.050882 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7hp2b\"" Sep 17 23:06:35.063900 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.063881 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pkh9l"] Sep 17 23:06:35.063990 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.063913 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kk5m4"] Sep 17 23:06:35.063990 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.063979 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kk5m4" Sep 17 23:06:35.066461 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.066422 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 17 23:06:35.066721 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.066645 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 17 23:06:35.066721 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.066686 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9ck9b\"" Sep 17 23:06:35.066837 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.066798 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 17 23:06:35.185407 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.185318 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8swmf\" (UniqueName: \"kubernetes.io/projected/5cd06380-e744-4e37-a5f9-3cac49aee5d4-kube-api-access-8swmf\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:06:35.185407 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.185366 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd06380-e744-4e37-a5f9-3cac49aee5d4-tmp-dir\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:06:35.185607 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.185427 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vbkq\" (UniqueName: \"kubernetes.io/projected/48bdac6b-06a2-4d13-8a48-c0d5b5af03b5-kube-api-access-6vbkq\") pod \"ingress-canary-kk5m4\" (UID: \"48bdac6b-06a2-4d13-8a48-c0d5b5af03b5\") " pod="openshift-ingress-canary/ingress-canary-kk5m4" Sep 17 23:06:35.185607 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.185551 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:06:35.185607 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.185590 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48bdac6b-06a2-4d13-8a48-c0d5b5af03b5-cert\") pod \"ingress-canary-kk5m4\" (UID: \"48bdac6b-06a2-4d13-8a48-c0d5b5af03b5\") " pod="openshift-ingress-canary/ingress-canary-kk5m4" Sep 17 23:06:35.185716 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.185622 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5cd06380-e744-4e37-a5f9-3cac49aee5d4-config-volume\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:06:35.285878 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.285844 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48bdac6b-06a2-4d13-8a48-c0d5b5af03b5-cert\") pod \"ingress-canary-kk5m4\" (UID: \"48bdac6b-06a2-4d13-8a48-c0d5b5af03b5\") " pod="openshift-ingress-canary/ingress-canary-kk5m4" Sep 17 23:06:35.286017 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.285891 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5cd06380-e744-4e37-a5f9-3cac49aee5d4-config-volume\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:06:35.286017 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.285918 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8swmf\" (UniqueName: \"kubernetes.io/projected/5cd06380-e744-4e37-a5f9-3cac49aee5d4-kube-api-access-8swmf\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:06:35.286017 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.285940 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd06380-e744-4e37-a5f9-3cac49aee5d4-tmp-dir\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:06:35.286017 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.285955 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vbkq\" (UniqueName: \"kubernetes.io/projected/48bdac6b-06a2-4d13-8a48-c0d5b5af03b5-kube-api-access-6vbkq\") pod \"ingress-canary-kk5m4\" (UID: \"48bdac6b-06a2-4d13-8a48-c0d5b5af03b5\") " pod="openshift-ingress-canary/ingress-canary-kk5m4" Sep 17 23:06:35.286017 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.285979 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:06:35.286220 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:35.286194 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 23:06:35.286294 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.286276 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd06380-e744-4e37-a5f9-3cac49aee5d4-tmp-dir\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:06:35.286342 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:35.286292 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls podName:5cd06380-e744-4e37-a5f9-3cac49aee5d4 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:35.786273345 +0000 UTC m=+34.720180459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls") pod "dns-default-pkh9l" (UID: "5cd06380-e744-4e37-a5f9-3cac49aee5d4") : secret "dns-default-metrics-tls" not found Sep 17 23:06:35.289995 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.289971 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48bdac6b-06a2-4d13-8a48-c0d5b5af03b5-cert\") pod \"ingress-canary-kk5m4\" (UID: \"48bdac6b-06a2-4d13-8a48-c0d5b5af03b5\") " pod="openshift-ingress-canary/ingress-canary-kk5m4" Sep 17 23:06:35.294327 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.294307 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vbkq\" (UniqueName: \"kubernetes.io/projected/48bdac6b-06a2-4d13-8a48-c0d5b5af03b5-kube-api-access-6vbkq\") pod \"ingress-canary-kk5m4\" (UID: \"48bdac6b-06a2-4d13-8a48-c0d5b5af03b5\") " pod="openshift-ingress-canary/ingress-canary-kk5m4" Sep 17 23:06:35.297401 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.297379 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5cd06380-e744-4e37-a5f9-3cac49aee5d4-config-volume\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:06:35.298939 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.298919 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8swmf\" (UniqueName: \"kubernetes.io/projected/5cd06380-e744-4e37-a5f9-3cac49aee5d4-kube-api-access-8swmf\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:06:35.373167 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.373136 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kk5m4" Sep 17 23:06:35.625913 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.625885 2582 generic.go:356] "Generic (PLEG): container finished" podID="d4c08617-40c6-47d5-bcc8-e4765ca7a477" containerID="fb4162528038576dbe9644bf774fee9d6b5855a4bd4b1fa498879d5b35e863b8" exitCode=0 Sep 17 23:06:35.626064 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.625933 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqrzv" event={"ID":"d4c08617-40c6-47d5-bcc8-e4765ca7a477","Type":"ContainerDied","Data":"fb4162528038576dbe9644bf774fee9d6b5855a4bd4b1fa498879d5b35e863b8"} Sep 17 23:06:35.774635 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.774608 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kk5m4"] Sep 17 23:06:35.782493 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:06:35.782470 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bdac6b_06a2_4d13_8a48_c0d5b5af03b5.slice/crio-ac232fe719df86501460b24ece9f25424596adceb4b3a7d0986ef5d32cc58d77 WatchSource:0}: Error finding container ac232fe719df86501460b24ece9f25424596adceb4b3a7d0986ef5d32cc58d77: Status 404 returned error can't find the container with id ac232fe719df86501460b24ece9f25424596adceb4b3a7d0986ef5d32cc58d77 Sep 17 23:06:35.790140 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.790011 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:06:35.790225 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:35.790156 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 23:06:35.790225 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:35.790213 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls podName:5cd06380-e744-4e37-a5f9-3cac49aee5d4 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:36.790196233 +0000 UTC m=+35.724103360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls") pod "dns-default-pkh9l" (UID: "5cd06380-e744-4e37-a5f9-3cac49aee5d4") : secret "dns-default-metrics-tls" not found Sep 17 23:06:35.990944 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.990915 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret\") pod \"global-pull-secret-syncer-drknm\" (UID: \"66bdcd7b-3b69-4bdd-9579-9e035e9d19b0\") " pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:35.991075 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.990957 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs\") pod \"network-metrics-daemon-xg2sz\" (UID: \"3752b568-d53f-49ac-9095-0f61cee96767\") " pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:35.991075 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:35.990981 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl4z7\" (UniqueName: \"kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7\") pod \"network-check-target-q8sw2\" (UID: \"62ca42a6-c013-4fce-b108-acefb7dab500\") " pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:35.991075 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:35.991046 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:35.991075 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:35.991058 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:35.991260 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:35.991098 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret podName:66bdcd7b-3b69-4bdd-9579-9e035e9d19b0 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:07.991084842 +0000 UTC m=+66.924991959 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret") pod "global-pull-secret-syncer-drknm" (UID: "66bdcd7b-3b69-4bdd-9579-9e035e9d19b0") : object "kube-system"/"original-pull-secret" not registered Sep 17 23:06:35.991260 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:35.991112 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs podName:3752b568-d53f-49ac-9095-0f61cee96767 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:07.991105403 +0000 UTC m=+66.925012516 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs") pod "network-metrics-daemon-xg2sz" (UID: "3752b568-d53f-49ac-9095-0f61cee96767") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 23:06:35.991260 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:35.991122 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 23:06:35.991260 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:35.991142 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 23:06:35.991260 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:35.991153 2582 projected.go:196] Error preparing data for projected volume kube-api-access-gl4z7 for pod openshift-network-diagnostics/network-check-target-q8sw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:35.991260 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:35.991203 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7 podName:62ca42a6-c013-4fce-b108-acefb7dab500 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:07.991189029 +0000 UTC m=+66.925096159 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gl4z7" (UniqueName: "kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7") pod "network-check-target-q8sw2" (UID: "62ca42a6-c013-4fce-b108-acefb7dab500") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 23:06:36.287909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:36.287832 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:06:36.287909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:36.287880 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:06:36.287909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:36.287880 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:06:36.290842 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:36.290659 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 23:06:36.290842 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:36.290685 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hrj9x\"" Sep 17 23:06:36.290842 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:36.290695 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 23:06:36.290842 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:36.290696 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-twn2q\"" Sep 17 23:06:36.290842 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:36.290770 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 23:06:36.291741 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:36.291373 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 23:06:36.632310 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:36.632212 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqrzv" event={"ID":"d4c08617-40c6-47d5-bcc8-e4765ca7a477","Type":"ContainerStarted","Data":"d0f78d24735c8dd27df2d3d9327d7e9b9efdee7dd6985cd3e159ebf9a5300af6"} Sep 17 23:06:36.633597 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:36.633562 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kk5m4" event={"ID":"48bdac6b-06a2-4d13-8a48-c0d5b5af03b5","Type":"ContainerStarted","Data":"ac232fe719df86501460b24ece9f25424596adceb4b3a7d0986ef5d32cc58d77"} Sep 17 23:06:36.656584 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:36.656537 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jqrzv" podStartSLOduration=6.013856607 podStartE2EDuration="33.656518089s" totalImagesPullingTime="27.642661482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:36.655806767 +0000 UTC m=+35.589713902" watchObservedRunningTime="2026-09-17 23:06:36.656518089 +0000 UTC m=+35.590425225" Sep 17 23:06:36.797277 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:36.797245 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:06:36.797441 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:36.797370 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 23:06:36.797441 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:36.797427 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls podName:5cd06380-e744-4e37-a5f9-3cac49aee5d4 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:38.797409432 +0000 UTC m=+37.731316562 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls") pod "dns-default-pkh9l" (UID: "5cd06380-e744-4e37-a5f9-3cac49aee5d4") : secret "dns-default-metrics-tls" not found Sep 17 23:06:38.639363 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:38.639293 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kk5m4" event={"ID":"48bdac6b-06a2-4d13-8a48-c0d5b5af03b5","Type":"ContainerStarted","Data":"0bfcef2586283db436f210c3cee411f29918bea6df882a65963dbff04cc50443"} Sep 17 23:06:38.653659 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:38.653589 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kk5m4" podStartSLOduration=1.484266976 podStartE2EDuration="3.653573831s" totalImagesPullingTime="2.169306855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:38.65306923 +0000 UTC m=+37.586976364" watchObservedRunningTime="2026-09-17 23:06:38.653573831 +0000 UTC m=+37.587480965" Sep 17 23:06:38.812192 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:38.812159 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:06:38.812348 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:38.812261 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 23:06:38.812348 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:38.812311 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls podName:5cd06380-e744-4e37-a5f9-3cac49aee5d4 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:42.812297534 +0000 UTC m=+41.746204659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls") pod "dns-default-pkh9l" (UID: "5cd06380-e744-4e37-a5f9-3cac49aee5d4") : secret "dns-default-metrics-tls" not found Sep 17 23:06:42.834986 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:42.834943 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:06:42.835394 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:42.835069 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 23:06:42.835394 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:42.835120 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls podName:5cd06380-e744-4e37-a5f9-3cac49aee5d4 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:50.835107143 +0000 UTC m=+49.769014256 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls") pod "dns-default-pkh9l" (UID: "5cd06380-e744-4e37-a5f9-3cac49aee5d4") : secret "dns-default-metrics-tls" not found Sep 17 23:06:50.890338 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:06:50.890302 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:06:50.890718 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:50.890416 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 23:06:50.890718 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:06:50.890469 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls podName:5cd06380-e744-4e37-a5f9-3cac49aee5d4 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:06.890456003 +0000 UTC m=+65.824363116 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls") pod "dns-default-pkh9l" (UID: "5cd06380-e744-4e37-a5f9-3cac49aee5d4") : secret "dns-default-metrics-tls" not found Sep 17 23:07:00.618158 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:00.618127 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-js95g" Sep 17 23:07:06.896259 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:06.896215 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:07:06.896758 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:07:06.896380 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 23:07:06.896758 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:07:06.896459 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls podName:5cd06380-e744-4e37-a5f9-3cac49aee5d4 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:38.896440379 +0000 UTC m=+97.830347494 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls") pod "dns-default-pkh9l" (UID: "5cd06380-e744-4e37-a5f9-3cac49aee5d4") : secret "dns-default-metrics-tls" not found Sep 17 23:07:08.003717 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:08.003675 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret\") pod \"global-pull-secret-syncer-drknm\" (UID: \"66bdcd7b-3b69-4bdd-9579-9e035e9d19b0\") " pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:07:08.003717 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:08.003721 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs\") pod \"network-metrics-daemon-xg2sz\" (UID: \"3752b568-d53f-49ac-9095-0f61cee96767\") " pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:07:08.004131 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:08.003741 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl4z7\" (UniqueName: \"kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7\") pod \"network-check-target-q8sw2\" (UID: \"62ca42a6-c013-4fce-b108-acefb7dab500\") " pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:07:08.005982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:08.005963 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 23:07:08.006029 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:08.005963 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 23:07:08.006073 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:08.006052 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 23:07:08.014776 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:07:08.014758 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 23:07:08.014836 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:07:08.014826 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs podName:3752b568-d53f-49ac-9095-0f61cee96767 nodeName:}" failed. No retries permitted until 2026-09-17 23:08:12.014811569 +0000 UTC m=+130.948718696 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs") pod "network-metrics-daemon-xg2sz" (UID: "3752b568-d53f-49ac-9095-0f61cee96767") : secret "metrics-daemon-secret" not found Sep 17 23:07:08.016712 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:08.016692 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 23:07:08.017776 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:08.017755 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bdcd7b-3b69-4bdd-9579-9e035e9d19b0-original-pull-secret\") pod \"global-pull-secret-syncer-drknm\" (UID: \"66bdcd7b-3b69-4bdd-9579-9e035e9d19b0\") " pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:07:08.027057 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:08.027036 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl4z7\" (UniqueName: \"kubernetes.io/projected/62ca42a6-c013-4fce-b108-acefb7dab500-kube-api-access-gl4z7\") pod \"network-check-target-q8sw2\" (UID: \"62ca42a6-c013-4fce-b108-acefb7dab500\") " pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:07:08.106719 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:08.106699 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-twn2q\"" Sep 17 23:07:08.114812 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:08.114795 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:07:08.116023 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:08.116004 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drknm" Sep 17 23:07:08.382555 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:08.382458 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-drknm"] Sep 17 23:07:08.385553 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:08.385528 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q8sw2"] Sep 17 23:07:08.385662 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:07:08.385638 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66bdcd7b_3b69_4bdd_9579_9e035e9d19b0.slice/crio-ce75c1d1d9e7e43b2c6d95d08c51528f59c3f9be993c36b0a01ede91c1450972 WatchSource:0}: Error finding container ce75c1d1d9e7e43b2c6d95d08c51528f59c3f9be993c36b0a01ede91c1450972: Status 404 returned error can't find the container with id ce75c1d1d9e7e43b2c6d95d08c51528f59c3f9be993c36b0a01ede91c1450972 Sep 17 23:07:08.388182 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:07:08.388160 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ca42a6_c013_4fce_b108_acefb7dab500.slice/crio-133743669cfd44e5ddc78ddc9f09c2294883ed6c25846e89e0885f007dee32a9 WatchSource:0}: Error finding container 133743669cfd44e5ddc78ddc9f09c2294883ed6c25846e89e0885f007dee32a9: Status 404 returned error can't find the container with id 133743669cfd44e5ddc78ddc9f09c2294883ed6c25846e89e0885f007dee32a9 Sep 17 23:07:08.711325 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:08.711242 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-drknm" event={"ID":"66bdcd7b-3b69-4bdd-9579-9e035e9d19b0","Type":"ContainerStarted","Data":"ce75c1d1d9e7e43b2c6d95d08c51528f59c3f9be993c36b0a01ede91c1450972"} Sep 17 23:07:08.712430 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:08.712403 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q8sw2" event={"ID":"62ca42a6-c013-4fce-b108-acefb7dab500","Type":"ContainerStarted","Data":"133743669cfd44e5ddc78ddc9f09c2294883ed6c25846e89e0885f007dee32a9"} Sep 17 23:07:13.726015 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:13.725983 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-drknm" event={"ID":"66bdcd7b-3b69-4bdd-9579-9e035e9d19b0","Type":"ContainerStarted","Data":"e88f263b48fa2694fdd5a344c62ab9ba937c0bc60b140a53f4eb3a261e7014cd"} Sep 17 23:07:13.727459 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:13.727438 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q8sw2" event={"ID":"62ca42a6-c013-4fce-b108-acefb7dab500","Type":"ContainerStarted","Data":"584aec2bd95bc4eaffbfecde7387794a5060f0d3abdca4152c5d3e18945891d4"} Sep 17 23:07:13.727575 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:13.727494 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:07:13.742467 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:13.742426 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-drknm" podStartSLOduration=66.382676844 podStartE2EDuration="1m10.742414406s" totalImagesPullingTime="4.359737562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:13.741803255 +0000 UTC m=+72.675710392" watchObservedRunningTime="2026-09-17 23:07:13.742414406 +0000 UTC m=+72.676321540" Sep 17 23:07:13.759571 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:13.759531 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q8sw2" podStartSLOduration=66.404561988 podStartE2EDuration="1m10.759519848s" totalImagesPullingTime="4.35495786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:13.759292072 +0000 UTC m=+72.693199209" watchObservedRunningTime="2026-09-17 23:07:13.759519848 +0000 UTC m=+72.693426979" Sep 17 23:07:28.417984 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:28.417868 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf"] Sep 17 23:07:28.420445 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:28.420429 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" Sep 17 23:07:28.422553 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:28.422531 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 17 23:07:28.422659 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:28.422549 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 17 23:07:28.423425 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:28.423410 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 17 23:07:28.423482 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:28.423440 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 17 23:07:28.429321 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:28.429298 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf"] Sep 17 23:07:28.432106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:28.432085 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bf3eb4e9-c80d-4166-a50b-28cc7f291fda-klusterlet-config\") pod \"klusterlet-addon-workmgr-6677cf84b5-9ppzf\" (UID: \"bf3eb4e9-c80d-4166-a50b-28cc7f291fda\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" Sep 17 23:07:28.432202 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:28.432123 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf3eb4e9-c80d-4166-a50b-28cc7f291fda-tmp\") pod \"klusterlet-addon-workmgr-6677cf84b5-9ppzf\" (UID: \"bf3eb4e9-c80d-4166-a50b-28cc7f291fda\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" Sep 17 23:07:28.432202 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:28.432143 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6j6\" (UniqueName: \"kubernetes.io/projected/bf3eb4e9-c80d-4166-a50b-28cc7f291fda-kube-api-access-ck6j6\") pod \"klusterlet-addon-workmgr-6677cf84b5-9ppzf\" (UID: \"bf3eb4e9-c80d-4166-a50b-28cc7f291fda\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" Sep 17 23:07:28.533480 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:28.533437 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bf3eb4e9-c80d-4166-a50b-28cc7f291fda-klusterlet-config\") pod \"klusterlet-addon-workmgr-6677cf84b5-9ppzf\" (UID: \"bf3eb4e9-c80d-4166-a50b-28cc7f291fda\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" Sep 17 23:07:28.533635 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:28.533482 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf3eb4e9-c80d-4166-a50b-28cc7f291fda-tmp\") pod \"klusterlet-addon-workmgr-6677cf84b5-9ppzf\" (UID: \"bf3eb4e9-c80d-4166-a50b-28cc7f291fda\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" Sep 17 23:07:28.533689 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:28.533673 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6j6\" (UniqueName: \"kubernetes.io/projected/bf3eb4e9-c80d-4166-a50b-28cc7f291fda-kube-api-access-ck6j6\") pod \"klusterlet-addon-workmgr-6677cf84b5-9ppzf\" (UID: \"bf3eb4e9-c80d-4166-a50b-28cc7f291fda\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" Sep 17 23:07:28.533834 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:28.533819 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf3eb4e9-c80d-4166-a50b-28cc7f291fda-tmp\") pod \"klusterlet-addon-workmgr-6677cf84b5-9ppzf\" (UID: \"bf3eb4e9-c80d-4166-a50b-28cc7f291fda\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" Sep 17 23:07:28.535999 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:28.535967 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bf3eb4e9-c80d-4166-a50b-28cc7f291fda-klusterlet-config\") pod \"klusterlet-addon-workmgr-6677cf84b5-9ppzf\" (UID: \"bf3eb4e9-c80d-4166-a50b-28cc7f291fda\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" Sep 17 23:07:28.541090 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:28.541064 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6j6\" (UniqueName: \"kubernetes.io/projected/bf3eb4e9-c80d-4166-a50b-28cc7f291fda-kube-api-access-ck6j6\") pod \"klusterlet-addon-workmgr-6677cf84b5-9ppzf\" (UID: \"bf3eb4e9-c80d-4166-a50b-28cc7f291fda\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" Sep 17 23:07:28.729461 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:28.729423 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" Sep 17 23:07:28.997659 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:28.997584 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf"] Sep 17 23:07:29.000910 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:07:29.000887 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3eb4e9_c80d_4166_a50b_28cc7f291fda.slice/crio-027798585a99c689efd3ccb99613af39c3ea03fd16b8b684bcd71ae5bd9a18aa WatchSource:0}: Error finding container 027798585a99c689efd3ccb99613af39c3ea03fd16b8b684bcd71ae5bd9a18aa: Status 404 returned error can't find the container with id 027798585a99c689efd3ccb99613af39c3ea03fd16b8b684bcd71ae5bd9a18aa Sep 17 23:07:29.766484 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:29.766434 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" event={"ID":"bf3eb4e9-c80d-4166-a50b-28cc7f291fda","Type":"ContainerStarted","Data":"027798585a99c689efd3ccb99613af39c3ea03fd16b8b684bcd71ae5bd9a18aa"} Sep 17 23:07:32.776931 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:32.776898 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" event={"ID":"bf3eb4e9-c80d-4166-a50b-28cc7f291fda","Type":"ContainerStarted","Data":"2cdb69f7613d3f47875555054544d10455aa57d66935c54b7f762e31df3e1b82"} Sep 17 23:07:32.777342 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:32.777102 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" Sep 17 23:07:32.778687 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:32.778667 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" Sep 17 23:07:32.791636 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:32.791597 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" podStartSLOduration=1.53792935 podStartE2EDuration="4.791583051s" totalImagesPullingTime="3.253653701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:32.790902096 +0000 UTC m=+91.724809236" watchObservedRunningTime="2026-09-17 23:07:32.791583051 +0000 UTC m=+91.725490188" Sep 17 23:07:38.910183 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:38.910132 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:07:38.910602 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:07:38.910286 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 23:07:38.910602 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:07:38.910367 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls podName:5cd06380-e744-4e37-a5f9-3cac49aee5d4 nodeName:}" failed. No retries permitted until 2026-09-17 23:08:42.91035076 +0000 UTC m=+161.844257888 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls") pod "dns-default-pkh9l" (UID: "5cd06380-e744-4e37-a5f9-3cac49aee5d4") : secret "dns-default-metrics-tls" not found Sep 17 23:07:44.732270 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:07:44.732241 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q8sw2" Sep 17 23:08:12.022708 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:08:12.022662 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs\") pod \"network-metrics-daemon-xg2sz\" (UID: \"3752b568-d53f-49ac-9095-0f61cee96767\") " pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:08:12.023198 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:08:12.022810 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 23:08:12.023198 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:08:12.022872 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs podName:3752b568-d53f-49ac-9095-0f61cee96767 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:14.022856382 +0000 UTC m=+252.956763511 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs") pod "network-metrics-daemon-xg2sz" (UID: "3752b568-d53f-49ac-9095-0f61cee96767") : secret "metrics-daemon-secret" not found Sep 17 23:08:38.057457 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:08:38.057410 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pkh9l" podUID="5cd06380-e744-4e37-a5f9-3cac49aee5d4" Sep 17 23:08:38.933401 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:08:38.933371 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pkh9l" Sep 17 23:08:39.310560 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:08:39.310514 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xg2sz" podUID="3752b568-d53f-49ac-9095-0f61cee96767" Sep 17 23:08:42.933300 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:08:42.933262 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:08:42.933881 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:08:42.933381 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 23:08:42.933881 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:08:42.933462 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls podName:5cd06380-e744-4e37-a5f9-3cac49aee5d4 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:44.933443121 +0000 UTC m=+283.867350254 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls") pod "dns-default-pkh9l" (UID: "5cd06380-e744-4e37-a5f9-3cac49aee5d4") : secret "dns-default-metrics-tls" not found Sep 17 23:08:51.287269 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:08:51.287178 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:09:32.778260 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:09:32.778205 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" podUID="bf3eb4e9-c80d-4166-a50b-28cc7f291fda" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.8:8000/readyz\": dial tcp 10.133.0.8:8000: connect: connection refused" Sep 17 23:09:33.060379 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:09:33.060291 2582 generic.go:356] "Generic (PLEG): container finished" podID="bf3eb4e9-c80d-4166-a50b-28cc7f291fda" containerID="2cdb69f7613d3f47875555054544d10455aa57d66935c54b7f762e31df3e1b82" exitCode=1 Sep 17 23:09:33.060379 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:09:33.060342 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" event={"ID":"bf3eb4e9-c80d-4166-a50b-28cc7f291fda","Type":"ContainerDied","Data":"2cdb69f7613d3f47875555054544d10455aa57d66935c54b7f762e31df3e1b82"} Sep 17 23:09:33.060677 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:09:33.060662 2582 scope.go:117] "RemoveContainer" containerID="2cdb69f7613d3f47875555054544d10455aa57d66935c54b7f762e31df3e1b82" Sep 17 23:09:34.064891 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:09:34.064858 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" event={"ID":"bf3eb4e9-c80d-4166-a50b-28cc7f291fda","Type":"ContainerStarted","Data":"c9a5be8ad996bc558004835cd774ec894f39490af2141a3154cfebbffe5f6d31"} Sep 17 23:09:34.065298 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:09:34.065153 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" Sep 17 23:09:34.065851 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:09:34.065830 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6677cf84b5-9ppzf" Sep 17 23:10:09.351378 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.351348 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-d6px4"] Sep 17 23:10:09.354464 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.354428 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-d6px4" Sep 17 23:10:09.355515 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.355476 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7fd9896dfd-2sbbg"] Sep 17 23:10:09.357081 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.357061 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 17 23:10:09.357595 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.357578 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 17 23:10:09.357713 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.357697 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8jcbq\"" Sep 17 23:10:09.357761 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.357749 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 17 23:10:09.357799 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.357766 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 17 23:10:09.358065 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.358045 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.360206 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.360187 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 17 23:10:09.360449 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.360435 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 17 23:10:09.360544 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.360460 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 17 23:10:09.360685 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.360666 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4qnpb\"" Sep 17 23:10:09.365615 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.365596 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-d6px4"] Sep 17 23:10:09.366025 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.366007 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 17 23:10:09.389159 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.389137 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fd9896dfd-2sbbg"] Sep 17 23:10:09.392048 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.392026 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9ff60ca3-2533-4ed5-9cc1-56bad886cc5d-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-d6px4\" (UID: \"9ff60ca3-2533-4ed5-9cc1-56bad886cc5d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-d6px4" Sep 17 23:10:09.392141 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.392060 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5acef9f4-6131-4772-abab-596a3d81af48-registry-certificates\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.392141 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.392084 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9ff60ca3-2533-4ed5-9cc1-56bad886cc5d-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-d6px4\" (UID: \"9ff60ca3-2533-4ed5-9cc1-56bad886cc5d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-d6px4" Sep 17 23:10:09.392242 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.392138 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5acef9f4-6131-4772-abab-596a3d81af48-ca-trust-extracted\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.392242 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.392168 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.392242 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.392190 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5acef9f4-6131-4772-abab-596a3d81af48-installation-pull-secrets\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.392242 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.392228 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5acef9f4-6131-4772-abab-596a3d81af48-trusted-ca\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.392424 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.392252 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5acef9f4-6131-4772-abab-596a3d81af48-image-registry-private-configuration\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.392424 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.392270 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-bound-sa-token\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.392424 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.392288 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp6pb\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-kube-api-access-jp6pb\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.392424 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.392315 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bdb5\" (UniqueName: \"kubernetes.io/projected/9ff60ca3-2533-4ed5-9cc1-56bad886cc5d-kube-api-access-8bdb5\") pod \"kube-storage-version-migrator-operator-55889f7b47-d6px4\" (UID: \"9ff60ca3-2533-4ed5-9cc1-56bad886cc5d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-d6px4" Sep 17 23:10:09.442976 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.442950 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jwjpz"] Sep 17 23:10:09.445660 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.445644 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jwjpz" Sep 17 23:10:09.447781 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.447764 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-n7ztd\"" Sep 17 23:10:09.447980 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.447966 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 17 23:10:09.448244 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.448228 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 17 23:10:09.449554 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.449536 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h"] Sep 17 23:10:09.452157 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.452141 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h" Sep 17 23:10:09.454409 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.454390 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 17 23:10:09.454632 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.454617 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 17 23:10:09.454692 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.454662 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 17 23:10:09.454886 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.454868 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wlz6s\"" Sep 17 23:10:09.456291 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.456271 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jwjpz"] Sep 17 23:10:09.465179 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.465162 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h"] Sep 17 23:10:09.493606 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.493584 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.493687 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.493610 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5acef9f4-6131-4772-abab-596a3d81af48-installation-pull-secrets\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.493687 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.493630 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt9xs\" (UniqueName: \"kubernetes.io/projected/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-kube-api-access-tt9xs\") pod \"cluster-samples-operator-685d7d68b5-mmq2h\" (UID: \"6af371a5-0c1e-484c-a3bc-abc43ab5bb93\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h" Sep 17 23:10:09.493687 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.493654 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5acef9f4-6131-4772-abab-596a3d81af48-trusted-ca\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.493687 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.493677 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-mmq2h\" (UID: \"6af371a5-0c1e-484c-a3bc-abc43ab5bb93\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h" Sep 17 23:10:09.493822 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:09.493720 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 23:10:09.493822 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:09.493734 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd9896dfd-2sbbg: secret "image-registry-tls" not found Sep 17 23:10:09.493822 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.493724 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5acef9f4-6131-4772-abab-596a3d81af48-image-registry-private-configuration\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.493822 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:09.493779 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls podName:5acef9f4-6131-4772-abab-596a3d81af48 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:09.993761985 +0000 UTC m=+248.927669119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls") pod "image-registry-7fd9896dfd-2sbbg" (UID: "5acef9f4-6131-4772-abab-596a3d81af48") : secret "image-registry-tls" not found Sep 17 23:10:09.493822 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.493801 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-bound-sa-token\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.493822 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.493818 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp6pb\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-kube-api-access-jp6pb\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.494095 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.493853 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bdb5\" (UniqueName: \"kubernetes.io/projected/9ff60ca3-2533-4ed5-9cc1-56bad886cc5d-kube-api-access-8bdb5\") pod \"kube-storage-version-migrator-operator-55889f7b47-d6px4\" (UID: \"9ff60ca3-2533-4ed5-9cc1-56bad886cc5d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-d6px4" Sep 17 23:10:09.494095 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.493907 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9ff60ca3-2533-4ed5-9cc1-56bad886cc5d-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-d6px4\" (UID: \"9ff60ca3-2533-4ed5-9cc1-56bad886cc5d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-d6px4" Sep 17 23:10:09.494095 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.493934 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rm6f\" (UniqueName: \"kubernetes.io/projected/c7efbb43-0384-4fed-ac1c-4f8664a12e72-kube-api-access-4rm6f\") pod \"volume-data-source-validator-5c4db8bdf9-jwjpz\" (UID: \"c7efbb43-0384-4fed-ac1c-4f8664a12e72\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jwjpz" Sep 17 23:10:09.494095 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.493961 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5acef9f4-6131-4772-abab-596a3d81af48-registry-certificates\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.494095 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.493988 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9ff60ca3-2533-4ed5-9cc1-56bad886cc5d-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-d6px4\" (UID: \"9ff60ca3-2533-4ed5-9cc1-56bad886cc5d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-d6px4" Sep 17 23:10:09.494095 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.494031 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5acef9f4-6131-4772-abab-596a3d81af48-ca-trust-extracted\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.494423 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.494401 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5acef9f4-6131-4772-abab-596a3d81af48-ca-trust-extracted\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.494628 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.494606 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5acef9f4-6131-4772-abab-596a3d81af48-registry-certificates\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.494759 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.494738 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5acef9f4-6131-4772-abab-596a3d81af48-trusted-ca\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.494800 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.494777 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9ff60ca3-2533-4ed5-9cc1-56bad886cc5d-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-d6px4\" (UID: \"9ff60ca3-2533-4ed5-9cc1-56bad886cc5d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-d6px4" Sep 17 23:10:09.496087 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.496062 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5acef9f4-6131-4772-abab-596a3d81af48-image-registry-private-configuration\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.496182 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.496098 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5acef9f4-6131-4772-abab-596a3d81af48-installation-pull-secrets\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.496182 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.496104 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9ff60ca3-2533-4ed5-9cc1-56bad886cc5d-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-d6px4\" (UID: \"9ff60ca3-2533-4ed5-9cc1-56bad886cc5d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-d6px4" Sep 17 23:10:09.502003 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.501980 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bdb5\" (UniqueName: \"kubernetes.io/projected/9ff60ca3-2533-4ed5-9cc1-56bad886cc5d-kube-api-access-8bdb5\") pod \"kube-storage-version-migrator-operator-55889f7b47-d6px4\" (UID: \"9ff60ca3-2533-4ed5-9cc1-56bad886cc5d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-d6px4" Sep 17 23:10:09.502106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.502085 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp6pb\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-kube-api-access-jp6pb\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.502381 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.502365 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-bound-sa-token\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.595219 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.595195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rm6f\" (UniqueName: \"kubernetes.io/projected/c7efbb43-0384-4fed-ac1c-4f8664a12e72-kube-api-access-4rm6f\") pod \"volume-data-source-validator-5c4db8bdf9-jwjpz\" (UID: \"c7efbb43-0384-4fed-ac1c-4f8664a12e72\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jwjpz" Sep 17 23:10:09.595348 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.595243 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt9xs\" (UniqueName: \"kubernetes.io/projected/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-kube-api-access-tt9xs\") pod \"cluster-samples-operator-685d7d68b5-mmq2h\" (UID: \"6af371a5-0c1e-484c-a3bc-abc43ab5bb93\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h" Sep 17 23:10:09.595348 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.595268 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-mmq2h\" (UID: \"6af371a5-0c1e-484c-a3bc-abc43ab5bb93\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h" Sep 17 23:10:09.595425 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:09.595350 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 23:10:09.595425 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:09.595402 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-samples-operator-tls podName:6af371a5-0c1e-484c-a3bc-abc43ab5bb93 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:10.095390228 +0000 UTC m=+249.029297354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-mmq2h" (UID: "6af371a5-0c1e-484c-a3bc-abc43ab5bb93") : secret "samples-operator-tls" not found Sep 17 23:10:09.603119 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.603069 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt9xs\" (UniqueName: \"kubernetes.io/projected/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-kube-api-access-tt9xs\") pod \"cluster-samples-operator-685d7d68b5-mmq2h\" (UID: \"6af371a5-0c1e-484c-a3bc-abc43ab5bb93\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h" Sep 17 23:10:09.603558 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.603541 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rm6f\" (UniqueName: \"kubernetes.io/projected/c7efbb43-0384-4fed-ac1c-4f8664a12e72-kube-api-access-4rm6f\") pod \"volume-data-source-validator-5c4db8bdf9-jwjpz\" (UID: \"c7efbb43-0384-4fed-ac1c-4f8664a12e72\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jwjpz" Sep 17 23:10:09.665534 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.665514 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-d6px4" Sep 17 23:10:09.753786 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.753758 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jwjpz" Sep 17 23:10:09.928307 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.928242 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-d6px4"] Sep 17 23:10:09.931416 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:10:09.931391 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff60ca3_2533_4ed5_9cc1_56bad886cc5d.slice/crio-f163e6213b3d57c99dd1db220232a2c77de72366b3470c0193a93aa189518b75 WatchSource:0}: Error finding container f163e6213b3d57c99dd1db220232a2c77de72366b3470c0193a93aa189518b75: Status 404 returned error can't find the container with id f163e6213b3d57c99dd1db220232a2c77de72366b3470c0193a93aa189518b75 Sep 17 23:10:09.998467 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:09.998439 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:09.998614 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:09.998598 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 23:10:09.998679 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:09.998617 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd9896dfd-2sbbg: secret "image-registry-tls" not found Sep 17 23:10:09.998679 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:09.998677 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls podName:5acef9f4-6131-4772-abab-596a3d81af48 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:10.998657341 +0000 UTC m=+249.932564454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls") pod "image-registry-7fd9896dfd-2sbbg" (UID: "5acef9f4-6131-4772-abab-596a3d81af48") : secret "image-registry-tls" not found Sep 17 23:10:10.016981 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:10.016959 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jwjpz"] Sep 17 23:10:10.019954 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:10:10.019934 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7efbb43_0384_4fed_ac1c_4f8664a12e72.slice/crio-d3948facbefbf59b400562f2cf48fc833c556f9a5ffdba5ed49388a7bdb4b8ba WatchSource:0}: Error finding container d3948facbefbf59b400562f2cf48fc833c556f9a5ffdba5ed49388a7bdb4b8ba: Status 404 returned error can't find the container with id d3948facbefbf59b400562f2cf48fc833c556f9a5ffdba5ed49388a7bdb4b8ba Sep 17 23:10:10.098708 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:10.098682 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-mmq2h\" (UID: \"6af371a5-0c1e-484c-a3bc-abc43ab5bb93\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h" Sep 17 23:10:10.098823 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:10.098811 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 23:10:10.098872 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:10.098860 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-samples-operator-tls podName:6af371a5-0c1e-484c-a3bc-abc43ab5bb93 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:11.098847475 +0000 UTC m=+250.032754588 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-mmq2h" (UID: "6af371a5-0c1e-484c-a3bc-abc43ab5bb93") : secret "samples-operator-tls" not found Sep 17 23:10:10.147161 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:10.147128 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jwjpz" event={"ID":"c7efbb43-0384-4fed-ac1c-4f8664a12e72","Type":"ContainerStarted","Data":"d3948facbefbf59b400562f2cf48fc833c556f9a5ffdba5ed49388a7bdb4b8ba"} Sep 17 23:10:10.148150 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:10.148130 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-d6px4" event={"ID":"9ff60ca3-2533-4ed5-9cc1-56bad886cc5d","Type":"ContainerStarted","Data":"f163e6213b3d57c99dd1db220232a2c77de72366b3470c0193a93aa189518b75"} Sep 17 23:10:11.005714 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:11.005676 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:11.006121 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:11.005825 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 23:10:11.006121 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:11.005840 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd9896dfd-2sbbg: secret "image-registry-tls" not found Sep 17 23:10:11.006121 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:11.005907 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls podName:5acef9f4-6131-4772-abab-596a3d81af48 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:13.005887684 +0000 UTC m=+251.939794823 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls") pod "image-registry-7fd9896dfd-2sbbg" (UID: "5acef9f4-6131-4772-abab-596a3d81af48") : secret "image-registry-tls" not found Sep 17 23:10:11.106128 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:11.106100 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-mmq2h\" (UID: \"6af371a5-0c1e-484c-a3bc-abc43ab5bb93\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h" Sep 17 23:10:11.106237 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:11.106194 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 23:10:11.106295 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:11.106286 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-samples-operator-tls podName:6af371a5-0c1e-484c-a3bc-abc43ab5bb93 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:13.106265432 +0000 UTC m=+252.040172591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-mmq2h" (UID: "6af371a5-0c1e-484c-a3bc-abc43ab5bb93") : secret "samples-operator-tls" not found Sep 17 23:10:12.162766 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:12.162726 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-d6px4" event={"ID":"9ff60ca3-2533-4ed5-9cc1-56bad886cc5d","Type":"ContainerStarted","Data":"9953ce7c789dad7f2c34b3813d2906997aa265ecb98f7f8fb9d1a6d9cd12286d"} Sep 17 23:10:12.164227 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:12.164203 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jwjpz" event={"ID":"c7efbb43-0384-4fed-ac1c-4f8664a12e72","Type":"ContainerStarted","Data":"a313968c4156ae4f49d3c9200b7ac578f8482f21ea7b81abe2c9581e9467c99d"} Sep 17 23:10:12.191949 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:12.191898 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-d6px4" podStartSLOduration=1.486441151 podStartE2EDuration="3.191880384s" totalImagesPullingTime="1.705439233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:12.177427436 +0000 UTC m=+251.111334570" watchObservedRunningTime="2026-09-17 23:10:12.191880384 +0000 UTC m=+251.125787520" Sep 17 23:10:12.192105 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:12.192082 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jwjpz" podStartSLOduration=2.114446344 podStartE2EDuration="3.192073442s" totalImagesPullingTime="1.077627098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:12.190626936 +0000 UTC m=+251.124534072" watchObservedRunningTime="2026-09-17 23:10:12.192073442 +0000 UTC m=+251.125980579" Sep 17 23:10:12.514214 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:12.514183 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-qkpcn"] Sep 17 23:10:12.517309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:12.517290 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qkpcn" Sep 17 23:10:12.519986 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:12.519967 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 17 23:10:12.519986 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:12.519975 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-b2rl9\"" Sep 17 23:10:12.520151 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:12.520002 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 17 23:10:12.525090 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:12.525025 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-qkpcn"] Sep 17 23:10:12.617243 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:12.617212 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sswnt\" (UniqueName: \"kubernetes.io/projected/3a4ed9d0-4126-4045-92c6-d6162c3aa45e-kube-api-access-sswnt\") pod \"migrator-76794cbd6c-qkpcn\" (UID: \"3a4ed9d0-4126-4045-92c6-d6162c3aa45e\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qkpcn" Sep 17 23:10:12.717806 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:12.717764 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sswnt\" (UniqueName: \"kubernetes.io/projected/3a4ed9d0-4126-4045-92c6-d6162c3aa45e-kube-api-access-sswnt\") pod \"migrator-76794cbd6c-qkpcn\" (UID: \"3a4ed9d0-4126-4045-92c6-d6162c3aa45e\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qkpcn" Sep 17 23:10:12.724809 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:12.724788 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sswnt\" (UniqueName: \"kubernetes.io/projected/3a4ed9d0-4126-4045-92c6-d6162c3aa45e-kube-api-access-sswnt\") pod \"migrator-76794cbd6c-qkpcn\" (UID: \"3a4ed9d0-4126-4045-92c6-d6162c3aa45e\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qkpcn" Sep 17 23:10:12.826869 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:12.826804 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qkpcn" Sep 17 23:10:13.020140 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:13.020101 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:13.020279 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:13.020227 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 23:10:13.020279 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:13.020240 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd9896dfd-2sbbg: secret "image-registry-tls" not found Sep 17 23:10:13.020350 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:13.020288 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls podName:5acef9f4-6131-4772-abab-596a3d81af48 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:17.020275807 +0000 UTC m=+255.954182920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls") pod "image-registry-7fd9896dfd-2sbbg" (UID: "5acef9f4-6131-4772-abab-596a3d81af48") : secret "image-registry-tls" not found Sep 17 23:10:13.092388 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:13.092329 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-qkpcn"] Sep 17 23:10:13.095544 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:10:13.095518 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a4ed9d0_4126_4045_92c6_d6162c3aa45e.slice/crio-1369e0e86457c51a783fbb5cff3583583ed7715fa278e45454049eec68863537 WatchSource:0}: Error finding container 1369e0e86457c51a783fbb5cff3583583ed7715fa278e45454049eec68863537: Status 404 returned error can't find the container with id 1369e0e86457c51a783fbb5cff3583583ed7715fa278e45454049eec68863537 Sep 17 23:10:13.121059 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:13.121027 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-mmq2h\" (UID: \"6af371a5-0c1e-484c-a3bc-abc43ab5bb93\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h" Sep 17 23:10:13.121202 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:13.121186 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 23:10:13.121268 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:13.121251 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-samples-operator-tls podName:6af371a5-0c1e-484c-a3bc-abc43ab5bb93 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:17.121231528 +0000 UTC m=+256.055138655 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-mmq2h" (UID: "6af371a5-0c1e-484c-a3bc-abc43ab5bb93") : secret "samples-operator-tls" not found Sep 17 23:10:13.167816 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:13.167788 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qkpcn" event={"ID":"3a4ed9d0-4126-4045-92c6-d6162c3aa45e","Type":"ContainerStarted","Data":"1369e0e86457c51a783fbb5cff3583583ed7715fa278e45454049eec68863537"} Sep 17 23:10:14.027419 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.027378 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs\") pod \"network-metrics-daemon-xg2sz\" (UID: \"3752b568-d53f-49ac-9095-0f61cee96767\") " pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:10:14.027583 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:14.027538 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 23:10:14.027648 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:14.027596 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs podName:3752b568-d53f-49ac-9095-0f61cee96767 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:16.027579286 +0000 UTC m=+374.961486414 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs") pod "network-metrics-daemon-xg2sz" (UID: "3752b568-d53f-49ac-9095-0f61cee96767") : secret "metrics-daemon-secret" not found Sep 17 23:10:14.171407 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.171379 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qkpcn" event={"ID":"3a4ed9d0-4126-4045-92c6-d6162c3aa45e","Type":"ContainerStarted","Data":"b949fa65a953d34184e1a717dc43acc21861edb9fb4a8fcf6a6b2a9ab5afaf77"} Sep 17 23:10:14.171690 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.171429 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qkpcn" event={"ID":"3a4ed9d0-4126-4045-92c6-d6162c3aa45e","Type":"ContainerStarted","Data":"b14ef5025d278f5fd4be712cf18a397052f27f332589dc9ec9c359736641fd61"} Sep 17 23:10:14.185782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.185743 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qkpcn" podStartSLOduration=1.240280351 podStartE2EDuration="2.185729867s" totalImagesPullingTime="945.449516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:14.185462708 +0000 UTC m=+253.119369842" watchObservedRunningTime="2026-09-17 23:10:14.185729867 +0000 UTC m=+253.119637043" Sep 17 23:10:14.616740 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.616710 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4nj87"] Sep 17 23:10:14.621965 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.621938 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:14.624070 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.624042 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 17 23:10:14.624200 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.624138 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qmjfd\"" Sep 17 23:10:14.624761 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.624743 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 17 23:10:14.624847 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.624770 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 17 23:10:14.624847 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.624784 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 17 23:10:14.628989 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.628957 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4nj87"] Sep 17 23:10:14.730738 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.730715 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f26392a-99d2-4431-928c-30bf72268393-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:14.730932 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.730758 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f26392a-99d2-4431-928c-30bf72268393-data-volume\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:14.730932 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.730810 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-428kf\" (UniqueName: \"kubernetes.io/projected/2f26392a-99d2-4431-928c-30bf72268393-kube-api-access-428kf\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:14.730932 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.730856 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:14.730932 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.730902 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2f26392a-99d2-4431-928c-30bf72268393-crio-socket\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:14.831762 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.831736 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-428kf\" (UniqueName: \"kubernetes.io/projected/2f26392a-99d2-4431-928c-30bf72268393-kube-api-access-428kf\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:14.831762 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.831764 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:14.831946 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.831793 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2f26392a-99d2-4431-928c-30bf72268393-crio-socket\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:14.831946 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.831845 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f26392a-99d2-4431-928c-30bf72268393-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:14.831946 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:14.831892 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 23:10:14.831946 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.831900 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f26392a-99d2-4431-928c-30bf72268393-data-volume\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:14.831946 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:14.831949 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls podName:2f26392a-99d2-4431-928c-30bf72268393 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:15.331932111 +0000 UTC m=+254.265839233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4nj87" (UID: "2f26392a-99d2-4431-928c-30bf72268393") : secret "insights-runtime-extractor-tls" not found Sep 17 23:10:14.832174 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:14.831952 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 23:10:14.832174 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:14.832008 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f26392a-99d2-4431-928c-30bf72268393-kube-rbac-proxy-cm podName:2f26392a-99d2-4431-928c-30bf72268393 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:15.331995312 +0000 UTC m=+254.265902425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2f26392a-99d2-4431-928c-30bf72268393-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4nj87" (UID: "2f26392a-99d2-4431-928c-30bf72268393") : configmap "kube-rbac-proxy" not found Sep 17 23:10:14.832174 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.831947 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2f26392a-99d2-4431-928c-30bf72268393-crio-socket\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:14.832266 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.832209 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f26392a-99d2-4431-928c-30bf72268393-data-volume\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:14.839658 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:14.839635 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-428kf\" (UniqueName: \"kubernetes.io/projected/2f26392a-99d2-4431-928c-30bf72268393-kube-api-access-428kf\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:15.336434 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:15.336402 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f26392a-99d2-4431-928c-30bf72268393-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:15.336805 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:15.336462 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:15.336805 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:15.336536 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 23:10:15.336805 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:15.336545 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 23:10:15.336805 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:15.336605 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f26392a-99d2-4431-928c-30bf72268393-kube-rbac-proxy-cm podName:2f26392a-99d2-4431-928c-30bf72268393 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:16.336585827 +0000 UTC m=+255.270492960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2f26392a-99d2-4431-928c-30bf72268393-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4nj87" (UID: "2f26392a-99d2-4431-928c-30bf72268393") : configmap "kube-rbac-proxy" not found Sep 17 23:10:15.336805 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:15.336629 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls podName:2f26392a-99d2-4431-928c-30bf72268393 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:16.336617875 +0000 UTC m=+255.270524999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4nj87" (UID: "2f26392a-99d2-4431-928c-30bf72268393") : secret "insights-runtime-extractor-tls" not found Sep 17 23:10:16.343018 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:16.342983 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f26392a-99d2-4431-928c-30bf72268393-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:16.343369 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:16.343047 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:16.343369 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:16.343097 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 23:10:16.343369 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:16.343139 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 23:10:16.343369 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:16.343158 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f26392a-99d2-4431-928c-30bf72268393-kube-rbac-proxy-cm podName:2f26392a-99d2-4431-928c-30bf72268393 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:18.343143693 +0000 UTC m=+257.277050818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2f26392a-99d2-4431-928c-30bf72268393-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4nj87" (UID: "2f26392a-99d2-4431-928c-30bf72268393") : configmap "kube-rbac-proxy" not found Sep 17 23:10:16.343369 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:16.343186 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls podName:2f26392a-99d2-4431-928c-30bf72268393 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:18.34317475 +0000 UTC m=+257.277081863 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4nj87" (UID: "2f26392a-99d2-4431-928c-30bf72268393") : secret "insights-runtime-extractor-tls" not found Sep 17 23:10:16.580389 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:16.580361 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k2bg2_bbda2aea-1bb6-4a24-a74f-fc8d79b07586/dns-node-resolver/0.log" Sep 17 23:10:17.047898 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:17.047856 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:17.048054 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:17.047971 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 23:10:17.048054 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:17.047984 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd9896dfd-2sbbg: secret "image-registry-tls" not found Sep 17 23:10:17.048054 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:17.048043 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls podName:5acef9f4-6131-4772-abab-596a3d81af48 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:25.048026686 +0000 UTC m=+263.981933815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls") pod "image-registry-7fd9896dfd-2sbbg" (UID: "5acef9f4-6131-4772-abab-596a3d81af48") : secret "image-registry-tls" not found Sep 17 23:10:17.148592 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:17.148564 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-mmq2h\" (UID: \"6af371a5-0c1e-484c-a3bc-abc43ab5bb93\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h" Sep 17 23:10:17.148719 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:17.148703 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 23:10:17.148771 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:17.148762 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-samples-operator-tls podName:6af371a5-0c1e-484c-a3bc-abc43ab5bb93 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:25.148747221 +0000 UTC m=+264.082654340 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-mmq2h" (UID: "6af371a5-0c1e-484c-a3bc-abc43ab5bb93") : secret "samples-operator-tls" not found Sep 17 23:10:17.780396 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:17.780372 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-trs6m_bdbecc0a-d250-434e-8478-2f6f57c094d3/node-ca/0.log" Sep 17 23:10:18.181307 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:18.181228 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kk5m4_48bdac6b-06a2-4d13-8a48-c0d5b5af03b5/serve-healthcheck-canary/0.log" Sep 17 23:10:18.358330 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:18.358297 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f26392a-99d2-4431-928c-30bf72268393-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:18.358560 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:18.358397 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:18.358560 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:18.358408 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 23:10:18.358560 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:18.358465 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f26392a-99d2-4431-928c-30bf72268393-kube-rbac-proxy-cm podName:2f26392a-99d2-4431-928c-30bf72268393 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:22.358448422 +0000 UTC m=+261.292355535 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2f26392a-99d2-4431-928c-30bf72268393-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4nj87" (UID: "2f26392a-99d2-4431-928c-30bf72268393") : configmap "kube-rbac-proxy" not found Sep 17 23:10:18.358560 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:18.358555 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 23:10:18.358752 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:18.358615 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls podName:2f26392a-99d2-4431-928c-30bf72268393 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:22.358598782 +0000 UTC m=+261.292505906 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4nj87" (UID: "2f26392a-99d2-4431-928c-30bf72268393") : secret "insights-runtime-extractor-tls" not found Sep 17 23:10:22.389034 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:22.388997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:22.389465 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:22.389095 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f26392a-99d2-4431-928c-30bf72268393-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:22.389465 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:22.389197 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 23:10:22.389465 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:22.389240 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 23:10:22.389465 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:22.389258 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls podName:2f26392a-99d2-4431-928c-30bf72268393 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:30.389243128 +0000 UTC m=+269.323150255 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4nj87" (UID: "2f26392a-99d2-4431-928c-30bf72268393") : secret "insights-runtime-extractor-tls" not found Sep 17 23:10:22.389465 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:22.389305 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2f26392a-99d2-4431-928c-30bf72268393-kube-rbac-proxy-cm podName:2f26392a-99d2-4431-928c-30bf72268393 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:30.389287151 +0000 UTC m=+269.323194289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2f26392a-99d2-4431-928c-30bf72268393-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4nj87" (UID: "2f26392a-99d2-4431-928c-30bf72268393") : configmap "kube-rbac-proxy" not found Sep 17 23:10:25.108899 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:25.108864 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:25.111102 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:25.111084 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls\") pod \"image-registry-7fd9896dfd-2sbbg\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:25.210109 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:25.210085 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-mmq2h\" (UID: \"6af371a5-0c1e-484c-a3bc-abc43ab5bb93\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h" Sep 17 23:10:25.212217 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:25.212191 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6af371a5-0c1e-484c-a3bc-abc43ab5bb93-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-mmq2h\" (UID: \"6af371a5-0c1e-484c-a3bc-abc43ab5bb93\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h" Sep 17 23:10:25.271264 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:25.271242 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:25.361139 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:25.361073 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h" Sep 17 23:10:25.538620 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:25.538593 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fd9896dfd-2sbbg"] Sep 17 23:10:25.541470 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:10:25.541444 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5acef9f4_6131_4772_abab_596a3d81af48.slice/crio-19fe4fd0229b0cba58e5515a20585513ad8326d42a96a6ab1a269855da32aa91 WatchSource:0}: Error finding container 19fe4fd0229b0cba58e5515a20585513ad8326d42a96a6ab1a269855da32aa91: Status 404 returned error can't find the container with id 19fe4fd0229b0cba58e5515a20585513ad8326d42a96a6ab1a269855da32aa91 Sep 17 23:10:25.626735 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:25.626684 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h"] Sep 17 23:10:26.207554 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:26.207516 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" event={"ID":"5acef9f4-6131-4772-abab-596a3d81af48","Type":"ContainerStarted","Data":"2061424c4323dd5bb66c46fdb1e015efc49852fa9d9e230bf84f75510868b6e2"} Sep 17 23:10:26.207999 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:26.207571 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" event={"ID":"5acef9f4-6131-4772-abab-596a3d81af48","Type":"ContainerStarted","Data":"19fe4fd0229b0cba58e5515a20585513ad8326d42a96a6ab1a269855da32aa91"} Sep 17 23:10:26.207999 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:26.207597 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:10:26.208528 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:26.208493 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h" event={"ID":"6af371a5-0c1e-484c-a3bc-abc43ab5bb93","Type":"ContainerStarted","Data":"7cced73e9bc721d50b98a65e7fb8fa6f64250f33b4ee903448cecfdb636a9c3d"} Sep 17 23:10:26.225408 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:26.225363 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" podStartSLOduration=17.225343927 podStartE2EDuration="17.225343927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:26.22438107 +0000 UTC m=+265.158288204" watchObservedRunningTime="2026-09-17 23:10:26.225343927 +0000 UTC m=+265.159251412" Sep 17 23:10:27.213320 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:27.213294 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h" event={"ID":"6af371a5-0c1e-484c-a3bc-abc43ab5bb93","Type":"ContainerStarted","Data":"52f9fcd6c98cdba327a730a5a8d63ce1c705e03a854821a03b6cd09d04a2eb38"} Sep 17 23:10:28.217628 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:28.217597 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h" event={"ID":"6af371a5-0c1e-484c-a3bc-abc43ab5bb93","Type":"ContainerStarted","Data":"fc89fbc56521ae69210a3db1109f667ce957d0a9f5812cc62e483337b3cbb1c3"} Sep 17 23:10:28.234047 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:28.234001 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-mmq2h" podStartSLOduration=17.787969204 podStartE2EDuration="19.23398781s" totalImagesPullingTime="1.446018606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:28.232871451 +0000 UTC m=+267.166778586" watchObservedRunningTime="2026-09-17 23:10:28.23398781 +0000 UTC m=+267.167894943" Sep 17 23:10:30.450630 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:30.450595 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f26392a-99d2-4431-928c-30bf72268393-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:30.451004 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:30.450649 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:30.451004 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:30.450752 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 23:10:30.451004 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:30.450815 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls podName:2f26392a-99d2-4431-928c-30bf72268393 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:46.450798639 +0000 UTC m=+285.384705754 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4nj87" (UID: "2f26392a-99d2-4431-928c-30bf72268393") : secret "insights-runtime-extractor-tls" not found Sep 17 23:10:30.451127 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:30.451116 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2f26392a-99d2-4431-928c-30bf72268393-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:37.894973 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:37.894937 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-gjxw7"] Sep 17 23:10:37.898131 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:37.898108 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-gjxw7" Sep 17 23:10:37.900354 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:37.900330 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 17 23:10:37.900444 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:37.900409 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-lv5dh\"" Sep 17 23:10:37.900578 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:37.900562 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 17 23:10:37.909032 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:37.909012 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-gjxw7"] Sep 17 23:10:37.914336 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:37.914317 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fd9896dfd-2sbbg"] Sep 17 23:10:37.949974 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:37.949953 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cb7955849-x2x4q"] Sep 17 23:10:37.951913 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:37.951900 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:37.965424 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:37.965399 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cb7955849-x2x4q"] Sep 17 23:10:38.001119 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.001096 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqc92\" (UniqueName: \"kubernetes.io/projected/19bd3f81-d2c5-4cfd-bb84-c3608eaaa14b-kube-api-access-tqc92\") pod \"downloads-6594f4c75-gjxw7\" (UID: \"19bd3f81-d2c5-4cfd-bb84-c3608eaaa14b\") " pod="openshift-console/downloads-6594f4c75-gjxw7" Sep 17 23:10:38.001206 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.001124 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec6eaa22-50b5-4788-a5e8-b8e783034759-registry-certificates\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.001206 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.001196 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec6eaa22-50b5-4788-a5e8-b8e783034759-installation-pull-secrets\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.001276 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.001227 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec6eaa22-50b5-4788-a5e8-b8e783034759-ca-trust-extracted\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.001276 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.001244 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw6gn\" (UniqueName: \"kubernetes.io/projected/ec6eaa22-50b5-4788-a5e8-b8e783034759-kube-api-access-fw6gn\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.001340 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.001282 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec6eaa22-50b5-4788-a5e8-b8e783034759-trusted-ca\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.001340 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.001320 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec6eaa22-50b5-4788-a5e8-b8e783034759-image-registry-private-configuration\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.001340 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.001337 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec6eaa22-50b5-4788-a5e8-b8e783034759-registry-tls\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.001440 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.001352 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec6eaa22-50b5-4788-a5e8-b8e783034759-bound-sa-token\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.101856 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.101835 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec6eaa22-50b5-4788-a5e8-b8e783034759-installation-pull-secrets\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.101956 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.101861 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec6eaa22-50b5-4788-a5e8-b8e783034759-ca-trust-extracted\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.101956 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.101878 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw6gn\" (UniqueName: \"kubernetes.io/projected/ec6eaa22-50b5-4788-a5e8-b8e783034759-kube-api-access-fw6gn\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.101956 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.101905 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec6eaa22-50b5-4788-a5e8-b8e783034759-trusted-ca\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.102078 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.102006 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec6eaa22-50b5-4788-a5e8-b8e783034759-image-registry-private-configuration\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.102078 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.102040 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec6eaa22-50b5-4788-a5e8-b8e783034759-registry-tls\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.102078 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.102062 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec6eaa22-50b5-4788-a5e8-b8e783034759-bound-sa-token\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.102220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.102176 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqc92\" (UniqueName: \"kubernetes.io/projected/19bd3f81-d2c5-4cfd-bb84-c3608eaaa14b-kube-api-access-tqc92\") pod \"downloads-6594f4c75-gjxw7\" (UID: \"19bd3f81-d2c5-4cfd-bb84-c3608eaaa14b\") " pod="openshift-console/downloads-6594f4c75-gjxw7" Sep 17 23:10:38.102271 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.102216 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec6eaa22-50b5-4788-a5e8-b8e783034759-registry-certificates\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.102271 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.102233 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec6eaa22-50b5-4788-a5e8-b8e783034759-ca-trust-extracted\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.102804 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.102782 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec6eaa22-50b5-4788-a5e8-b8e783034759-trusted-ca\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.103030 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.103007 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec6eaa22-50b5-4788-a5e8-b8e783034759-registry-certificates\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.104417 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.104396 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec6eaa22-50b5-4788-a5e8-b8e783034759-image-registry-private-configuration\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.104516 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.104459 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec6eaa22-50b5-4788-a5e8-b8e783034759-registry-tls\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.104611 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.104590 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec6eaa22-50b5-4788-a5e8-b8e783034759-installation-pull-secrets\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.110259 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.110217 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqc92\" (UniqueName: \"kubernetes.io/projected/19bd3f81-d2c5-4cfd-bb84-c3608eaaa14b-kube-api-access-tqc92\") pod \"downloads-6594f4c75-gjxw7\" (UID: \"19bd3f81-d2c5-4cfd-bb84-c3608eaaa14b\") " pod="openshift-console/downloads-6594f4c75-gjxw7" Sep 17 23:10:38.110366 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.110268 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec6eaa22-50b5-4788-a5e8-b8e783034759-bound-sa-token\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.110810 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.110793 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw6gn\" (UniqueName: \"kubernetes.io/projected/ec6eaa22-50b5-4788-a5e8-b8e783034759-kube-api-access-fw6gn\") pod \"image-registry-cb7955849-x2x4q\" (UID: \"ec6eaa22-50b5-4788-a5e8-b8e783034759\") " pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.206673 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.206633 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-gjxw7" Sep 17 23:10:38.259803 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.259779 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:38.471450 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.471404 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-gjxw7"] Sep 17 23:10:38.531265 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:38.531237 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cb7955849-x2x4q"] Sep 17 23:10:38.534556 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:10:38.534532 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6eaa22_50b5_4788_a5e8_b8e783034759.slice/crio-5ec6aebcf43680a19a56d69345d8a995a17b59abb3901118f699095c730216d1 WatchSource:0}: Error finding container 5ec6aebcf43680a19a56d69345d8a995a17b59abb3901118f699095c730216d1: Status 404 returned error can't find the container with id 5ec6aebcf43680a19a56d69345d8a995a17b59abb3901118f699095c730216d1 Sep 17 23:10:39.250014 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:39.249982 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb7955849-x2x4q" event={"ID":"ec6eaa22-50b5-4788-a5e8-b8e783034759","Type":"ContainerStarted","Data":"ef01327c7ea73e8a2c35cba24534874a61abf1f543904c3dcfacde4a6e96e02f"} Sep 17 23:10:39.250453 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:39.250038 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb7955849-x2x4q" event={"ID":"ec6eaa22-50b5-4788-a5e8-b8e783034759","Type":"ContainerStarted","Data":"5ec6aebcf43680a19a56d69345d8a995a17b59abb3901118f699095c730216d1"} Sep 17 23:10:39.250453 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:39.250110 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:10:39.251820 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:39.251797 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-gjxw7" event={"ID":"19bd3f81-d2c5-4cfd-bb84-c3608eaaa14b","Type":"ContainerStarted","Data":"9d15ae6de65d63a717a0765cd4ba049c98070e0dadf14231a6a83efc0f76fe82"} Sep 17 23:10:39.269342 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:39.269298 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-cb7955849-x2x4q" podStartSLOduration=2.269285784 podStartE2EDuration="2.269285784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:39.268280827 +0000 UTC m=+278.202187963" watchObservedRunningTime="2026-09-17 23:10:39.269285784 +0000 UTC m=+278.203192910" Sep 17 23:10:41.934818 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:41.934770 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pkh9l" podUID="5cd06380-e744-4e37-a5f9-3cac49aee5d4" Sep 17 23:10:42.262898 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:42.262862 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pkh9l" Sep 17 23:10:44.964046 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:44.964002 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:10:44.966734 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:44.966710 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cd06380-e744-4e37-a5f9-3cac49aee5d4-metrics-tls\") pod \"dns-default-pkh9l\" (UID: \"5cd06380-e744-4e37-a5f9-3cac49aee5d4\") " pod="openshift-dns/dns-default-pkh9l" Sep 17 23:10:45.266195 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:45.266166 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7hp2b\"" Sep 17 23:10:45.274063 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:45.274040 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pkh9l" Sep 17 23:10:45.554682 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:45.554523 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pkh9l"] Sep 17 23:10:45.556878 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:10:45.556850 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd06380_e744_4e37_a5f9_3cac49aee5d4.slice/crio-96e5ec9ab6fe2aaee4f881a669d31d8fa4955fe6f0e27070c57e56568a589d53 WatchSource:0}: Error finding container 96e5ec9ab6fe2aaee4f881a669d31d8fa4955fe6f0e27070c57e56568a589d53: Status 404 returned error can't find the container with id 96e5ec9ab6fe2aaee4f881a669d31d8fa4955fe6f0e27070c57e56568a589d53 Sep 17 23:10:46.279459 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:46.279415 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pkh9l" event={"ID":"5cd06380-e744-4e37-a5f9-3cac49aee5d4","Type":"ContainerStarted","Data":"96e5ec9ab6fe2aaee4f881a669d31d8fa4955fe6f0e27070c57e56568a589d53"} Sep 17 23:10:46.476964 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:46.476924 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:46.480217 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:46.480056 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2f26392a-99d2-4431-928c-30bf72268393-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4nj87\" (UID: \"2f26392a-99d2-4431-928c-30bf72268393\") " pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:46.731017 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:46.730980 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4nj87" Sep 17 23:10:47.477818 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:47.477745 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4nj87"] Sep 17 23:10:47.480965 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:10:47.480941 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f26392a_99d2_4431_928c_30bf72268393.slice/crio-c1d512604237ed7bbb482680be6d04ecbca293eb7bd794a0e05150d003ce2f4f WatchSource:0}: Error finding container c1d512604237ed7bbb482680be6d04ecbca293eb7bd794a0e05150d003ce2f4f: Status 404 returned error can't find the container with id c1d512604237ed7bbb482680be6d04ecbca293eb7bd794a0e05150d003ce2f4f Sep 17 23:10:47.920551 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:47.920514 2582 patch_prober.go:28] interesting pod/image-registry-7fd9896dfd-2sbbg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 17 23:10:47.920791 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:47.920576 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" podUID="5acef9f4-6131-4772-abab-596a3d81af48" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 23:10:48.289306 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:48.289269 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4nj87" event={"ID":"2f26392a-99d2-4431-928c-30bf72268393","Type":"ContainerStarted","Data":"479c3aa819f6d5b77480ceed6279c8825c9b98159ae028e1bf35fe876356366f"} Sep 17 23:10:48.289462 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:48.289337 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4nj87" event={"ID":"2f26392a-99d2-4431-928c-30bf72268393","Type":"ContainerStarted","Data":"c1fe71ca81d50f0849dd2ab76edf5c0af40e372000375bba5a676f8a31a45a73"} Sep 17 23:10:48.289462 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:48.289361 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4nj87" event={"ID":"2f26392a-99d2-4431-928c-30bf72268393","Type":"ContainerStarted","Data":"c1d512604237ed7bbb482680be6d04ecbca293eb7bd794a0e05150d003ce2f4f"} Sep 17 23:10:48.291366 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:48.291337 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pkh9l" event={"ID":"5cd06380-e744-4e37-a5f9-3cac49aee5d4","Type":"ContainerStarted","Data":"4759fae1ce6615c1d653a78714405f3adbde362200e6968247297781829d07a9"} Sep 17 23:10:48.291479 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:48.291390 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pkh9l" event={"ID":"5cd06380-e744-4e37-a5f9-3cac49aee5d4","Type":"ContainerStarted","Data":"ad1acd71fa0238dbb7eba88f0a3d489aa17b9758dc09bbea9d5c3c6c239accf3"} Sep 17 23:10:48.291581 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:48.291483 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pkh9l" Sep 17 23:10:48.306704 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:48.306660 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pkh9l" podStartSLOduration=251.617259335 podStartE2EDuration="4m13.30664573s" totalImagesPullingTime="1.689386395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:48.306089477 +0000 UTC m=+287.239996614" watchObservedRunningTime="2026-09-17 23:10:48.30664573 +0000 UTC m=+287.240552864" Sep 17 23:10:54.288336 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:10:54.288289 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xg2sz" podUID="3752b568-d53f-49ac-9095-0f61cee96767" Sep 17 23:10:54.362715 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.362680 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-t9g4f"] Sep 17 23:10:54.365805 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.365785 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.368963 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.368804 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 17 23:10:54.368963 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.368817 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 17 23:10:54.369206 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.369088 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 17 23:10:54.369206 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.369156 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 17 23:10:54.369206 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.369188 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 17 23:10:54.369356 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.369282 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rvdxr\"" Sep 17 23:10:54.369440 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.369417 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 17 23:10:54.369582 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.369462 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-trds7"] Sep 17 23:10:54.374515 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.373276 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.375645 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.375613 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 17 23:10:54.375749 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.375614 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n6f65\"" Sep 17 23:10:54.376085 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.376063 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 17 23:10:54.378722 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.378702 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 17 23:10:54.386991 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.386971 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-t9g4f"] Sep 17 23:10:54.546120 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.546033 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7971320f-7694-45b5-8fe4-964eb59d081e-node-exporter-tls\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.546120 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.546080 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6c946e8-c084-434e-b49d-7ab905a4be36-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-t9g4f\" (UID: \"e6c946e8-c084-434e-b49d-7ab905a4be36\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.546120 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.546114 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7971320f-7694-45b5-8fe4-964eb59d081e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.546339 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.546161 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7971320f-7694-45b5-8fe4-964eb59d081e-node-exporter-textfile\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.546339 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.546216 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fl5v\" (UniqueName: \"kubernetes.io/projected/7971320f-7694-45b5-8fe4-964eb59d081e-kube-api-access-4fl5v\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.546339 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.546246 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7971320f-7694-45b5-8fe4-964eb59d081e-sys\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.546339 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.546272 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7971320f-7694-45b5-8fe4-964eb59d081e-node-exporter-wtmp\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.546339 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.546298 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e6c946e8-c084-434e-b49d-7ab905a4be36-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-t9g4f\" (UID: \"e6c946e8-c084-434e-b49d-7ab905a4be36\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.546339 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.546328 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6c946e8-c084-434e-b49d-7ab905a4be36-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-t9g4f\" (UID: \"e6c946e8-c084-434e-b49d-7ab905a4be36\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.546587 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.546366 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7971320f-7694-45b5-8fe4-964eb59d081e-node-exporter-accelerators-collector-config\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.546587 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.546422 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7971320f-7694-45b5-8fe4-964eb59d081e-root\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.546587 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.546455 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7s6g\" (UniqueName: \"kubernetes.io/projected/e6c946e8-c084-434e-b49d-7ab905a4be36-kube-api-access-h7s6g\") pod \"kube-state-metrics-84c885658c-t9g4f\" (UID: \"e6c946e8-c084-434e-b49d-7ab905a4be36\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.546587 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.546481 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7971320f-7694-45b5-8fe4-964eb59d081e-metrics-client-ca\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.546587 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.546537 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6c946e8-c084-434e-b49d-7ab905a4be36-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-t9g4f\" (UID: \"e6c946e8-c084-434e-b49d-7ab905a4be36\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.546587 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.546558 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e6c946e8-c084-434e-b49d-7ab905a4be36-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-t9g4f\" (UID: \"e6c946e8-c084-434e-b49d-7ab905a4be36\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.647649 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.647616 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7971320f-7694-45b5-8fe4-964eb59d081e-node-exporter-accelerators-collector-config\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.647830 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.647659 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7971320f-7694-45b5-8fe4-964eb59d081e-root\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.647830 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.647770 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7s6g\" (UniqueName: \"kubernetes.io/projected/e6c946e8-c084-434e-b49d-7ab905a4be36-kube-api-access-h7s6g\") pod \"kube-state-metrics-84c885658c-t9g4f\" (UID: \"e6c946e8-c084-434e-b49d-7ab905a4be36\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.647830 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.647797 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7971320f-7694-45b5-8fe4-964eb59d081e-root\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.647830 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.647811 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7971320f-7694-45b5-8fe4-964eb59d081e-metrics-client-ca\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.648029 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.647869 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6c946e8-c084-434e-b49d-7ab905a4be36-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-t9g4f\" (UID: \"e6c946e8-c084-434e-b49d-7ab905a4be36\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.648029 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.647901 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e6c946e8-c084-434e-b49d-7ab905a4be36-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-t9g4f\" (UID: \"e6c946e8-c084-434e-b49d-7ab905a4be36\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.648029 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.647937 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7971320f-7694-45b5-8fe4-964eb59d081e-node-exporter-tls\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.648029 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.647964 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6c946e8-c084-434e-b49d-7ab905a4be36-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-t9g4f\" (UID: \"e6c946e8-c084-434e-b49d-7ab905a4be36\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.648205 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.648176 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7971320f-7694-45b5-8fe4-964eb59d081e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.648264 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.648222 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7971320f-7694-45b5-8fe4-964eb59d081e-node-exporter-textfile\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.648325 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.648282 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fl5v\" (UniqueName: \"kubernetes.io/projected/7971320f-7694-45b5-8fe4-964eb59d081e-kube-api-access-4fl5v\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.648325 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.648311 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7971320f-7694-45b5-8fe4-964eb59d081e-node-exporter-accelerators-collector-config\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.648432 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.648341 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7971320f-7694-45b5-8fe4-964eb59d081e-metrics-client-ca\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.648432 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.648318 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7971320f-7694-45b5-8fe4-964eb59d081e-sys\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.648432 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.648374 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7971320f-7694-45b5-8fe4-964eb59d081e-sys\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.648432 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.648395 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7971320f-7694-45b5-8fe4-964eb59d081e-node-exporter-wtmp\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.648653 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.648542 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7971320f-7694-45b5-8fe4-964eb59d081e-node-exporter-wtmp\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.648653 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.648575 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e6c946e8-c084-434e-b49d-7ab905a4be36-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-t9g4f\" (UID: \"e6c946e8-c084-434e-b49d-7ab905a4be36\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.648653 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.648613 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6c946e8-c084-434e-b49d-7ab905a4be36-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-t9g4f\" (UID: \"e6c946e8-c084-434e-b49d-7ab905a4be36\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.648653 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.648622 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7971320f-7694-45b5-8fe4-964eb59d081e-node-exporter-textfile\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.648909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.648881 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e6c946e8-c084-434e-b49d-7ab905a4be36-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-t9g4f\" (UID: \"e6c946e8-c084-434e-b49d-7ab905a4be36\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.648990 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.648925 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e6c946e8-c084-434e-b49d-7ab905a4be36-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-t9g4f\" (UID: \"e6c946e8-c084-434e-b49d-7ab905a4be36\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.649334 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.649310 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6c946e8-c084-434e-b49d-7ab905a4be36-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-t9g4f\" (UID: \"e6c946e8-c084-434e-b49d-7ab905a4be36\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.651054 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.651021 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7971320f-7694-45b5-8fe4-964eb59d081e-node-exporter-tls\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.651167 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.651092 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6c946e8-c084-434e-b49d-7ab905a4be36-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-t9g4f\" (UID: \"e6c946e8-c084-434e-b49d-7ab905a4be36\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.651540 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.651518 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6c946e8-c084-434e-b49d-7ab905a4be36-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-t9g4f\" (UID: \"e6c946e8-c084-434e-b49d-7ab905a4be36\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.651921 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.651901 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7971320f-7694-45b5-8fe4-964eb59d081e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.655645 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.655621 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fl5v\" (UniqueName: \"kubernetes.io/projected/7971320f-7694-45b5-8fe4-964eb59d081e-kube-api-access-4fl5v\") pod \"node-exporter-trds7\" (UID: \"7971320f-7694-45b5-8fe4-964eb59d081e\") " pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:54.655995 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.655974 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7s6g\" (UniqueName: \"kubernetes.io/projected/e6c946e8-c084-434e-b49d-7ab905a4be36-kube-api-access-h7s6g\") pod \"kube-state-metrics-84c885658c-t9g4f\" (UID: \"e6c946e8-c084-434e-b49d-7ab905a4be36\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.682107 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.682081 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" Sep 17 23:10:54.688860 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:54.688838 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-trds7" Sep 17 23:10:55.330669 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:10:55.330640 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7971320f_7694_45b5_8fe4_964eb59d081e.slice/crio-87bd3ad63498ff41c891ce44d7c1496a5282c0de23ef8e94b64f6ae37b3ada32 WatchSource:0}: Error finding container 87bd3ad63498ff41c891ce44d7c1496a5282c0de23ef8e94b64f6ae37b3ada32: Status 404 returned error can't find the container with id 87bd3ad63498ff41c891ce44d7c1496a5282c0de23ef8e94b64f6ae37b3ada32 Sep 17 23:10:55.595957 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:55.595868 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-t9g4f"] Sep 17 23:10:55.599939 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:10:55.599900 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c946e8_c084_434e_b49d_7ab905a4be36.slice/crio-76eeeb1d07124df3401f9dc29e265e3e2714a6ff02337afb03ebf91b18121e6d WatchSource:0}: Error finding container 76eeeb1d07124df3401f9dc29e265e3e2714a6ff02337afb03ebf91b18121e6d: Status 404 returned error can't find the container with id 76eeeb1d07124df3401f9dc29e265e3e2714a6ff02337afb03ebf91b18121e6d Sep 17 23:10:56.326666 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.326622 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-trds7" event={"ID":"7971320f-7694-45b5-8fe4-964eb59d081e","Type":"ContainerStarted","Data":"87bd3ad63498ff41c891ce44d7c1496a5282c0de23ef8e94b64f6ae37b3ada32"} Sep 17 23:10:56.328315 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.328283 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" event={"ID":"e6c946e8-c084-434e-b49d-7ab905a4be36","Type":"ContainerStarted","Data":"76eeeb1d07124df3401f9dc29e265e3e2714a6ff02337afb03ebf91b18121e6d"} Sep 17 23:10:56.330717 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.330692 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-gjxw7" event={"ID":"19bd3f81-d2c5-4cfd-bb84-c3608eaaa14b","Type":"ContainerStarted","Data":"c29fd202bd0afd4c4cfaae20364acf3c07fdc839a71fd25e6815814759cc8ffa"} Sep 17 23:10:56.331229 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.330963 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-gjxw7" Sep 17 23:10:56.335631 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.335607 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7858b594b4-bqbbp"] Sep 17 23:10:56.339464 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.339332 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.347299 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.347127 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 17 23:10:56.347465 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.347450 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 17 23:10:56.347719 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.347682 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 17 23:10:56.347987 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.347874 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 17 23:10:56.348102 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.348085 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 17 23:10:56.348397 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.348278 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-w9t9p\"" Sep 17 23:10:56.348647 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.348458 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-eifu5ig53qush\"" Sep 17 23:10:56.348932 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.348909 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-gjxw7" Sep 17 23:10:56.356839 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.354570 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7858b594b4-bqbbp"] Sep 17 23:10:56.356839 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.356076 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-gjxw7" podStartSLOduration=2.438148451 podStartE2EDuration="19.356057271s" totalImagesPullingTime="16.91790882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:56.354939114 +0000 UTC m=+295.288846250" watchObservedRunningTime="2026-09-17 23:10:56.356057271 +0000 UTC m=+295.289964407" Sep 17 23:10:56.466322 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.466278 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/da235c63-916e-40bd-ab57-ffc8d26f12f9-secret-thanos-querier-tls\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.466479 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.466349 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h8z4\" (UniqueName: \"kubernetes.io/projected/da235c63-916e-40bd-ab57-ffc8d26f12f9-kube-api-access-8h8z4\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.466479 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.466384 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da235c63-916e-40bd-ab57-ffc8d26f12f9-secret-grpc-tls\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.466479 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.466414 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da235c63-916e-40bd-ab57-ffc8d26f12f9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.466479 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.466443 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da235c63-916e-40bd-ab57-ffc8d26f12f9-metrics-client-ca\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.466479 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.466475 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da235c63-916e-40bd-ab57-ffc8d26f12f9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.466776 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.466529 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/da235c63-916e-40bd-ab57-ffc8d26f12f9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.466776 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.466578 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/da235c63-916e-40bd-ab57-ffc8d26f12f9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.567742 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.567703 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/da235c63-916e-40bd-ab57-ffc8d26f12f9-secret-thanos-querier-tls\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.568011 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.567774 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h8z4\" (UniqueName: \"kubernetes.io/projected/da235c63-916e-40bd-ab57-ffc8d26f12f9-kube-api-access-8h8z4\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.568011 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.567805 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da235c63-916e-40bd-ab57-ffc8d26f12f9-secret-grpc-tls\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.568011 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.567844 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da235c63-916e-40bd-ab57-ffc8d26f12f9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.568011 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.567870 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da235c63-916e-40bd-ab57-ffc8d26f12f9-metrics-client-ca\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.568011 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.567901 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da235c63-916e-40bd-ab57-ffc8d26f12f9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.568011 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.567932 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/da235c63-916e-40bd-ab57-ffc8d26f12f9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.568011 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.567977 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/da235c63-916e-40bd-ab57-ffc8d26f12f9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.570188 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.570108 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da235c63-916e-40bd-ab57-ffc8d26f12f9-metrics-client-ca\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.573678 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.573632 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/da235c63-916e-40bd-ab57-ffc8d26f12f9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.574698 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.574653 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da235c63-916e-40bd-ab57-ffc8d26f12f9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.581952 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.581878 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da235c63-916e-40bd-ab57-ffc8d26f12f9-secret-grpc-tls\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.582758 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.582715 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da235c63-916e-40bd-ab57-ffc8d26f12f9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.583304 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.583262 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/da235c63-916e-40bd-ab57-ffc8d26f12f9-secret-thanos-querier-tls\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.583776 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.583754 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/da235c63-916e-40bd-ab57-ffc8d26f12f9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.586005 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.585984 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h8z4\" (UniqueName: \"kubernetes.io/projected/da235c63-916e-40bd-ab57-ffc8d26f12f9-kube-api-access-8h8z4\") pod \"thanos-querier-7858b594b4-bqbbp\" (UID: \"da235c63-916e-40bd-ab57-ffc8d26f12f9\") " pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.653137 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.653089 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:10:56.978659 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:56.978540 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7858b594b4-bqbbp"] Sep 17 23:10:56.986103 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:10:56.986069 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda235c63_916e_40bd_ab57_ffc8d26f12f9.slice/crio-1e32d35ae36fc9d5287da8f725ef0fe675066dc3f69f5fc7eee05d4d2aa3f356 WatchSource:0}: Error finding container 1e32d35ae36fc9d5287da8f725ef0fe675066dc3f69f5fc7eee05d4d2aa3f356: Status 404 returned error can't find the container with id 1e32d35ae36fc9d5287da8f725ef0fe675066dc3f69f5fc7eee05d4d2aa3f356 Sep 17 23:10:57.338813 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:57.338776 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4nj87" event={"ID":"2f26392a-99d2-4431-928c-30bf72268393","Type":"ContainerStarted","Data":"036deae43c3ab0ea3bf851a61c00242e8f74e2c41d0181a4738f4a9513faa09d"} Sep 17 23:10:57.341511 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:57.341356 2582 generic.go:356] "Generic (PLEG): container finished" podID="7971320f-7694-45b5-8fe4-964eb59d081e" containerID="0c75459a6491ea9ce5626bfc7432601849c78bf7a8aed57026168c6e273f2965" exitCode=0 Sep 17 23:10:57.341511 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:57.341442 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-trds7" event={"ID":"7971320f-7694-45b5-8fe4-964eb59d081e","Type":"ContainerDied","Data":"0c75459a6491ea9ce5626bfc7432601849c78bf7a8aed57026168c6e273f2965"} Sep 17 23:10:57.345638 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:57.345555 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" event={"ID":"da235c63-916e-40bd-ab57-ffc8d26f12f9","Type":"ContainerStarted","Data":"1e32d35ae36fc9d5287da8f725ef0fe675066dc3f69f5fc7eee05d4d2aa3f356"} Sep 17 23:10:57.356753 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:57.356698 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4nj87" podStartSLOduration=34.292363096 podStartE2EDuration="43.356681517s" totalImagesPullingTime="9.064318421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:57.355376195 +0000 UTC m=+296.289283353" watchObservedRunningTime="2026-09-17 23:10:57.356681517 +0000 UTC m=+296.290588653" Sep 17 23:10:57.919641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:57.919608 2582 patch_prober.go:28] interesting pod/image-registry-7fd9896dfd-2sbbg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 17 23:10:57.919798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:57.919664 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" podUID="5acef9f4-6131-4772-abab-596a3d81af48" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 23:10:58.266515 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.266383 2582 patch_prober.go:28] interesting pod/image-registry-cb7955849-x2x4q container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 17 23:10:58.266643 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.266542 2582 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-cb7955849-x2x4q" podUID="ec6eaa22-50b5-4788-a5e8-b8e783034759" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 23:10:58.299637 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.299213 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pkh9l" Sep 17 23:10:58.354632 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.354577 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-trds7" event={"ID":"7971320f-7694-45b5-8fe4-964eb59d081e","Type":"ContainerStarted","Data":"ad199c2f6763ca07a48480a20430b7e36fd6765072510f08c54ff0f5c0c813c4"} Sep 17 23:10:58.355038 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.354641 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-trds7" event={"ID":"7971320f-7694-45b5-8fe4-964eb59d081e","Type":"ContainerStarted","Data":"4338af84b28e9ad095be184aa2c57c4e30414bbd478e7db659560801a1d6d85f"} Sep 17 23:10:58.356957 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.356927 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" event={"ID":"e6c946e8-c084-434e-b49d-7ab905a4be36","Type":"ContainerStarted","Data":"045c276f623d0a56dac621a3d77cb241d701b0e971387001091f4010e6fd02e4"} Sep 17 23:10:58.357081 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.356988 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" event={"ID":"e6c946e8-c084-434e-b49d-7ab905a4be36","Type":"ContainerStarted","Data":"4d9a53b6368bf9232f11d62019717e3eb2c04ab331301d4c86508fd9f302d6f3"} Sep 17 23:10:58.754212 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.753999 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-trds7" podStartSLOduration=3.321913907 podStartE2EDuration="4.753975795s" totalImagesPullingTime="1.351640789s" totalInitContainerRuntime="80.421099ms" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:58.38946041 +0000 UTC m=+297.323367558" watchObservedRunningTime="2026-09-17 23:10:58.753975795 +0000 UTC m=+297.687882932" Sep 17 23:10:58.754433 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.754321 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7"] Sep 17 23:10:58.781345 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.781320 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7"] Sep 17 23:10:58.781591 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.781440 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.784397 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.784370 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 17 23:10:58.784677 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.784660 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 17 23:10:58.785628 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.785192 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ml9bk\"" Sep 17 23:10:58.785628 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.785293 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 17 23:10:58.785628 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.785409 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-58vvkldg3p35r\"" Sep 17 23:10:58.785628 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.785556 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 17 23:10:58.888756 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.888719 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1ee2d368-c317-40c4-a001-46103a67a613-client-ca-bundle\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.888949 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.888804 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1ee2d368-c317-40c4-a001-46103a67a613-secret-metrics-server-tls\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.888949 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.888888 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1ee2d368-c317-40c4-a001-46103a67a613-metrics-server-audit-profiles\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.888949 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.888919 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1ee2d368-c317-40c4-a001-46103a67a613-audit-log\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.888949 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.888948 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1ee2d368-c317-40c4-a001-46103a67a613-secret-metrics-server-client-certs\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.889163 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.888995 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ee2d368-c317-40c4-a001-46103a67a613-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.889163 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.889034 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69s9j\" (UniqueName: \"kubernetes.io/projected/1ee2d368-c317-40c4-a001-46103a67a613-kube-api-access-69s9j\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.990238 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.990201 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1ee2d368-c317-40c4-a001-46103a67a613-client-ca-bundle\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.990458 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.990295 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1ee2d368-c317-40c4-a001-46103a67a613-secret-metrics-server-tls\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.990458 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.990346 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1ee2d368-c317-40c4-a001-46103a67a613-metrics-server-audit-profiles\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.990458 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.990374 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1ee2d368-c317-40c4-a001-46103a67a613-audit-log\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.990458 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.990402 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1ee2d368-c317-40c4-a001-46103a67a613-secret-metrics-server-client-certs\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.990458 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.990435 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ee2d368-c317-40c4-a001-46103a67a613-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.990742 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.990471 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69s9j\" (UniqueName: \"kubernetes.io/projected/1ee2d368-c317-40c4-a001-46103a67a613-kube-api-access-69s9j\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.992213 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.991541 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ee2d368-c317-40c4-a001-46103a67a613-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.992213 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.991922 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1ee2d368-c317-40c4-a001-46103a67a613-metrics-server-audit-profiles\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.992213 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.992174 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1ee2d368-c317-40c4-a001-46103a67a613-audit-log\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.993752 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.993728 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1ee2d368-c317-40c4-a001-46103a67a613-secret-metrics-server-tls\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.994595 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.994455 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1ee2d368-c317-40c4-a001-46103a67a613-client-ca-bundle\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.995142 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.995117 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1ee2d368-c317-40c4-a001-46103a67a613-secret-metrics-server-client-certs\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:58.999605 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:58.999582 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69s9j\" (UniqueName: \"kubernetes.io/projected/1ee2d368-c317-40c4-a001-46103a67a613-kube-api-access-69s9j\") pod \"metrics-server-7b56dd7c9d-4rmh7\" (UID: \"1ee2d368-c317-40c4-a001-46103a67a613\") " pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:59.095566 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.095479 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:10:59.199461 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.199389 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bb6fb9c4f-25w6d"] Sep 17 23:10:59.223525 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.223462 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.223753 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.223365 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bb6fb9c4f-25w6d"] Sep 17 23:10:59.226227 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.226189 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 17 23:10:59.227219 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.227200 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 17 23:10:59.227561 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.227546 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 17 23:10:59.228738 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.228010 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 17 23:10:59.228738 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.228081 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rk242\"" Sep 17 23:10:59.228738 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.228645 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 17 23:10:59.238851 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.236952 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 17 23:10:59.365439 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.365339 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" event={"ID":"e6c946e8-c084-434e-b49d-7ab905a4be36","Type":"ContainerStarted","Data":"e419f03c70ef131790899b89a6113f34c642c12ec38a51a05e464276caafe377"} Sep 17 23:10:59.383013 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.382953 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-t9g4f" podStartSLOduration=2.939338154 podStartE2EDuration="5.382933181s" totalImagesPullingTime="2.443595027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:59.382293393 +0000 UTC m=+298.316200531" watchObservedRunningTime="2026-09-17 23:10:59.382933181 +0000 UTC m=+298.316840318" Sep 17 23:10:59.393298 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.393267 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96488f3c-5712-4da9-8f7b-788025e9439b-console-oauth-config\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.393440 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.393319 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-service-ca\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.393440 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.393365 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jss2h\" (UniqueName: \"kubernetes.io/projected/96488f3c-5712-4da9-8f7b-788025e9439b-kube-api-access-jss2h\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.393440 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.393401 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-oauth-serving-cert\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.393650 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.393451 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-trusted-ca-bundle\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.393650 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.393517 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-console-config\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.393650 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.393560 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96488f3c-5712-4da9-8f7b-788025e9439b-console-serving-cert\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.494756 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.494715 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jss2h\" (UniqueName: \"kubernetes.io/projected/96488f3c-5712-4da9-8f7b-788025e9439b-kube-api-access-jss2h\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.495547 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.494773 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-oauth-serving-cert\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.495547 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.494826 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-trusted-ca-bundle\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.495547 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.494879 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-console-config\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.495547 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.494932 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96488f3c-5712-4da9-8f7b-788025e9439b-console-serving-cert\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.495803 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.495579 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96488f3c-5712-4da9-8f7b-788025e9439b-console-oauth-config\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.495803 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.495623 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-service-ca\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.496050 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.496013 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-oauth-serving-cert\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.496486 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.496465 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-console-config\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.496806 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.496766 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-service-ca\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.497085 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.497047 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-trusted-ca-bundle\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.501418 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.501365 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96488f3c-5712-4da9-8f7b-788025e9439b-console-oauth-config\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.501719 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.501695 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96488f3c-5712-4da9-8f7b-788025e9439b-console-serving-cert\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.504196 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.504175 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jss2h\" (UniqueName: \"kubernetes.io/projected/96488f3c-5712-4da9-8f7b-788025e9439b-kube-api-access-jss2h\") pod \"console-6bb6fb9c4f-25w6d\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:10:59.542039 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:10:59.541834 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:11:00.066518 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:00.066413 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bb6fb9c4f-25w6d"] Sep 17 23:11:00.080206 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:00.080177 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7"] Sep 17 23:11:00.148111 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:11:00.148077 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96488f3c_5712_4da9_8f7b_788025e9439b.slice/crio-73ad74070d7e4526a32747abad925e80baeb652c380daab34492f813c15c423a WatchSource:0}: Error finding container 73ad74070d7e4526a32747abad925e80baeb652c380daab34492f813c15c423a: Status 404 returned error can't find the container with id 73ad74070d7e4526a32747abad925e80baeb652c380daab34492f813c15c423a Sep 17 23:11:00.149423 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:11:00.149122 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee2d368_c317_40c4_a001_46103a67a613.slice/crio-1495781ae103227026c18f2b787f8e4e97c3ef95a6b9a76b743600c8264aabee WatchSource:0}: Error finding container 1495781ae103227026c18f2b787f8e4e97c3ef95a6b9a76b743600c8264aabee: Status 404 returned error can't find the container with id 1495781ae103227026c18f2b787f8e4e97c3ef95a6b9a76b743600c8264aabee Sep 17 23:11:00.260019 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:00.259986 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cb7955849-x2x4q" Sep 17 23:11:00.374447 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:00.374419 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" event={"ID":"1ee2d368-c317-40c4-a001-46103a67a613","Type":"ContainerStarted","Data":"1495781ae103227026c18f2b787f8e4e97c3ef95a6b9a76b743600c8264aabee"} Sep 17 23:11:00.376441 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:00.376413 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" event={"ID":"da235c63-916e-40bd-ab57-ffc8d26f12f9","Type":"ContainerStarted","Data":"5f6c943a042f387e25fb5a34a941227749d22f27e51eed2aa4d2567a1cf6c863"} Sep 17 23:11:00.377894 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:00.377868 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bb6fb9c4f-25w6d" event={"ID":"96488f3c-5712-4da9-8f7b-788025e9439b","Type":"ContainerStarted","Data":"73ad74070d7e4526a32747abad925e80baeb652c380daab34492f813c15c423a"} Sep 17 23:11:01.389103 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:01.388947 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" event={"ID":"da235c63-916e-40bd-ab57-ffc8d26f12f9","Type":"ContainerStarted","Data":"f80ec3f8076cf93fb829b19f35b40e8612c33c35c304eec7392d60a34eb67e59"} Sep 17 23:11:01.389103 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:01.389035 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" event={"ID":"da235c63-916e-40bd-ab57-ffc8d26f12f9","Type":"ContainerStarted","Data":"6c540007f306f985a2f1f3e54a8aa773bbe2beacdf43ba3473c408d3685d0cf6"} Sep 17 23:11:02.933278 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:02.933171 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" podUID="5acef9f4-6131-4772-abab-596a3d81af48" containerName="registry" containerID="cri-o://2061424c4323dd5bb66c46fdb1e015efc49852fa9d9e230bf84f75510868b6e2" gracePeriod=30 Sep 17 23:11:03.556279 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:03.406555 2582 generic.go:356] "Generic (PLEG): container finished" podID="5acef9f4-6131-4772-abab-596a3d81af48" containerID="2061424c4323dd5bb66c46fdb1e015efc49852fa9d9e230bf84f75510868b6e2" exitCode=0 Sep 17 23:11:03.556279 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:03.406616 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" event={"ID":"5acef9f4-6131-4772-abab-596a3d81af48","Type":"ContainerDied","Data":"2061424c4323dd5bb66c46fdb1e015efc49852fa9d9e230bf84f75510868b6e2"} Sep 17 23:11:03.815052 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:03.814972 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bb6fb9c4f-25w6d"] Sep 17 23:11:04.447910 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.447889 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:11:04.550575 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.550530 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5acef9f4-6131-4772-abab-596a3d81af48-installation-pull-secrets\") pod \"5acef9f4-6131-4772-abab-596a3d81af48\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " Sep 17 23:11:04.550740 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.550621 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5acef9f4-6131-4772-abab-596a3d81af48-image-registry-private-configuration\") pod \"5acef9f4-6131-4772-abab-596a3d81af48\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " Sep 17 23:11:04.550740 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.550647 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls\") pod \"5acef9f4-6131-4772-abab-596a3d81af48\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " Sep 17 23:11:04.550740 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.550666 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-bound-sa-token\") pod \"5acef9f4-6131-4772-abab-596a3d81af48\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " Sep 17 23:11:04.550740 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.550702 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5acef9f4-6131-4772-abab-596a3d81af48-ca-trust-extracted\") pod \"5acef9f4-6131-4772-abab-596a3d81af48\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " Sep 17 23:11:04.550740 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.550730 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp6pb\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-kube-api-access-jp6pb\") pod \"5acef9f4-6131-4772-abab-596a3d81af48\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " Sep 17 23:11:04.550967 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.550763 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5acef9f4-6131-4772-abab-596a3d81af48-registry-certificates\") pod \"5acef9f4-6131-4772-abab-596a3d81af48\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " Sep 17 23:11:04.550967 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.550808 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5acef9f4-6131-4772-abab-596a3d81af48-trusted-ca\") pod \"5acef9f4-6131-4772-abab-596a3d81af48\" (UID: \"5acef9f4-6131-4772-abab-596a3d81af48\") " Sep 17 23:11:04.551388 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.551330 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5acef9f4-6131-4772-abab-596a3d81af48-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5acef9f4-6131-4772-abab-596a3d81af48" (UID: "5acef9f4-6131-4772-abab-596a3d81af48"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:11:04.551537 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.551389 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5acef9f4-6131-4772-abab-596a3d81af48-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5acef9f4-6131-4772-abab-596a3d81af48" (UID: "5acef9f4-6131-4772-abab-596a3d81af48"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:11:04.553116 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.553093 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5acef9f4-6131-4772-abab-596a3d81af48-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5acef9f4-6131-4772-abab-596a3d81af48" (UID: "5acef9f4-6131-4772-abab-596a3d81af48"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:11:04.553231 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.553208 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5acef9f4-6131-4772-abab-596a3d81af48" (UID: "5acef9f4-6131-4772-abab-596a3d81af48"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:11:04.553293 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.553277 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5acef9f4-6131-4772-abab-596a3d81af48" (UID: "5acef9f4-6131-4772-abab-596a3d81af48"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:11:04.553444 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.553425 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-kube-api-access-jp6pb" (OuterVolumeSpecName: "kube-api-access-jp6pb") pod "5acef9f4-6131-4772-abab-596a3d81af48" (UID: "5acef9f4-6131-4772-abab-596a3d81af48"). InnerVolumeSpecName "kube-api-access-jp6pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:11:04.553636 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.553612 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5acef9f4-6131-4772-abab-596a3d81af48-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5acef9f4-6131-4772-abab-596a3d81af48" (UID: "5acef9f4-6131-4772-abab-596a3d81af48"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:11:04.559064 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.559027 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5acef9f4-6131-4772-abab-596a3d81af48-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5acef9f4-6131-4772-abab-596a3d81af48" (UID: "5acef9f4-6131-4772-abab-596a3d81af48"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:11:04.651648 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.651611 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp6pb\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-kube-api-access-jp6pb\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:11:04.651648 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.651640 2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5acef9f4-6131-4772-abab-596a3d81af48-registry-certificates\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:11:04.651648 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.651652 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5acef9f4-6131-4772-abab-596a3d81af48-trusted-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:11:04.651861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.651660 2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5acef9f4-6131-4772-abab-596a3d81af48-installation-pull-secrets\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:11:04.651861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.651671 2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5acef9f4-6131-4772-abab-596a3d81af48-image-registry-private-configuration\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:11:04.651861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.651680 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-registry-tls\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:11:04.651861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.651688 2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5acef9f4-6131-4772-abab-596a3d81af48-bound-sa-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:11:04.651861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:04.651696 2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5acef9f4-6131-4772-abab-596a3d81af48-ca-trust-extracted\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:11:05.414763 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:05.414731 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" event={"ID":"5acef9f4-6131-4772-abab-596a3d81af48","Type":"ContainerDied","Data":"19fe4fd0229b0cba58e5515a20585513ad8326d42a96a6ab1a269855da32aa91"} Sep 17 23:11:05.414763 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:05.414762 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd9896dfd-2sbbg" Sep 17 23:11:05.414980 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:05.414794 2582 scope.go:117] "RemoveContainer" containerID="2061424c4323dd5bb66c46fdb1e015efc49852fa9d9e230bf84f75510868b6e2" Sep 17 23:11:05.430942 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:05.430912 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fd9896dfd-2sbbg"] Sep 17 23:11:05.433897 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:05.433873 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7fd9896dfd-2sbbg"] Sep 17 23:11:06.422401 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:06.422364 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" event={"ID":"da235c63-916e-40bd-ab57-ffc8d26f12f9","Type":"ContainerStarted","Data":"56d25bedd0d285d4a65e5418311061276f5e7da319584380aab43a8ef78bc265"} Sep 17 23:11:06.674780 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:06.674721 2582 kubelet.go:1648] "Image garbage collection succeeded" Sep 17 23:11:07.287888 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:07.287859 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:11:07.298386 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:07.298314 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5acef9f4-6131-4772-abab-596a3d81af48" path="/var/lib/kubelet/pods/5acef9f4-6131-4772-abab-596a3d81af48/volumes" Sep 17 23:11:07.429023 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:07.428975 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" event={"ID":"1ee2d368-c317-40c4-a001-46103a67a613","Type":"ContainerStarted","Data":"96579461c10fa70df6695c81d9b554dbb1776a9526072b581012127e1b6e8066"} Sep 17 23:11:07.432059 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:07.432026 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" event={"ID":"da235c63-916e-40bd-ab57-ffc8d26f12f9","Type":"ContainerStarted","Data":"3fadcf5d52c98320117e797746f9be269c874b8dc7dbe174200adc0f3f5a0190"} Sep 17 23:11:07.432191 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:07.432090 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" event={"ID":"da235c63-916e-40bd-ab57-ffc8d26f12f9","Type":"ContainerStarted","Data":"8380b48e833654ed044dfd545f24a4f300ea5155e3edef2b7713d3f755973123"} Sep 17 23:11:07.432191 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:07.432137 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:11:07.433978 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:07.433953 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bb6fb9c4f-25w6d" event={"ID":"96488f3c-5712-4da9-8f7b-788025e9439b","Type":"ContainerStarted","Data":"d2d7b805ce6f58c3cc466e63a6cb0c938aa3e7ddf048e5431af7cf47df621984"} Sep 17 23:11:07.437897 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:07.437879 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" Sep 17 23:11:07.447597 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:07.447546 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" podStartSLOduration=2.971646971 podStartE2EDuration="9.447528891s" totalImagesPullingTime="6.47588192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:07.447188227 +0000 UTC m=+306.381095363" watchObservedRunningTime="2026-09-17 23:11:07.447528891 +0000 UTC m=+306.381436026" Sep 17 23:11:07.464568 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:07.464525 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bb6fb9c4f-25w6d" podStartSLOduration=1.585882171 podStartE2EDuration="8.46451217s" totalImagesPullingTime="6.878629999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:07.463387595 +0000 UTC m=+306.397294729" watchObservedRunningTime="2026-09-17 23:11:07.46451217 +0000 UTC m=+306.398419297" Sep 17 23:11:07.486681 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:07.486640 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7858b594b4-bqbbp" podStartSLOduration=6.223290961 podStartE2EDuration="11.486626604s" totalImagesPullingTime="5.263335643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:07.485020573 +0000 UTC m=+306.418927713" watchObservedRunningTime="2026-09-17 23:11:07.486626604 +0000 UTC m=+306.420533740" Sep 17 23:11:09.542438 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:09.542406 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:11:19.096253 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:19.096217 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:11:19.096881 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:19.096312 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:11:32.452644 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.452579 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6bb6fb9c4f-25w6d" podUID="96488f3c-5712-4da9-8f7b-788025e9439b" containerName="console" containerID="cri-o://d2d7b805ce6f58c3cc466e63a6cb0c938aa3e7ddf048e5431af7cf47df621984" gracePeriod=15 Sep 17 23:11:32.524603 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.524573 2582 generic.go:356] "Generic (PLEG): container finished" podID="9ff60ca3-2533-4ed5-9cc1-56bad886cc5d" containerID="9953ce7c789dad7f2c34b3813d2906997aa265ecb98f7f8fb9d1a6d9cd12286d" exitCode=0 Sep 17 23:11:32.524764 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.524646 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-d6px4" event={"ID":"9ff60ca3-2533-4ed5-9cc1-56bad886cc5d","Type":"ContainerDied","Data":"9953ce7c789dad7f2c34b3813d2906997aa265ecb98f7f8fb9d1a6d9cd12286d"} Sep 17 23:11:32.524964 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.524950 2582 scope.go:117] "RemoveContainer" containerID="9953ce7c789dad7f2c34b3813d2906997aa265ecb98f7f8fb9d1a6d9cd12286d" Sep 17 23:11:32.745634 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.745602 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:11:32.882512 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.882467 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-console-config\") pod \"96488f3c-5712-4da9-8f7b-788025e9439b\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " Sep 17 23:11:32.882699 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.882574 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96488f3c-5712-4da9-8f7b-788025e9439b-console-serving-cert\") pod \"96488f3c-5712-4da9-8f7b-788025e9439b\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " Sep 17 23:11:32.882699 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.882671 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96488f3c-5712-4da9-8f7b-788025e9439b-console-oauth-config\") pod \"96488f3c-5712-4da9-8f7b-788025e9439b\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " Sep 17 23:11:32.882817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.882709 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-service-ca\") pod \"96488f3c-5712-4da9-8f7b-788025e9439b\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " Sep 17 23:11:32.882817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.882774 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jss2h\" (UniqueName: \"kubernetes.io/projected/96488f3c-5712-4da9-8f7b-788025e9439b-kube-api-access-jss2h\") pod \"96488f3c-5712-4da9-8f7b-788025e9439b\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " Sep 17 23:11:32.882817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.882798 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-trusted-ca-bundle\") pod \"96488f3c-5712-4da9-8f7b-788025e9439b\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " Sep 17 23:11:32.882966 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.882827 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-oauth-serving-cert\") pod \"96488f3c-5712-4da9-8f7b-788025e9439b\" (UID: \"96488f3c-5712-4da9-8f7b-788025e9439b\") " Sep 17 23:11:32.882966 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.882899 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-console-config" (OuterVolumeSpecName: "console-config") pod "96488f3c-5712-4da9-8f7b-788025e9439b" (UID: "96488f3c-5712-4da9-8f7b-788025e9439b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:11:32.883090 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.883062 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-console-config\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:11:32.883168 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.883143 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-service-ca" (OuterVolumeSpecName: "service-ca") pod "96488f3c-5712-4da9-8f7b-788025e9439b" (UID: "96488f3c-5712-4da9-8f7b-788025e9439b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:11:32.883223 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.883194 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "96488f3c-5712-4da9-8f7b-788025e9439b" (UID: "96488f3c-5712-4da9-8f7b-788025e9439b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:11:32.883711 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.883682 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "96488f3c-5712-4da9-8f7b-788025e9439b" (UID: "96488f3c-5712-4da9-8f7b-788025e9439b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:11:32.884998 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.884974 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96488f3c-5712-4da9-8f7b-788025e9439b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "96488f3c-5712-4da9-8f7b-788025e9439b" (UID: "96488f3c-5712-4da9-8f7b-788025e9439b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:11:32.885058 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.885028 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96488f3c-5712-4da9-8f7b-788025e9439b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "96488f3c-5712-4da9-8f7b-788025e9439b" (UID: "96488f3c-5712-4da9-8f7b-788025e9439b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:11:32.885253 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.885232 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96488f3c-5712-4da9-8f7b-788025e9439b-kube-api-access-jss2h" (OuterVolumeSpecName: "kube-api-access-jss2h") pod "96488f3c-5712-4da9-8f7b-788025e9439b" (UID: "96488f3c-5712-4da9-8f7b-788025e9439b"). InnerVolumeSpecName "kube-api-access-jss2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:11:32.983860 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.983826 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jss2h\" (UniqueName: \"kubernetes.io/projected/96488f3c-5712-4da9-8f7b-788025e9439b-kube-api-access-jss2h\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:11:32.983860 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.983860 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-trusted-ca-bundle\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:11:32.984049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.983869 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-oauth-serving-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:11:32.984049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.983877 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96488f3c-5712-4da9-8f7b-788025e9439b-console-serving-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:11:32.984049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.983886 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96488f3c-5712-4da9-8f7b-788025e9439b-console-oauth-config\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:11:32.984049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:32.983894 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96488f3c-5712-4da9-8f7b-788025e9439b-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:11:33.530695 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:33.530659 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-d6px4" event={"ID":"9ff60ca3-2533-4ed5-9cc1-56bad886cc5d","Type":"ContainerStarted","Data":"cb56c9b30c5a975ef2b8f89183ef9f200a4c7203e572647dddf97d4bb331c85f"} Sep 17 23:11:33.531895 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:33.531870 2582 generic.go:356] "Generic (PLEG): container finished" podID="96488f3c-5712-4da9-8f7b-788025e9439b" containerID="d2d7b805ce6f58c3cc466e63a6cb0c938aa3e7ddf048e5431af7cf47df621984" exitCode=0 Sep 17 23:11:33.532004 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:33.531937 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bb6fb9c4f-25w6d" event={"ID":"96488f3c-5712-4da9-8f7b-788025e9439b","Type":"ContainerDied","Data":"d2d7b805ce6f58c3cc466e63a6cb0c938aa3e7ddf048e5431af7cf47df621984"} Sep 17 23:11:33.532004 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:33.531971 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bb6fb9c4f-25w6d" event={"ID":"96488f3c-5712-4da9-8f7b-788025e9439b","Type":"ContainerDied","Data":"73ad74070d7e4526a32747abad925e80baeb652c380daab34492f813c15c423a"} Sep 17 23:11:33.532004 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:33.531989 2582 scope.go:117] "RemoveContainer" containerID="d2d7b805ce6f58c3cc466e63a6cb0c938aa3e7ddf048e5431af7cf47df621984" Sep 17 23:11:33.532095 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:33.531986 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bb6fb9c4f-25w6d" Sep 17 23:11:33.540193 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:33.540169 2582 scope.go:117] "RemoveContainer" containerID="d2d7b805ce6f58c3cc466e63a6cb0c938aa3e7ddf048e5431af7cf47df621984" Sep 17 23:11:33.540521 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:11:33.540485 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2d7b805ce6f58c3cc466e63a6cb0c938aa3e7ddf048e5431af7cf47df621984\": container with ID starting with d2d7b805ce6f58c3cc466e63a6cb0c938aa3e7ddf048e5431af7cf47df621984 not found: ID does not exist" containerID="d2d7b805ce6f58c3cc466e63a6cb0c938aa3e7ddf048e5431af7cf47df621984" Sep 17 23:11:33.540596 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:33.540535 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2d7b805ce6f58c3cc466e63a6cb0c938aa3e7ddf048e5431af7cf47df621984"} err="failed to get container status \"d2d7b805ce6f58c3cc466e63a6cb0c938aa3e7ddf048e5431af7cf47df621984\": rpc error: code = NotFound desc = could not find container \"d2d7b805ce6f58c3cc466e63a6cb0c938aa3e7ddf048e5431af7cf47df621984\": container with ID starting with d2d7b805ce6f58c3cc466e63a6cb0c938aa3e7ddf048e5431af7cf47df621984 not found: ID does not exist" Sep 17 23:11:33.565697 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:33.565607 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bb6fb9c4f-25w6d"] Sep 17 23:11:33.571867 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:33.571835 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bb6fb9c4f-25w6d"] Sep 17 23:11:35.292029 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:35.291998 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96488f3c-5712-4da9-8f7b-788025e9439b" path="/var/lib/kubelet/pods/96488f3c-5712-4da9-8f7b-788025e9439b/volumes" Sep 17 23:11:39.101782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:39.101752 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:11:39.105523 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:11:39.105488 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7b56dd7c9d-4rmh7" Sep 17 23:12:10.920031 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.919998 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65bdd89887-4xl2h"] Sep 17 23:12:10.920439 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.920338 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5acef9f4-6131-4772-abab-596a3d81af48" containerName="registry" Sep 17 23:12:10.920439 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.920351 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5acef9f4-6131-4772-abab-596a3d81af48" containerName="registry" Sep 17 23:12:10.920439 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.920368 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96488f3c-5712-4da9-8f7b-788025e9439b" containerName="console" Sep 17 23:12:10.920439 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.920373 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="96488f3c-5712-4da9-8f7b-788025e9439b" containerName="console" Sep 17 23:12:10.920439 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.920416 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="96488f3c-5712-4da9-8f7b-788025e9439b" containerName="console" Sep 17 23:12:10.920439 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.920424 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5acef9f4-6131-4772-abab-596a3d81af48" containerName="registry" Sep 17 23:12:10.923304 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.923285 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:10.925590 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.925561 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 17 23:12:10.925712 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.925621 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 17 23:12:10.925778 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.925747 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rk242\"" Sep 17 23:12:10.926042 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.926019 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 17 23:12:10.926139 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.926094 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 17 23:12:10.926219 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.926205 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 17 23:12:10.932553 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.932519 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 17 23:12:10.940973 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.940952 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65bdd89887-4xl2h"] Sep 17 23:12:10.990632 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.990608 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4b06529-61f5-4617-8b85-152485d9f1ca-console-oauth-config\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:10.990786 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.990648 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4b06529-61f5-4617-8b85-152485d9f1ca-console-serving-cert\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:10.990786 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.990666 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-console-config\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:10.990786 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.990766 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t9pr\" (UniqueName: \"kubernetes.io/projected/b4b06529-61f5-4617-8b85-152485d9f1ca-kube-api-access-8t9pr\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:10.990906 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.990800 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-oauth-serving-cert\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:10.990906 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.990862 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-service-ca\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:10.990906 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:10.990888 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-trusted-ca-bundle\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:11.091495 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:11.091465 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t9pr\" (UniqueName: \"kubernetes.io/projected/b4b06529-61f5-4617-8b85-152485d9f1ca-kube-api-access-8t9pr\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:11.091634 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:11.091517 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-oauth-serving-cert\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:11.091634 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:11.091552 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-service-ca\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:11.091634 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:11.091567 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-trusted-ca-bundle\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:11.091634 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:11.091586 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4b06529-61f5-4617-8b85-152485d9f1ca-console-oauth-config\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:11.091634 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:11.091629 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4b06529-61f5-4617-8b85-152485d9f1ca-console-serving-cert\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:11.091871 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:11.091745 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-console-config\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:11.092424 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:11.092381 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-oauth-serving-cert\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:11.092424 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:11.092381 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-service-ca\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:11.092607 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:11.092386 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-console-config\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:11.092607 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:11.092547 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-trusted-ca-bundle\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:11.094160 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:11.094130 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4b06529-61f5-4617-8b85-152485d9f1ca-console-oauth-config\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:11.094276 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:11.094220 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4b06529-61f5-4617-8b85-152485d9f1ca-console-serving-cert\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:11.099052 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:11.099031 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t9pr\" (UniqueName: \"kubernetes.io/projected/b4b06529-61f5-4617-8b85-152485d9f1ca-kube-api-access-8t9pr\") pod \"console-65bdd89887-4xl2h\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:11.233733 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:11.233703 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:11.501782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:11.501757 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65bdd89887-4xl2h"] Sep 17 23:12:11.504478 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:12:11.504446 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b06529_61f5_4617_8b85_152485d9f1ca.slice/crio-ac7bcdc65e02ac4ae99bf910de782f61c6ab1b6485bcfc195503c30f64095de3 WatchSource:0}: Error finding container ac7bcdc65e02ac4ae99bf910de782f61c6ab1b6485bcfc195503c30f64095de3: Status 404 returned error can't find the container with id ac7bcdc65e02ac4ae99bf910de782f61c6ab1b6485bcfc195503c30f64095de3 Sep 17 23:12:11.677055 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:11.677022 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65bdd89887-4xl2h" event={"ID":"b4b06529-61f5-4617-8b85-152485d9f1ca","Type":"ContainerStarted","Data":"4a95a124608e7ebed93227040ebc17c3825c8d1c03a55bb6baf491dc6a15a1e2"} Sep 17 23:12:11.677216 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:11.677071 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65bdd89887-4xl2h" event={"ID":"b4b06529-61f5-4617-8b85-152485d9f1ca","Type":"ContainerStarted","Data":"ac7bcdc65e02ac4ae99bf910de782f61c6ab1b6485bcfc195503c30f64095de3"} Sep 17 23:12:11.693831 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:11.693781 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65bdd89887-4xl2h" podStartSLOduration=1.693767046 podStartE2EDuration="1.693767046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:11.692662987 +0000 UTC m=+370.626570125" watchObservedRunningTime="2026-09-17 23:12:11.693767046 +0000 UTC m=+370.627674182" Sep 17 23:12:16.127602 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:16.127569 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs\") pod \"network-metrics-daemon-xg2sz\" (UID: \"3752b568-d53f-49ac-9095-0f61cee96767\") " pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:12:16.129754 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:16.129735 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3752b568-d53f-49ac-9095-0f61cee96767-metrics-certs\") pod \"network-metrics-daemon-xg2sz\" (UID: \"3752b568-d53f-49ac-9095-0f61cee96767\") " pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:12:16.291311 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:16.291279 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hrj9x\"" Sep 17 23:12:16.298755 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:16.298732 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xg2sz" Sep 17 23:12:16.564885 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:16.564860 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xg2sz"] Sep 17 23:12:16.569957 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:12:16.569930 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3752b568_d53f_49ac_9095_0f61cee96767.slice/crio-4fd76481574e099d53d610229c0f995ee83219f639d18006513fe49d3ebd1a3f WatchSource:0}: Error finding container 4fd76481574e099d53d610229c0f995ee83219f639d18006513fe49d3ebd1a3f: Status 404 returned error can't find the container with id 4fd76481574e099d53d610229c0f995ee83219f639d18006513fe49d3ebd1a3f Sep 17 23:12:16.572265 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:16.572247 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 23:12:16.696625 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:16.696589 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xg2sz" event={"ID":"3752b568-d53f-49ac-9095-0f61cee96767","Type":"ContainerStarted","Data":"4fd76481574e099d53d610229c0f995ee83219f639d18006513fe49d3ebd1a3f"} Sep 17 23:12:17.702901 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:17.702825 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xg2sz" event={"ID":"3752b568-d53f-49ac-9095-0f61cee96767","Type":"ContainerStarted","Data":"0625f465c6da30c91b3494af151bcd513227e9d368f278838a58ff7220abd356"} Sep 17 23:12:17.702901 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:17.702882 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xg2sz" event={"ID":"3752b568-d53f-49ac-9095-0f61cee96767","Type":"ContainerStarted","Data":"144b5f1db694df887ee546614ab00430cef645ff8e057a9425f7bd71bf748f00"} Sep 17 23:12:17.718476 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:17.718430 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xg2sz" podStartSLOduration=373.858846243 podStartE2EDuration="6m14.718415098s" totalImagesPullingTime="859.568855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:17.717555689 +0000 UTC m=+376.651462824" watchObservedRunningTime="2026-09-17 23:12:17.718415098 +0000 UTC m=+376.652322233" Sep 17 23:12:21.234727 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:21.234700 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:21.235098 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:21.234747 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:21.239321 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:21.239298 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:12:21.721428 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:12:21.721399 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:13:58.108101 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:58.108055 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-r7n8f"] Sep 17 23:13:58.110040 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:58.110023 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r7n8f" Sep 17 23:13:58.112290 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:58.112262 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 17 23:13:58.112438 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:58.112395 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 17 23:13:58.113083 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:58.113069 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-l7jvd\"" Sep 17 23:13:58.119170 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:58.119145 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-r7n8f"] Sep 17 23:13:58.179197 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:58.179162 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6969bd4f-8891-4ba1-bfe9-d3c98225d49e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-r7n8f\" (UID: \"6969bd4f-8891-4ba1-bfe9-d3c98225d49e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r7n8f" Sep 17 23:13:58.179330 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:58.179231 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m286b\" (UniqueName: \"kubernetes.io/projected/6969bd4f-8891-4ba1-bfe9-d3c98225d49e-kube-api-access-m286b\") pod \"cert-manager-cainjector-559cdd8c87-r7n8f\" (UID: \"6969bd4f-8891-4ba1-bfe9-d3c98225d49e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r7n8f" Sep 17 23:13:58.279742 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:58.279716 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6969bd4f-8891-4ba1-bfe9-d3c98225d49e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-r7n8f\" (UID: \"6969bd4f-8891-4ba1-bfe9-d3c98225d49e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r7n8f" Sep 17 23:13:58.279865 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:58.279779 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m286b\" (UniqueName: \"kubernetes.io/projected/6969bd4f-8891-4ba1-bfe9-d3c98225d49e-kube-api-access-m286b\") pod \"cert-manager-cainjector-559cdd8c87-r7n8f\" (UID: \"6969bd4f-8891-4ba1-bfe9-d3c98225d49e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r7n8f" Sep 17 23:13:58.287294 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:58.287265 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6969bd4f-8891-4ba1-bfe9-d3c98225d49e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-r7n8f\" (UID: \"6969bd4f-8891-4ba1-bfe9-d3c98225d49e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r7n8f" Sep 17 23:13:58.287396 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:58.287278 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m286b\" (UniqueName: \"kubernetes.io/projected/6969bd4f-8891-4ba1-bfe9-d3c98225d49e-kube-api-access-m286b\") pod \"cert-manager-cainjector-559cdd8c87-r7n8f\" (UID: \"6969bd4f-8891-4ba1-bfe9-d3c98225d49e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r7n8f" Sep 17 23:13:58.429373 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:58.429310 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r7n8f" Sep 17 23:13:58.698390 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:58.698301 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-r7n8f"] Sep 17 23:13:58.701824 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:13:58.701793 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6969bd4f_8891_4ba1_bfe9_d3c98225d49e.slice/crio-3d0429ad47ef8f23d89b94fe1d97742900366213726e5003353ddfdced7450be WatchSource:0}: Error finding container 3d0429ad47ef8f23d89b94fe1d97742900366213726e5003353ddfdced7450be: Status 404 returned error can't find the container with id 3d0429ad47ef8f23d89b94fe1d97742900366213726e5003353ddfdced7450be Sep 17 23:13:59.083079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:59.083046 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r7n8f" event={"ID":"6969bd4f-8891-4ba1-bfe9-d3c98225d49e","Type":"ContainerStarted","Data":"3d0429ad47ef8f23d89b94fe1d97742900366213726e5003353ddfdced7450be"} Sep 17 23:13:59.639668 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:59.639637 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fs5wm"] Sep 17 23:13:59.642456 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:59.642438 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-fs5wm" Sep 17 23:13:59.644490 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:59.644459 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-mj7kj\"" Sep 17 23:13:59.652265 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:59.652236 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fs5wm"] Sep 17 23:13:59.691071 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:59.691045 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrw5h\" (UniqueName: \"kubernetes.io/projected/81eea559-747b-4211-b712-c7f4500f1657-kube-api-access-mrw5h\") pod \"cert-manager-webhook-6486ffd997-fs5wm\" (UID: \"81eea559-747b-4211-b712-c7f4500f1657\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fs5wm" Sep 17 23:13:59.691175 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:59.691123 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81eea559-747b-4211-b712-c7f4500f1657-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fs5wm\" (UID: \"81eea559-747b-4211-b712-c7f4500f1657\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fs5wm" Sep 17 23:13:59.792400 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:59.792377 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrw5h\" (UniqueName: \"kubernetes.io/projected/81eea559-747b-4211-b712-c7f4500f1657-kube-api-access-mrw5h\") pod \"cert-manager-webhook-6486ffd997-fs5wm\" (UID: \"81eea559-747b-4211-b712-c7f4500f1657\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fs5wm" Sep 17 23:13:59.792518 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:59.792418 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81eea559-747b-4211-b712-c7f4500f1657-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fs5wm\" (UID: \"81eea559-747b-4211-b712-c7f4500f1657\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fs5wm" Sep 17 23:13:59.799971 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:59.799949 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81eea559-747b-4211-b712-c7f4500f1657-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fs5wm\" (UID: \"81eea559-747b-4211-b712-c7f4500f1657\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fs5wm" Sep 17 23:13:59.800152 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:59.800132 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrw5h\" (UniqueName: \"kubernetes.io/projected/81eea559-747b-4211-b712-c7f4500f1657-kube-api-access-mrw5h\") pod \"cert-manager-webhook-6486ffd997-fs5wm\" (UID: \"81eea559-747b-4211-b712-c7f4500f1657\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fs5wm" Sep 17 23:13:59.953707 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:13:59.953641 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-fs5wm" Sep 17 23:14:00.226034 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:00.226003 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fs5wm"] Sep 17 23:14:00.232182 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:14:00.232051 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81eea559_747b_4211_b712_c7f4500f1657.slice/crio-8ed06e5f10f8562f7226200716b838ed502bc6e13a7ab21d8ead9f09bf4ea1e2 WatchSource:0}: Error finding container 8ed06e5f10f8562f7226200716b838ed502bc6e13a7ab21d8ead9f09bf4ea1e2: Status 404 returned error can't find the container with id 8ed06e5f10f8562f7226200716b838ed502bc6e13a7ab21d8ead9f09bf4ea1e2 Sep 17 23:14:01.094234 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:01.094195 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-fs5wm" event={"ID":"81eea559-747b-4211-b712-c7f4500f1657","Type":"ContainerStarted","Data":"8ed06e5f10f8562f7226200716b838ed502bc6e13a7ab21d8ead9f09bf4ea1e2"} Sep 17 23:14:03.105108 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:03.105062 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r7n8f" event={"ID":"6969bd4f-8891-4ba1-bfe9-d3c98225d49e","Type":"ContainerStarted","Data":"4af99cf38e6ab3184b25e777c2f85299e8fa82894b4287f491c834bbacb3251f"} Sep 17 23:14:03.106802 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:03.106777 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-fs5wm" event={"ID":"81eea559-747b-4211-b712-c7f4500f1657","Type":"ContainerStarted","Data":"643f2efb3d4dd950afcd788177c5118ad2ee03b2d12cf0280c415ec498b167f5"} Sep 17 23:14:03.106914 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:03.106899 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-fs5wm" Sep 17 23:14:03.120175 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:03.120029 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r7n8f" podStartSLOduration=0.898368354 podStartE2EDuration="5.120010349s" totalImagesPullingTime="4.221641995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:03.119559351 +0000 UTC m=+482.053466487" watchObservedRunningTime="2026-09-17 23:14:03.120010349 +0000 UTC m=+482.053917489" Sep 17 23:14:03.138018 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:03.137967 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-fs5wm" podStartSLOduration=1.435762499 podStartE2EDuration="4.137947456s" totalImagesPullingTime="2.702184957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:03.135798565 +0000 UTC m=+482.069705697" watchObservedRunningTime="2026-09-17 23:14:03.137947456 +0000 UTC m=+482.071854593" Sep 17 23:14:09.112583 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:09.112536 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-fs5wm" Sep 17 23:14:27.256560 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.256523 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw"] Sep 17 23:14:27.259961 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.259941 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" Sep 17 23:14:27.262581 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.262562 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 17 23:14:27.262897 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.262878 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 17 23:14:27.263127 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.263109 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-24qcm\"" Sep 17 23:14:27.263209 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.263113 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 17 23:14:27.263209 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.263180 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 17 23:14:27.281019 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.280961 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw"] Sep 17 23:14:27.416116 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.416073 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scsk5\" (UniqueName: \"kubernetes.io/projected/60f5593a-58fb-4ff3-9df6-7b9a43eb49e2-kube-api-access-scsk5\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-xl5hw\" (UID: \"60f5593a-58fb-4ff3-9df6-7b9a43eb49e2\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" Sep 17 23:14:27.416312 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.416136 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60f5593a-58fb-4ff3-9df6-7b9a43eb49e2-tmp\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-xl5hw\" (UID: \"60f5593a-58fb-4ff3-9df6-7b9a43eb49e2\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" Sep 17 23:14:27.416312 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.416184 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/60f5593a-58fb-4ff3-9df6-7b9a43eb49e2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-xl5hw\" (UID: \"60f5593a-58fb-4ff3-9df6-7b9a43eb49e2\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" Sep 17 23:14:27.416312 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.416287 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/60f5593a-58fb-4ff3-9df6-7b9a43eb49e2-manifests\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-xl5hw\" (UID: \"60f5593a-58fb-4ff3-9df6-7b9a43eb49e2\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" Sep 17 23:14:27.416588 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.416406 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60f5593a-58fb-4ff3-9df6-7b9a43eb49e2-webhook-cert\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-xl5hw\" (UID: \"60f5593a-58fb-4ff3-9df6-7b9a43eb49e2\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" Sep 17 23:14:27.517218 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.517132 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/60f5593a-58fb-4ff3-9df6-7b9a43eb49e2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-xl5hw\" (UID: \"60f5593a-58fb-4ff3-9df6-7b9a43eb49e2\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" Sep 17 23:14:27.517218 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.517184 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/60f5593a-58fb-4ff3-9df6-7b9a43eb49e2-manifests\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-xl5hw\" (UID: \"60f5593a-58fb-4ff3-9df6-7b9a43eb49e2\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" Sep 17 23:14:27.517218 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.517213 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60f5593a-58fb-4ff3-9df6-7b9a43eb49e2-webhook-cert\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-xl5hw\" (UID: \"60f5593a-58fb-4ff3-9df6-7b9a43eb49e2\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" Sep 17 23:14:27.517402 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.517240 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scsk5\" (UniqueName: \"kubernetes.io/projected/60f5593a-58fb-4ff3-9df6-7b9a43eb49e2-kube-api-access-scsk5\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-xl5hw\" (UID: \"60f5593a-58fb-4ff3-9df6-7b9a43eb49e2\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" Sep 17 23:14:27.517402 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.517266 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60f5593a-58fb-4ff3-9df6-7b9a43eb49e2-tmp\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-xl5hw\" (UID: \"60f5593a-58fb-4ff3-9df6-7b9a43eb49e2\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" Sep 17 23:14:27.517767 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.517747 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/60f5593a-58fb-4ff3-9df6-7b9a43eb49e2-manifests\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-xl5hw\" (UID: \"60f5593a-58fb-4ff3-9df6-7b9a43eb49e2\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" Sep 17 23:14:27.519653 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.519625 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60f5593a-58fb-4ff3-9df6-7b9a43eb49e2-tmp\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-xl5hw\" (UID: \"60f5593a-58fb-4ff3-9df6-7b9a43eb49e2\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" Sep 17 23:14:27.519771 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.519755 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/60f5593a-58fb-4ff3-9df6-7b9a43eb49e2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-xl5hw\" (UID: \"60f5593a-58fb-4ff3-9df6-7b9a43eb49e2\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" Sep 17 23:14:27.519813 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.519755 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60f5593a-58fb-4ff3-9df6-7b9a43eb49e2-webhook-cert\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-xl5hw\" (UID: \"60f5593a-58fb-4ff3-9df6-7b9a43eb49e2\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" Sep 17 23:14:27.531365 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.531332 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scsk5\" (UniqueName: \"kubernetes.io/projected/60f5593a-58fb-4ff3-9df6-7b9a43eb49e2-kube-api-access-scsk5\") pod \"opendatahub-operator-controller-manager-747fdf4bf5-xl5hw\" (UID: \"60f5593a-58fb-4ff3-9df6-7b9a43eb49e2\") " pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" Sep 17 23:14:27.570404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.570382 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" Sep 17 23:14:27.843158 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:27.843133 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw"] Sep 17 23:14:27.845835 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:14:27.845812 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f5593a_58fb_4ff3_9df6_7b9a43eb49e2.slice/crio-9b94d546fd77e8ff7ada3dcca046580ce3ed2fe52cf6b7297787b8812bf62e1e WatchSource:0}: Error finding container 9b94d546fd77e8ff7ada3dcca046580ce3ed2fe52cf6b7297787b8812bf62e1e: Status 404 returned error can't find the container with id 9b94d546fd77e8ff7ada3dcca046580ce3ed2fe52cf6b7297787b8812bf62e1e Sep 17 23:14:28.205343 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:28.205266 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" event={"ID":"60f5593a-58fb-4ff3-9df6-7b9a43eb49e2","Type":"ContainerStarted","Data":"9b94d546fd77e8ff7ada3dcca046580ce3ed2fe52cf6b7297787b8812bf62e1e"} Sep 17 23:14:31.221323 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:31.221292 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" event={"ID":"60f5593a-58fb-4ff3-9df6-7b9a43eb49e2","Type":"ContainerStarted","Data":"5e68e17f9408803db1bb372cc5d7729d6e4f746fe8d37f0e631fb567008565ce"} Sep 17 23:14:32.226934 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:32.226905 2582 generic.go:356] "Generic (PLEG): container finished" podID="60f5593a-58fb-4ff3-9df6-7b9a43eb49e2" containerID="5e68e17f9408803db1bb372cc5d7729d6e4f746fe8d37f0e631fb567008565ce" exitCode=0 Sep 17 23:14:32.227281 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:32.226993 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" event={"ID":"60f5593a-58fb-4ff3-9df6-7b9a43eb49e2","Type":"ContainerDied","Data":"5e68e17f9408803db1bb372cc5d7729d6e4f746fe8d37f0e631fb567008565ce"} Sep 17 23:14:33.233721 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:33.233682 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" event={"ID":"60f5593a-58fb-4ff3-9df6-7b9a43eb49e2","Type":"ContainerStarted","Data":"83deb63569326d9c2c4f0fa2ea244b7b010d56b99f1f779d9cf1347de4f30bf3"} Sep 17 23:14:33.234181 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:33.233822 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" Sep 17 23:14:33.255839 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:33.255796 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" podStartSLOduration=1.756564056 podStartE2EDuration="6.2557817s" totalImagesPullingTime="3.433640344s" totalInitContainerRuntime="1.0655773s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:33.254677544 +0000 UTC m=+512.188584696" watchObservedRunningTime="2026-09-17 23:14:33.2557817 +0000 UTC m=+512.189688834" Sep 17 23:14:44.240339 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:44.240310 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-747fdf4bf5-xl5hw" Sep 17 23:14:46.716615 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:46.716531 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d4c67b97b-lcbjm"] Sep 17 23:14:46.722724 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:46.721343 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d4c67b97b-lcbjm" Sep 17 23:14:46.724912 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:46.724885 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 17 23:14:46.725058 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:46.725038 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 17 23:14:46.725172 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:46.725132 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 17 23:14:46.726016 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:46.725996 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hvlzs\"" Sep 17 23:14:46.726074 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:46.726035 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 17 23:14:46.733316 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:46.733295 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d4c67b97b-lcbjm"] Sep 17 23:14:46.772969 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:46.772942 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlqtp\" (UniqueName: \"kubernetes.io/projected/f3db4a6f-b531-4ebe-bcd6-f56239784525-kube-api-access-rlqtp\") pod \"kube-auth-proxy-d4c67b97b-lcbjm\" (UID: \"f3db4a6f-b531-4ebe-bcd6-f56239784525\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-lcbjm" Sep 17 23:14:46.773127 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:46.772981 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3db4a6f-b531-4ebe-bcd6-f56239784525-tls-certs\") pod \"kube-auth-proxy-d4c67b97b-lcbjm\" (UID: \"f3db4a6f-b531-4ebe-bcd6-f56239784525\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-lcbjm" Sep 17 23:14:46.773127 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:46.773008 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3db4a6f-b531-4ebe-bcd6-f56239784525-tmp\") pod \"kube-auth-proxy-d4c67b97b-lcbjm\" (UID: \"f3db4a6f-b531-4ebe-bcd6-f56239784525\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-lcbjm" Sep 17 23:14:46.874059 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:46.874030 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlqtp\" (UniqueName: \"kubernetes.io/projected/f3db4a6f-b531-4ebe-bcd6-f56239784525-kube-api-access-rlqtp\") pod \"kube-auth-proxy-d4c67b97b-lcbjm\" (UID: \"f3db4a6f-b531-4ebe-bcd6-f56239784525\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-lcbjm" Sep 17 23:14:46.874212 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:46.874075 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3db4a6f-b531-4ebe-bcd6-f56239784525-tls-certs\") pod \"kube-auth-proxy-d4c67b97b-lcbjm\" (UID: \"f3db4a6f-b531-4ebe-bcd6-f56239784525\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-lcbjm" Sep 17 23:14:46.874212 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:46.874187 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3db4a6f-b531-4ebe-bcd6-f56239784525-tmp\") pod \"kube-auth-proxy-d4c67b97b-lcbjm\" (UID: \"f3db4a6f-b531-4ebe-bcd6-f56239784525\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-lcbjm" Sep 17 23:14:46.874323 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:14:46.874296 2582 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 17 23:14:46.874372 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:14:46.874355 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3db4a6f-b531-4ebe-bcd6-f56239784525-tls-certs podName:f3db4a6f-b531-4ebe-bcd6-f56239784525 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:47.374340225 +0000 UTC m=+526.308247339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f3db4a6f-b531-4ebe-bcd6-f56239784525-tls-certs") pod "kube-auth-proxy-d4c67b97b-lcbjm" (UID: "f3db4a6f-b531-4ebe-bcd6-f56239784525") : secret "kube-auth-proxy-tls" not found Sep 17 23:14:46.876401 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:46.876376 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3db4a6f-b531-4ebe-bcd6-f56239784525-tmp\") pod \"kube-auth-proxy-d4c67b97b-lcbjm\" (UID: \"f3db4a6f-b531-4ebe-bcd6-f56239784525\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-lcbjm" Sep 17 23:14:46.907903 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:46.907878 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlqtp\" (UniqueName: \"kubernetes.io/projected/f3db4a6f-b531-4ebe-bcd6-f56239784525-kube-api-access-rlqtp\") pod \"kube-auth-proxy-d4c67b97b-lcbjm\" (UID: \"f3db4a6f-b531-4ebe-bcd6-f56239784525\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-lcbjm" Sep 17 23:14:47.378949 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:47.378905 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3db4a6f-b531-4ebe-bcd6-f56239784525-tls-certs\") pod \"kube-auth-proxy-d4c67b97b-lcbjm\" (UID: \"f3db4a6f-b531-4ebe-bcd6-f56239784525\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-lcbjm" Sep 17 23:14:47.379114 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:14:47.379050 2582 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 17 23:14:47.379114 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:14:47.379111 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3db4a6f-b531-4ebe-bcd6-f56239784525-tls-certs podName:f3db4a6f-b531-4ebe-bcd6-f56239784525 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:48.379095838 +0000 UTC m=+527.313002965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f3db4a6f-b531-4ebe-bcd6-f56239784525-tls-certs") pod "kube-auth-proxy-d4c67b97b-lcbjm" (UID: "f3db4a6f-b531-4ebe-bcd6-f56239784525") : secret "kube-auth-proxy-tls" not found Sep 17 23:14:48.095718 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.095684 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc"] Sep 17 23:14:48.099067 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.099046 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.101138 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.101111 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 17 23:14:48.101427 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.101408 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 17 23:14:48.101511 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.101484 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 17 23:14:48.101571 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.101531 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6t4sq\"" Sep 17 23:14:48.101755 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.101741 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 17 23:14:48.113509 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.113477 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc"] Sep 17 23:14:48.186449 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.186421 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/51ba5947-19fc-4d88-9914-72ebb223e1a0-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.186449 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.186450 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65fqn\" (UniqueName: \"kubernetes.io/projected/51ba5947-19fc-4d88-9914-72ebb223e1a0-kube-api-access-65fqn\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.186640 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.186541 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.186640 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.186575 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.186718 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.186662 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.186718 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.186690 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ba5947-19fc-4d88-9914-72ebb223e1a0-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.186718 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.186712 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.287219 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.287195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.287329 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.287221 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.287329 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.287249 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.287329 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.287300 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ba5947-19fc-4d88-9914-72ebb223e1a0-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.287427 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.287338 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.287427 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.287394 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/51ba5947-19fc-4d88-9914-72ebb223e1a0-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.287427 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.287416 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65fqn\" (UniqueName: \"kubernetes.io/projected/51ba5947-19fc-4d88-9914-72ebb223e1a0-kube-api-access-65fqn\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.287880 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.287852 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.290021 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.289998 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ba5947-19fc-4d88-9914-72ebb223e1a0-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.290217 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.290199 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/51ba5947-19fc-4d88-9914-72ebb223e1a0-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.290278 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.290215 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.290278 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.290261 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.295020 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.295002 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65fqn\" (UniqueName: \"kubernetes.io/projected/51ba5947-19fc-4d88-9914-72ebb223e1a0-kube-api-access-65fqn\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.295569 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.295551 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-phwhc\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.388610 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.388558 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3db4a6f-b531-4ebe-bcd6-f56239784525-tls-certs\") pod \"kube-auth-proxy-d4c67b97b-lcbjm\" (UID: \"f3db4a6f-b531-4ebe-bcd6-f56239784525\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-lcbjm" Sep 17 23:14:48.390553 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.390533 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3db4a6f-b531-4ebe-bcd6-f56239784525-tls-certs\") pod \"kube-auth-proxy-d4c67b97b-lcbjm\" (UID: \"f3db4a6f-b531-4ebe-bcd6-f56239784525\") " pod="openshift-ingress/kube-auth-proxy-d4c67b97b-lcbjm" Sep 17 23:14:48.408455 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.408434 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:48.533285 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.533256 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d4c67b97b-lcbjm" Sep 17 23:14:48.681780 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.681758 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc"] Sep 17 23:14:48.684475 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:14:48.684440 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ba5947_19fc_4d88_9914_72ebb223e1a0.slice/crio-309e14bf4439aa910ddec39c9d8282966d92b7150c269b20766a1118c7746d5a WatchSource:0}: Error finding container 309e14bf4439aa910ddec39c9d8282966d92b7150c269b20766a1118c7746d5a: Status 404 returned error can't find the container with id 309e14bf4439aa910ddec39c9d8282966d92b7150c269b20766a1118c7746d5a Sep 17 23:14:48.802247 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:48.802224 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d4c67b97b-lcbjm"] Sep 17 23:14:48.804702 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:14:48.804668 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3db4a6f_b531_4ebe_bcd6_f56239784525.slice/crio-5636660eb4ff8ea316731167647da72d5e157fb65307326addea64b24e5207f5 WatchSource:0}: Error finding container 5636660eb4ff8ea316731167647da72d5e157fb65307326addea64b24e5207f5: Status 404 returned error can't find the container with id 5636660eb4ff8ea316731167647da72d5e157fb65307326addea64b24e5207f5 Sep 17 23:14:49.302029 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:49.301989 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d4c67b97b-lcbjm" event={"ID":"f3db4a6f-b531-4ebe-bcd6-f56239784525","Type":"ContainerStarted","Data":"5636660eb4ff8ea316731167647da72d5e157fb65307326addea64b24e5207f5"} Sep 17 23:14:49.303453 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:49.303422 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" event={"ID":"51ba5947-19fc-4d88-9914-72ebb223e1a0","Type":"ContainerStarted","Data":"309e14bf4439aa910ddec39c9d8282966d92b7150c269b20766a1118c7746d5a"} Sep 17 23:14:52.495248 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:52.495200 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:14:52.495620 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:52.495291 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:14:53.331639 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:53.331602 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d4c67b97b-lcbjm" event={"ID":"f3db4a6f-b531-4ebe-bcd6-f56239784525","Type":"ContainerStarted","Data":"4fae3907e71166b65594227172711569ec3656c6545e147fc586efb10e7df83e"} Sep 17 23:14:53.333200 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:53.333172 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" event={"ID":"51ba5947-19fc-4d88-9914-72ebb223e1a0","Type":"ContainerStarted","Data":"5332ded9e27f1e01c3025c413a389c73e860790e7675296dd64448752e7d3db5"} Sep 17 23:14:53.333340 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:53.333222 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:53.335222 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:53.335197 2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-phwhc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 23:14:53.335317 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:53.335246 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" podUID="51ba5947-19fc-4d88-9914-72ebb223e1a0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 23:14:53.349326 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:53.349284 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d4c67b97b-lcbjm" podStartSLOduration=3.6599107 podStartE2EDuration="7.349269774s" totalImagesPullingTime="3.689359074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:53.347265941 +0000 UTC m=+532.281173080" watchObservedRunningTime="2026-09-17 23:14:53.349269774 +0000 UTC m=+532.283176910" Sep 17 23:14:53.369395 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:53.369350 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" podStartSLOduration=1.560808337 podStartE2EDuration="5.369330906s" totalImagesPullingTime="3.808522569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:53.368646232 +0000 UTC m=+532.302553367" watchObservedRunningTime="2026-09-17 23:14:53.369330906 +0000 UTC m=+532.303238043" Sep 17 23:14:54.337453 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:54.337425 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:14:57.858429 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:57.858396 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-k7l6b"] Sep 17 23:14:57.863233 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:57.863210 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-k7l6b" Sep 17 23:14:57.865433 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:57.865414 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 17 23:14:57.875679 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:57.875657 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-k7l6b"] Sep 17 23:14:57.977017 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:57.976983 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c004f0a-2484-466f-8a7d-1f34096f8080-tls-certs\") pod \"model-serving-api-54d89bd5fc-k7l6b\" (UID: \"7c004f0a-2484-466f-8a7d-1f34096f8080\") " pod="opendatahub/model-serving-api-54d89bd5fc-k7l6b" Sep 17 23:14:57.977197 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:57.977104 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqv9x\" (UniqueName: \"kubernetes.io/projected/7c004f0a-2484-466f-8a7d-1f34096f8080-kube-api-access-vqv9x\") pod \"model-serving-api-54d89bd5fc-k7l6b\" (UID: \"7c004f0a-2484-466f-8a7d-1f34096f8080\") " pod="opendatahub/model-serving-api-54d89bd5fc-k7l6b" Sep 17 23:14:58.078178 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:58.078137 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqv9x\" (UniqueName: \"kubernetes.io/projected/7c004f0a-2484-466f-8a7d-1f34096f8080-kube-api-access-vqv9x\") pod \"model-serving-api-54d89bd5fc-k7l6b\" (UID: \"7c004f0a-2484-466f-8a7d-1f34096f8080\") " pod="opendatahub/model-serving-api-54d89bd5fc-k7l6b" Sep 17 23:14:58.078354 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:58.078205 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c004f0a-2484-466f-8a7d-1f34096f8080-tls-certs\") pod \"model-serving-api-54d89bd5fc-k7l6b\" (UID: \"7c004f0a-2484-466f-8a7d-1f34096f8080\") " pod="opendatahub/model-serving-api-54d89bd5fc-k7l6b" Sep 17 23:14:58.080733 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:58.080709 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c004f0a-2484-466f-8a7d-1f34096f8080-tls-certs\") pod \"model-serving-api-54d89bd5fc-k7l6b\" (UID: \"7c004f0a-2484-466f-8a7d-1f34096f8080\") " pod="opendatahub/model-serving-api-54d89bd5fc-k7l6b" Sep 17 23:14:58.087126 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:58.087098 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqv9x\" (UniqueName: \"kubernetes.io/projected/7c004f0a-2484-466f-8a7d-1f34096f8080-kube-api-access-vqv9x\") pod \"model-serving-api-54d89bd5fc-k7l6b\" (UID: \"7c004f0a-2484-466f-8a7d-1f34096f8080\") " pod="opendatahub/model-serving-api-54d89bd5fc-k7l6b" Sep 17 23:14:58.173323 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:58.173241 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-k7l6b" Sep 17 23:14:58.451798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:58.451725 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-k7l6b"] Sep 17 23:14:58.455203 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:14:58.455177 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c004f0a_2484_466f_8a7d_1f34096f8080.slice/crio-1b3eb7692a4bb23c53a48af42a3d1cfaa355e092f34b214e033616b02ff8116b WatchSource:0}: Error finding container 1b3eb7692a4bb23c53a48af42a3d1cfaa355e092f34b214e033616b02ff8116b: Status 404 returned error can't find the container with id 1b3eb7692a4bb23c53a48af42a3d1cfaa355e092f34b214e033616b02ff8116b Sep 17 23:14:59.363356 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:14:59.363315 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-k7l6b" event={"ID":"7c004f0a-2484-466f-8a7d-1f34096f8080","Type":"ContainerStarted","Data":"1b3eb7692a4bb23c53a48af42a3d1cfaa355e092f34b214e033616b02ff8116b"} Sep 17 23:15:01.374465 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:01.374431 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-k7l6b" event={"ID":"7c004f0a-2484-466f-8a7d-1f34096f8080","Type":"ContainerStarted","Data":"7bc0354a749daa78c827d3543b2de1bb5ff758b1361b52e3f0ad01fc1782f752"} Sep 17 23:15:01.374895 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:01.374539 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-k7l6b" Sep 17 23:15:01.396697 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:01.396650 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-k7l6b" podStartSLOduration=2.512832935 podStartE2EDuration="4.396634616s" totalImagesPullingTime="1.883801681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:01.395591109 +0000 UTC m=+540.329498244" watchObservedRunningTime="2026-09-17 23:15:01.396634616 +0000 UTC m=+540.330541751" Sep 17 23:15:06.539899 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.539864 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9gdrq"] Sep 17 23:15:06.558445 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.558413 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gdrq" Sep 17 23:15:06.562819 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.562793 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-btzx4\"" Sep 17 23:15:06.563361 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.563319 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 17 23:15:06.564749 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.564719 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9gdrq"] Sep 17 23:15:06.571596 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.571571 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lw5wt"] Sep 17 23:15:06.575021 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.575001 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lw5wt" Sep 17 23:15:06.582837 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.582562 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 17 23:15:06.582837 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.582587 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-bzblg\"" Sep 17 23:15:06.594735 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.594711 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lw5wt"] Sep 17 23:15:06.660463 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.660422 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dad3084b-e040-483e-9c23-171a0573347c-cert\") pod \"kserve-controller-manager-d6f4b65b8-9gdrq\" (UID: \"dad3084b-e040-483e-9c23-171a0573347c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gdrq" Sep 17 23:15:06.660618 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.660467 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp5nc\" (UniqueName: \"kubernetes.io/projected/dad3084b-e040-483e-9c23-171a0573347c-kube-api-access-zp5nc\") pod \"kserve-controller-manager-d6f4b65b8-9gdrq\" (UID: \"dad3084b-e040-483e-9c23-171a0573347c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gdrq" Sep 17 23:15:06.660618 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.660540 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd805404-17a7-41b6-9972-b3570a138562-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lw5wt\" (UID: \"dd805404-17a7-41b6-9972-b3570a138562\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lw5wt" Sep 17 23:15:06.660686 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.660622 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grfdz\" (UniqueName: \"kubernetes.io/projected/dd805404-17a7-41b6-9972-b3570a138562-kube-api-access-grfdz\") pod \"llmisvc-controller-manager-777c6bd7d5-lw5wt\" (UID: \"dd805404-17a7-41b6-9972-b3570a138562\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lw5wt" Sep 17 23:15:06.761130 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.761101 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dad3084b-e040-483e-9c23-171a0573347c-cert\") pod \"kserve-controller-manager-d6f4b65b8-9gdrq\" (UID: \"dad3084b-e040-483e-9c23-171a0573347c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gdrq" Sep 17 23:15:06.761271 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.761137 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp5nc\" (UniqueName: \"kubernetes.io/projected/dad3084b-e040-483e-9c23-171a0573347c-kube-api-access-zp5nc\") pod \"kserve-controller-manager-d6f4b65b8-9gdrq\" (UID: \"dad3084b-e040-483e-9c23-171a0573347c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gdrq" Sep 17 23:15:06.761271 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.761193 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd805404-17a7-41b6-9972-b3570a138562-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lw5wt\" (UID: \"dd805404-17a7-41b6-9972-b3570a138562\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lw5wt" Sep 17 23:15:06.761271 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.761239 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grfdz\" (UniqueName: \"kubernetes.io/projected/dd805404-17a7-41b6-9972-b3570a138562-kube-api-access-grfdz\") pod \"llmisvc-controller-manager-777c6bd7d5-lw5wt\" (UID: \"dd805404-17a7-41b6-9972-b3570a138562\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lw5wt" Sep 17 23:15:06.761519 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:15:06.761331 2582 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 17 23:15:06.761519 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:15:06.761409 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd805404-17a7-41b6-9972-b3570a138562-cert podName:dd805404-17a7-41b6-9972-b3570a138562 nodeName:}" failed. No retries permitted until 2026-09-17 23:15:07.261389617 +0000 UTC m=+546.195296732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/dd805404-17a7-41b6-9972-b3570a138562-cert") pod "llmisvc-controller-manager-777c6bd7d5-lw5wt" (UID: "dd805404-17a7-41b6-9972-b3570a138562") : secret "llmisvc-webhook-server-cert" not found Sep 17 23:15:06.763639 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.763612 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dad3084b-e040-483e-9c23-171a0573347c-cert\") pod \"kserve-controller-manager-d6f4b65b8-9gdrq\" (UID: \"dad3084b-e040-483e-9c23-171a0573347c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gdrq" Sep 17 23:15:06.774048 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.774023 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp5nc\" (UniqueName: \"kubernetes.io/projected/dad3084b-e040-483e-9c23-171a0573347c-kube-api-access-zp5nc\") pod \"kserve-controller-manager-d6f4b65b8-9gdrq\" (UID: \"dad3084b-e040-483e-9c23-171a0573347c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gdrq" Sep 17 23:15:06.781146 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.781116 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grfdz\" (UniqueName: \"kubernetes.io/projected/dd805404-17a7-41b6-9972-b3570a138562-kube-api-access-grfdz\") pod \"llmisvc-controller-manager-777c6bd7d5-lw5wt\" (UID: \"dd805404-17a7-41b6-9972-b3570a138562\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lw5wt" Sep 17 23:15:06.872477 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:06.872419 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gdrq" Sep 17 23:15:07.141616 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:07.141591 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9gdrq"] Sep 17 23:15:07.143878 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:15:07.143848 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad3084b_e040_483e_9c23_171a0573347c.slice/crio-b45273b0849d3005b290dd9a7ef4a97af8632c8db42658e833c221aa7426a0fa WatchSource:0}: Error finding container b45273b0849d3005b290dd9a7ef4a97af8632c8db42658e833c221aa7426a0fa: Status 404 returned error can't find the container with id b45273b0849d3005b290dd9a7ef4a97af8632c8db42658e833c221aa7426a0fa Sep 17 23:15:07.266375 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:07.266324 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd805404-17a7-41b6-9972-b3570a138562-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lw5wt\" (UID: \"dd805404-17a7-41b6-9972-b3570a138562\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lw5wt" Sep 17 23:15:07.268917 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:07.268893 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd805404-17a7-41b6-9972-b3570a138562-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lw5wt\" (UID: \"dd805404-17a7-41b6-9972-b3570a138562\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lw5wt" Sep 17 23:15:07.403069 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:07.402968 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gdrq" event={"ID":"dad3084b-e040-483e-9c23-171a0573347c","Type":"ContainerStarted","Data":"b45273b0849d3005b290dd9a7ef4a97af8632c8db42658e833c221aa7426a0fa"} Sep 17 23:15:07.490044 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:07.490007 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lw5wt" Sep 17 23:15:07.775943 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:07.775908 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lw5wt"] Sep 17 23:15:07.779293 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:15:07.779266 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddd805404_17a7_41b6_9972_b3570a138562.slice/crio-258d5940f3c6610eba1d0fb3a444a81caeb49cb41660c86b2e75df9ee8c87950 WatchSource:0}: Error finding container 258d5940f3c6610eba1d0fb3a444a81caeb49cb41660c86b2e75df9ee8c87950: Status 404 returned error can't find the container with id 258d5940f3c6610eba1d0fb3a444a81caeb49cb41660c86b2e75df9ee8c87950 Sep 17 23:15:08.410565 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:08.410529 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lw5wt" event={"ID":"dd805404-17a7-41b6-9972-b3570a138562","Type":"ContainerStarted","Data":"258d5940f3c6610eba1d0fb3a444a81caeb49cb41660c86b2e75df9ee8c87950"} Sep 17 23:15:08.924174 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:08.924140 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc"] Sep 17 23:15:08.924631 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:08.924352 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" podUID="51ba5947-19fc-4d88-9914-72ebb223e1a0" containerName="discovery" containerID="cri-o://5332ded9e27f1e01c3025c413a389c73e860790e7675296dd64448752e7d3db5" gracePeriod=30 Sep 17 23:15:09.268178 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.268152 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:15:09.386104 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.386076 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-csr-dns-cert\") pod \"51ba5947-19fc-4d88-9914-72ebb223e1a0\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " Sep 17 23:15:09.386264 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.386125 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-csr-ca-configmap\") pod \"51ba5947-19fc-4d88-9914-72ebb223e1a0\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " Sep 17 23:15:09.386264 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.386154 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-kubeconfig\") pod \"51ba5947-19fc-4d88-9914-72ebb223e1a0\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " Sep 17 23:15:09.386391 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.386265 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-token\") pod \"51ba5947-19fc-4d88-9914-72ebb223e1a0\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " Sep 17 23:15:09.386391 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.386306 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ba5947-19fc-4d88-9914-72ebb223e1a0-local-certs\") pod \"51ba5947-19fc-4d88-9914-72ebb223e1a0\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " Sep 17 23:15:09.386483 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.386397 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/51ba5947-19fc-4d88-9914-72ebb223e1a0-cacerts\") pod \"51ba5947-19fc-4d88-9914-72ebb223e1a0\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " Sep 17 23:15:09.386483 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.386447 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65fqn\" (UniqueName: \"kubernetes.io/projected/51ba5947-19fc-4d88-9914-72ebb223e1a0-kube-api-access-65fqn\") pod \"51ba5947-19fc-4d88-9914-72ebb223e1a0\" (UID: \"51ba5947-19fc-4d88-9914-72ebb223e1a0\") " Sep 17 23:15:09.386664 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.386563 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "51ba5947-19fc-4d88-9914-72ebb223e1a0" (UID: "51ba5947-19fc-4d88-9914-72ebb223e1a0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:15:09.387244 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.386869 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-csr-ca-configmap\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:15:09.388629 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.388602 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ba5947-19fc-4d88-9914-72ebb223e1a0-kube-api-access-65fqn" (OuterVolumeSpecName: "kube-api-access-65fqn") pod "51ba5947-19fc-4d88-9914-72ebb223e1a0" (UID: "51ba5947-19fc-4d88-9914-72ebb223e1a0"). InnerVolumeSpecName "kube-api-access-65fqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:15:09.388756 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.388689 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "51ba5947-19fc-4d88-9914-72ebb223e1a0" (UID: "51ba5947-19fc-4d88-9914-72ebb223e1a0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:15:09.389332 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.389311 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51ba5947-19fc-4d88-9914-72ebb223e1a0-cacerts" (OuterVolumeSpecName: "cacerts") pod "51ba5947-19fc-4d88-9914-72ebb223e1a0" (UID: "51ba5947-19fc-4d88-9914-72ebb223e1a0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:15:09.389446 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.389427 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "51ba5947-19fc-4d88-9914-72ebb223e1a0" (UID: "51ba5947-19fc-4d88-9914-72ebb223e1a0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:15:09.389562 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.389461 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-token" (OuterVolumeSpecName: "istio-token") pod "51ba5947-19fc-4d88-9914-72ebb223e1a0" (UID: "51ba5947-19fc-4d88-9914-72ebb223e1a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:15:09.389562 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.389531 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ba5947-19fc-4d88-9914-72ebb223e1a0-local-certs" (OuterVolumeSpecName: "local-certs") pod "51ba5947-19fc-4d88-9914-72ebb223e1a0" (UID: "51ba5947-19fc-4d88-9914-72ebb223e1a0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:15:09.416779 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.416756 2582 generic.go:356] "Generic (PLEG): container finished" podID="51ba5947-19fc-4d88-9914-72ebb223e1a0" containerID="5332ded9e27f1e01c3025c413a389c73e860790e7675296dd64448752e7d3db5" exitCode=0 Sep 17 23:15:09.416909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.416789 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" event={"ID":"51ba5947-19fc-4d88-9914-72ebb223e1a0","Type":"ContainerDied","Data":"5332ded9e27f1e01c3025c413a389c73e860790e7675296dd64448752e7d3db5"} Sep 17 23:15:09.416909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.416811 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" Sep 17 23:15:09.416909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.416821 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc" event={"ID":"51ba5947-19fc-4d88-9914-72ebb223e1a0","Type":"ContainerDied","Data":"309e14bf4439aa910ddec39c9d8282966d92b7150c269b20766a1118c7746d5a"} Sep 17 23:15:09.416909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.416839 2582 scope.go:117] "RemoveContainer" containerID="5332ded9e27f1e01c3025c413a389c73e860790e7675296dd64448752e7d3db5" Sep 17 23:15:09.439826 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.439799 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc"] Sep 17 23:15:09.445134 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.445105 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-phwhc"] Sep 17 23:15:09.487471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.487447 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-csr-dns-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:15:09.487471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.487470 2582 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-kubeconfig\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:15:09.487688 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.487485 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ba5947-19fc-4d88-9914-72ebb223e1a0-istio-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:15:09.487688 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.487513 2582 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ba5947-19fc-4d88-9914-72ebb223e1a0-local-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:15:09.487688 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.487526 2582 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/51ba5947-19fc-4d88-9914-72ebb223e1a0-cacerts\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:15:09.487688 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.487537 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65fqn\" (UniqueName: \"kubernetes.io/projected/51ba5947-19fc-4d88-9914-72ebb223e1a0-kube-api-access-65fqn\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:15:09.559176 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.559131 2582 scope.go:117] "RemoveContainer" containerID="5332ded9e27f1e01c3025c413a389c73e860790e7675296dd64448752e7d3db5" Sep 17 23:15:09.559621 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:15:09.559592 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5332ded9e27f1e01c3025c413a389c73e860790e7675296dd64448752e7d3db5\": container with ID starting with 5332ded9e27f1e01c3025c413a389c73e860790e7675296dd64448752e7d3db5 not found: ID does not exist" containerID="5332ded9e27f1e01c3025c413a389c73e860790e7675296dd64448752e7d3db5" Sep 17 23:15:09.559733 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:09.559631 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5332ded9e27f1e01c3025c413a389c73e860790e7675296dd64448752e7d3db5"} err="failed to get container status \"5332ded9e27f1e01c3025c413a389c73e860790e7675296dd64448752e7d3db5\": rpc error: code = NotFound desc = could not find container \"5332ded9e27f1e01c3025c413a389c73e860790e7675296dd64448752e7d3db5\": container with ID starting with 5332ded9e27f1e01c3025c413a389c73e860790e7675296dd64448752e7d3db5 not found: ID does not exist" Sep 17 23:15:10.424265 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:10.424227 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gdrq" event={"ID":"dad3084b-e040-483e-9c23-171a0573347c","Type":"ContainerStarted","Data":"efda942ae3ae5aaaebe6945afccdd88540348e71cde88facd00e4f220ecaf1b5"} Sep 17 23:15:10.424764 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:10.424373 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gdrq" Sep 17 23:15:10.443805 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:10.443758 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gdrq" podStartSLOduration=2.295477557 podStartE2EDuration="4.443743457s" totalImagesPullingTime="2.1482659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:10.442785792 +0000 UTC m=+549.376692927" watchObservedRunningTime="2026-09-17 23:15:10.443743457 +0000 UTC m=+549.377650592" Sep 17 23:15:11.293617 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:11.293548 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51ba5947-19fc-4d88-9914-72ebb223e1a0" path="/var/lib/kubelet/pods/51ba5947-19fc-4d88-9914-72ebb223e1a0/volumes" Sep 17 23:15:11.431036 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:11.431004 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lw5wt" event={"ID":"dd805404-17a7-41b6-9972-b3570a138562","Type":"ContainerStarted","Data":"04e98151e7dae805b953122be4fabebef66bfb1d41217f1c3846eb2b359e4916"} Sep 17 23:15:11.431406 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:11.431136 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lw5wt" Sep 17 23:15:11.448287 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:11.448246 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lw5wt" podStartSLOduration=2.1862735620000002 podStartE2EDuration="5.448231487s" totalImagesPullingTime="3.261957925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:11.447007996 +0000 UTC m=+550.380915131" watchObservedRunningTime="2026-09-17 23:15:11.448231487 +0000 UTC m=+550.382138623" Sep 17 23:15:12.382923 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:12.382896 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-k7l6b" Sep 17 23:15:41.436372 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:41.436342 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9gdrq" Sep 17 23:15:42.439168 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:15:42.439137 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lw5wt" Sep 17 23:16:17.270928 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.270852 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bdf65cc46-s9l7f"] Sep 17 23:16:17.273062 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.271212 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51ba5947-19fc-4d88-9914-72ebb223e1a0" containerName="discovery" Sep 17 23:16:17.273062 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.271226 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="51ba5947-19fc-4d88-9914-72ebb223e1a0" containerName="discovery" Sep 17 23:16:17.273062 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.271290 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="51ba5947-19fc-4d88-9914-72ebb223e1a0" containerName="discovery" Sep 17 23:16:17.274094 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.274079 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.283766 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.283744 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bdf65cc46-s9l7f"] Sep 17 23:16:17.432959 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.432919 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64798c7a-5f3a-44f1-8aec-d4c676708015-console-serving-cert\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.433167 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.432981 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rsvs\" (UniqueName: \"kubernetes.io/projected/64798c7a-5f3a-44f1-8aec-d4c676708015-kube-api-access-2rsvs\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.433167 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.433016 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64798c7a-5f3a-44f1-8aec-d4c676708015-console-oauth-config\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.433167 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.433118 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64798c7a-5f3a-44f1-8aec-d4c676708015-service-ca\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.433353 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.433199 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64798c7a-5f3a-44f1-8aec-d4c676708015-oauth-serving-cert\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.433421 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.433358 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64798c7a-5f3a-44f1-8aec-d4c676708015-console-config\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.433421 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.433400 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64798c7a-5f3a-44f1-8aec-d4c676708015-trusted-ca-bundle\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.534719 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.534627 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64798c7a-5f3a-44f1-8aec-d4c676708015-oauth-serving-cert\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.534719 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.534674 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64798c7a-5f3a-44f1-8aec-d4c676708015-console-config\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.534719 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.534701 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64798c7a-5f3a-44f1-8aec-d4c676708015-trusted-ca-bundle\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.534994 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.534743 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64798c7a-5f3a-44f1-8aec-d4c676708015-console-serving-cert\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.534994 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.534792 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rsvs\" (UniqueName: \"kubernetes.io/projected/64798c7a-5f3a-44f1-8aec-d4c676708015-kube-api-access-2rsvs\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.534994 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.534820 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64798c7a-5f3a-44f1-8aec-d4c676708015-console-oauth-config\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.534994 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.534855 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64798c7a-5f3a-44f1-8aec-d4c676708015-service-ca\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.535440 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.535408 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64798c7a-5f3a-44f1-8aec-d4c676708015-oauth-serving-cert\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.535612 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.535467 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64798c7a-5f3a-44f1-8aec-d4c676708015-console-config\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.535612 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.535603 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64798c7a-5f3a-44f1-8aec-d4c676708015-service-ca\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.535713 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.535626 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64798c7a-5f3a-44f1-8aec-d4c676708015-trusted-ca-bundle\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.537267 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.537249 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64798c7a-5f3a-44f1-8aec-d4c676708015-console-oauth-config\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.537335 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.537315 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64798c7a-5f3a-44f1-8aec-d4c676708015-console-serving-cert\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.542224 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.542204 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rsvs\" (UniqueName: \"kubernetes.io/projected/64798c7a-5f3a-44f1-8aec-d4c676708015-kube-api-access-2rsvs\") pod \"console-6bdf65cc46-s9l7f\" (UID: \"64798c7a-5f3a-44f1-8aec-d4c676708015\") " pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.583883 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.583857 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:17.853889 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:17.853831 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bdf65cc46-s9l7f"] Sep 17 23:16:17.856540 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:16:17.856511 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64798c7a_5f3a_44f1_8aec_d4c676708015.slice/crio-cc0c04d3c42b788854249ca34286c7323220bb1f5f4b3959811f9483f3c38117 WatchSource:0}: Error finding container cc0c04d3c42b788854249ca34286c7323220bb1f5f4b3959811f9483f3c38117: Status 404 returned error can't find the container with id cc0c04d3c42b788854249ca34286c7323220bb1f5f4b3959811f9483f3c38117 Sep 17 23:16:18.014251 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:18.014219 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn"] Sep 17 23:16:18.017540 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:18.017519 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" Sep 17 23:16:18.019906 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:18.019885 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 17 23:16:18.020003 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:18.019911 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-xgj6n\"" Sep 17 23:16:18.020003 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:18.019893 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 17 23:16:18.028463 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:18.028442 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn"] Sep 17 23:16:18.141225 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:18.141112 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww5zb\" (UniqueName: \"kubernetes.io/projected/d1cf10b8-2408-45b7-a149-e769a9425175-kube-api-access-ww5zb\") pod \"limitador-operator-controller-manager-74495c69f8-m9dhn\" (UID: \"d1cf10b8-2408-45b7-a149-e769a9425175\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" Sep 17 23:16:18.242178 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:18.242146 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww5zb\" (UniqueName: \"kubernetes.io/projected/d1cf10b8-2408-45b7-a149-e769a9425175-kube-api-access-ww5zb\") pod \"limitador-operator-controller-manager-74495c69f8-m9dhn\" (UID: \"d1cf10b8-2408-45b7-a149-e769a9425175\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" Sep 17 23:16:18.260292 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:18.260263 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww5zb\" (UniqueName: \"kubernetes.io/projected/d1cf10b8-2408-45b7-a149-e769a9425175-kube-api-access-ww5zb\") pod \"limitador-operator-controller-manager-74495c69f8-m9dhn\" (UID: \"d1cf10b8-2408-45b7-a149-e769a9425175\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" Sep 17 23:16:18.328673 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:18.328625 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" Sep 17 23:16:18.603638 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:18.603612 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn"] Sep 17 23:16:18.606109 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:16:18.606075 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1cf10b8_2408_45b7_a149_e769a9425175.slice/crio-2832cb5967c903bceb9b876bfc681319bb9ccdd72da58237a871ce9be542a9fa WatchSource:0}: Error finding container 2832cb5967c903bceb9b876bfc681319bb9ccdd72da58237a871ce9be542a9fa: Status 404 returned error can't find the container with id 2832cb5967c903bceb9b876bfc681319bb9ccdd72da58237a871ce9be542a9fa Sep 17 23:16:18.730392 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:18.730357 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" event={"ID":"d1cf10b8-2408-45b7-a149-e769a9425175","Type":"ContainerStarted","Data":"2832cb5967c903bceb9b876bfc681319bb9ccdd72da58237a871ce9be542a9fa"} Sep 17 23:16:18.732102 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:18.732071 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdf65cc46-s9l7f" event={"ID":"64798c7a-5f3a-44f1-8aec-d4c676708015","Type":"ContainerStarted","Data":"aecad5cc6a36c40c6c0587dfb38dbf9eb46d05313b3c6259f8297a780a20e736"} Sep 17 23:16:18.732234 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:18.732128 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdf65cc46-s9l7f" event={"ID":"64798c7a-5f3a-44f1-8aec-d4c676708015","Type":"ContainerStarted","Data":"cc0c04d3c42b788854249ca34286c7323220bb1f5f4b3959811f9483f3c38117"} Sep 17 23:16:18.755573 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:18.755530 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bdf65cc46-s9l7f" podStartSLOduration=1.755516531 podStartE2EDuration="1.755516531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:18.754359925 +0000 UTC m=+617.688267058" watchObservedRunningTime="2026-09-17 23:16:18.755516531 +0000 UTC m=+617.689423658" Sep 17 23:16:21.750260 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:21.750229 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" event={"ID":"d1cf10b8-2408-45b7-a149-e769a9425175","Type":"ContainerStarted","Data":"6c805be9c6ef6ad5bbb321f673e38ec196663cff7268a906d914b5fdc818d705"} Sep 17 23:16:21.750741 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:21.750296 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" Sep 17 23:16:21.770708 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:21.770664 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" podStartSLOduration=2.252155287 podStartE2EDuration="4.770647209s" totalImagesPullingTime="2.518491922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:21.769112471 +0000 UTC m=+620.703019609" watchObservedRunningTime="2026-09-17 23:16:21.770647209 +0000 UTC m=+620.704554344" Sep 17 23:16:24.017698 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:24.017663 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh"] Sep 17 23:16:24.021246 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:24.021225 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:24.023868 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:24.023847 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hgccr\"" Sep 17 23:16:24.042322 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:24.042302 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh"] Sep 17 23:16:24.093464 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:24.093438 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a35567f-c261-4ccb-b734-ded284fc5ec9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kgvgh\" (UID: \"4a35567f-c261-4ccb-b734-ded284fc5ec9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:24.093587 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:24.093489 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlfvz\" (UniqueName: \"kubernetes.io/projected/4a35567f-c261-4ccb-b734-ded284fc5ec9-kube-api-access-tlfvz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kgvgh\" (UID: \"4a35567f-c261-4ccb-b734-ded284fc5ec9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:24.194717 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:24.194684 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlfvz\" (UniqueName: \"kubernetes.io/projected/4a35567f-c261-4ccb-b734-ded284fc5ec9-kube-api-access-tlfvz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kgvgh\" (UID: \"4a35567f-c261-4ccb-b734-ded284fc5ec9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:24.194837 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:24.194753 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a35567f-c261-4ccb-b734-ded284fc5ec9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kgvgh\" (UID: \"4a35567f-c261-4ccb-b734-ded284fc5ec9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:24.195051 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:24.195034 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a35567f-c261-4ccb-b734-ded284fc5ec9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kgvgh\" (UID: \"4a35567f-c261-4ccb-b734-ded284fc5ec9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:24.202995 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:24.202968 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlfvz\" (UniqueName: \"kubernetes.io/projected/4a35567f-c261-4ccb-b734-ded284fc5ec9-kube-api-access-tlfvz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kgvgh\" (UID: \"4a35567f-c261-4ccb-b734-ded284fc5ec9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:24.331525 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:24.331436 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:24.604104 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:24.604080 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh"] Sep 17 23:16:24.606627 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:16:24.606600 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a35567f_c261_4ccb_b734_ded284fc5ec9.slice/crio-0451d02db0549e66df550d2a88b7cb3e31764e7f8d32355a68a6588d9aead73a WatchSource:0}: Error finding container 0451d02db0549e66df550d2a88b7cb3e31764e7f8d32355a68a6588d9aead73a: Status 404 returned error can't find the container with id 0451d02db0549e66df550d2a88b7cb3e31764e7f8d32355a68a6588d9aead73a Sep 17 23:16:24.764871 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:24.764838 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" event={"ID":"4a35567f-c261-4ccb-b734-ded284fc5ec9","Type":"ContainerStarted","Data":"0451d02db0549e66df550d2a88b7cb3e31764e7f8d32355a68a6588d9aead73a"} Sep 17 23:16:27.496493 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:27.496464 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-ztvbq"] Sep 17 23:16:27.500541 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:27.500515 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-ztvbq" Sep 17 23:16:27.502878 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:27.502857 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-7tqv9\"" Sep 17 23:16:27.511216 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:27.511160 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-ztvbq"] Sep 17 23:16:27.523833 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:27.523812 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nfvl\" (UniqueName: \"kubernetes.io/projected/241122b8-292b-4710-96f6-8d0419e36c88-kube-api-access-7nfvl\") pod \"authorino-operator-76b478bd78-ztvbq\" (UID: \"241122b8-292b-4710-96f6-8d0419e36c88\") " pod="kuadrant-system/authorino-operator-76b478bd78-ztvbq" Sep 17 23:16:27.584078 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:27.584046 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:27.584222 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:27.584110 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:27.589187 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:27.589124 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:27.624721 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:27.624693 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nfvl\" (UniqueName: \"kubernetes.io/projected/241122b8-292b-4710-96f6-8d0419e36c88-kube-api-access-7nfvl\") pod \"authorino-operator-76b478bd78-ztvbq\" (UID: \"241122b8-292b-4710-96f6-8d0419e36c88\") " pod="kuadrant-system/authorino-operator-76b478bd78-ztvbq" Sep 17 23:16:27.632911 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:27.632890 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nfvl\" (UniqueName: \"kubernetes.io/projected/241122b8-292b-4710-96f6-8d0419e36c88-kube-api-access-7nfvl\") pod \"authorino-operator-76b478bd78-ztvbq\" (UID: \"241122b8-292b-4710-96f6-8d0419e36c88\") " pod="kuadrant-system/authorino-operator-76b478bd78-ztvbq" Sep 17 23:16:27.786262 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:27.786178 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bdf65cc46-s9l7f" Sep 17 23:16:27.814401 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:27.814293 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-ztvbq" Sep 17 23:16:27.843513 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:27.843481 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65bdd89887-4xl2h"] Sep 17 23:16:28.499739 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:28.499646 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-ztvbq"] Sep 17 23:16:28.502398 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:16:28.502368 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241122b8_292b_4710_96f6_8d0419e36c88.slice/crio-267bdf42c7685442266dd81d70bf34aef29ec77f53f2d44e17e90052435bc99e WatchSource:0}: Error finding container 267bdf42c7685442266dd81d70bf34aef29ec77f53f2d44e17e90052435bc99e: Status 404 returned error can't find the container with id 267bdf42c7685442266dd81d70bf34aef29ec77f53f2d44e17e90052435bc99e Sep 17 23:16:28.788045 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:28.787957 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-ztvbq" event={"ID":"241122b8-292b-4710-96f6-8d0419e36c88","Type":"ContainerStarted","Data":"267bdf42c7685442266dd81d70bf34aef29ec77f53f2d44e17e90052435bc99e"} Sep 17 23:16:28.789643 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:28.789615 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" event={"ID":"4a35567f-c261-4ccb-b734-ded284fc5ec9","Type":"ContainerStarted","Data":"ac771c29c72348dcca5ffac2aeccf1f82f1ca625e7862ee390c99a4bcd525d51"} Sep 17 23:16:28.789786 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:28.789664 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:28.814249 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:28.814206 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" podStartSLOduration=2.120406121 podStartE2EDuration="5.81419198s" totalImagesPullingTime="3.693785859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:28.812848308 +0000 UTC m=+627.746755445" watchObservedRunningTime="2026-09-17 23:16:28.81419198 +0000 UTC m=+627.748099116" Sep 17 23:16:30.684294 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:30.684211 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pqwsr"] Sep 17 23:16:30.687738 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:30.687719 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pqwsr" Sep 17 23:16:30.689904 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:30.689878 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 17 23:16:30.689996 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:30.689954 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-4ldl7\"" Sep 17 23:16:30.699606 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:30.699584 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pqwsr"] Sep 17 23:16:30.754292 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:30.754259 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtzgm\" (UniqueName: \"kubernetes.io/projected/853502d1-76ef-4606-a89d-57cde294e9fc-kube-api-access-rtzgm\") pod \"dns-operator-controller-manager-7b84cd7c9c-pqwsr\" (UID: \"853502d1-76ef-4606-a89d-57cde294e9fc\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pqwsr" Sep 17 23:16:30.801896 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:30.801865 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-ztvbq" event={"ID":"241122b8-292b-4710-96f6-8d0419e36c88","Type":"ContainerStarted","Data":"c0aabde7c0b2d026fa17d3e5f30b8b07b6bb062fa15bf5a009962de5befec423"} Sep 17 23:16:30.802029 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:30.802000 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-ztvbq" Sep 17 23:16:30.827426 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:30.827383 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-ztvbq" podStartSLOduration=2.033634974 podStartE2EDuration="3.827368937s" totalImagesPullingTime="1.793733963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:30.825047815 +0000 UTC m=+629.758954950" watchObservedRunningTime="2026-09-17 23:16:30.827368937 +0000 UTC m=+629.761276072" Sep 17 23:16:30.855046 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:30.855019 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtzgm\" (UniqueName: \"kubernetes.io/projected/853502d1-76ef-4606-a89d-57cde294e9fc-kube-api-access-rtzgm\") pod \"dns-operator-controller-manager-7b84cd7c9c-pqwsr\" (UID: \"853502d1-76ef-4606-a89d-57cde294e9fc\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pqwsr" Sep 17 23:16:30.863075 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:30.863047 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtzgm\" (UniqueName: \"kubernetes.io/projected/853502d1-76ef-4606-a89d-57cde294e9fc-kube-api-access-rtzgm\") pod \"dns-operator-controller-manager-7b84cd7c9c-pqwsr\" (UID: \"853502d1-76ef-4606-a89d-57cde294e9fc\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pqwsr" Sep 17 23:16:30.997192 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:30.997142 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pqwsr" Sep 17 23:16:31.272341 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:31.272292 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pqwsr"] Sep 17 23:16:31.274685 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:16:31.274656 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853502d1_76ef_4606_a89d_57cde294e9fc.slice/crio-0ae2c291ed24059a7ce212ad080cffa81d2c9d05ff538ea5c6ced6f0379b464c WatchSource:0}: Error finding container 0ae2c291ed24059a7ce212ad080cffa81d2c9d05ff538ea5c6ced6f0379b464c: Status 404 returned error can't find the container with id 0ae2c291ed24059a7ce212ad080cffa81d2c9d05ff538ea5c6ced6f0379b464c Sep 17 23:16:31.808275 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:31.808237 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pqwsr" event={"ID":"853502d1-76ef-4606-a89d-57cde294e9fc","Type":"ContainerStarted","Data":"0ae2c291ed24059a7ce212ad080cffa81d2c9d05ff538ea5c6ced6f0379b464c"} Sep 17 23:16:32.757896 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:32.757868 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" Sep 17 23:16:33.820199 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:33.820160 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pqwsr" event={"ID":"853502d1-76ef-4606-a89d-57cde294e9fc","Type":"ContainerStarted","Data":"72a453a08ed86d263afa13b7b87d9e92516f1ec7e7564d53aaa6100f81b88f64"} Sep 17 23:16:33.820594 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:33.820279 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pqwsr" Sep 17 23:16:33.837365 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:33.837319 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pqwsr" podStartSLOduration=1.9261716770000001 podStartE2EDuration="3.837305006s" totalImagesPullingTime="1.911133329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:33.835699697 +0000 UTC m=+632.769606833" watchObservedRunningTime="2026-09-17 23:16:33.837305006 +0000 UTC m=+632.771212142" Sep 17 23:16:39.799333 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:39.799300 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:41.402591 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.402558 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6"] Sep 17 23:16:41.406077 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.406061 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6" Sep 17 23:16:41.422745 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.422722 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6"] Sep 17 23:16:41.445832 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.445808 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgk7w\" (UniqueName: \"kubernetes.io/projected/481b5fcb-59a6-4586-9776-ec6fff353a43-kube-api-access-fgk7w\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-q4jb6\" (UID: \"481b5fcb-59a6-4586-9776-ec6fff353a43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6" Sep 17 23:16:41.445955 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.445860 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/481b5fcb-59a6-4586-9776-ec6fff353a43-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-q4jb6\" (UID: \"481b5fcb-59a6-4586-9776-ec6fff353a43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6" Sep 17 23:16:41.547097 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.547068 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgk7w\" (UniqueName: \"kubernetes.io/projected/481b5fcb-59a6-4586-9776-ec6fff353a43-kube-api-access-fgk7w\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-q4jb6\" (UID: \"481b5fcb-59a6-4586-9776-ec6fff353a43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6" Sep 17 23:16:41.547235 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.547120 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/481b5fcb-59a6-4586-9776-ec6fff353a43-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-q4jb6\" (UID: \"481b5fcb-59a6-4586-9776-ec6fff353a43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6" Sep 17 23:16:41.547479 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.547463 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/481b5fcb-59a6-4586-9776-ec6fff353a43-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-q4jb6\" (UID: \"481b5fcb-59a6-4586-9776-ec6fff353a43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6" Sep 17 23:16:41.556622 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.556606 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgk7w\" (UniqueName: \"kubernetes.io/projected/481b5fcb-59a6-4586-9776-ec6fff353a43-kube-api-access-fgk7w\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-q4jb6\" (UID: \"481b5fcb-59a6-4586-9776-ec6fff353a43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6" Sep 17 23:16:41.715777 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.715711 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6" Sep 17 23:16:41.810356 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.810331 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-ztvbq" Sep 17 23:16:41.863480 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.863452 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh"] Sep 17 23:16:41.863627 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.863603 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" podUID="4a35567f-c261-4ccb-b734-ded284fc5ec9" containerName="manager" containerID="cri-o://ac771c29c72348dcca5ffac2aeccf1f82f1ca625e7862ee390c99a4bcd525d51" gracePeriod=2 Sep 17 23:16:41.890368 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.890345 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh"] Sep 17 23:16:41.894078 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:41.894048 2582 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"4a35567f-c261-4ccb-b734-ded284fc5ec9\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-kgvgh\": pods \"kuadrant-operator-controller-manager-76f8f686bf-kgvgh\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:41.921839 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.921815 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6"] Sep 17 23:16:41.943817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.943792 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6"] Sep 17 23:16:41.945690 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.945667 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs"] Sep 17 23:16:41.946201 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.946184 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a35567f-c261-4ccb-b734-ded284fc5ec9" containerName="manager" Sep 17 23:16:41.946201 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.946203 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a35567f-c261-4ccb-b734-ded284fc5ec9" containerName="manager" Sep 17 23:16:41.946349 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.946282 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a35567f-c261-4ccb-b734-ded284fc5ec9" containerName="manager" Sep 17 23:16:41.949308 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.949288 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs" Sep 17 23:16:41.951584 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:41.951550 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kgvgh\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="4a35567f-c261-4ccb-b734-ded284fc5ec9" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:41.964220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.964190 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn"] Sep 17 23:16:41.964426 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.964404 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" podUID="d1cf10b8-2408-45b7-a149-e769a9425175" containerName="manager" containerID="cri-o://6c805be9c6ef6ad5bbb321f673e38ec196663cff7268a906d914b5fdc818d705" gracePeriod=2 Sep 17 23:16:41.966431 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.966376 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5"] Sep 17 23:16:41.971225 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.970363 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs"] Sep 17 23:16:41.971225 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.970540 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5" Sep 17 23:16:41.983556 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.983465 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn"] Sep 17 23:16:41.989052 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.989028 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5"] Sep 17 23:16:41.999781 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:41.999755 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-96vl8"] Sep 17 23:16:42.000220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.000205 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1cf10b8-2408-45b7-a149-e769a9425175" containerName="manager" Sep 17 23:16:42.000286 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.000222 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1cf10b8-2408-45b7-a149-e769a9425175" containerName="manager" Sep 17 23:16:42.000324 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.000285 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1cf10b8-2408-45b7-a149-e769a9425175" containerName="manager" Sep 17 23:16:42.003202 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.003189 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-96vl8" Sep 17 23:16:42.023488 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.023460 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-96vl8"] Sep 17 23:16:42.049010 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:42.048984 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kgvgh\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="4a35567f-c261-4ccb-b734-ded284fc5ec9" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:42.051403 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:42.051355 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-m9dhn\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="d1cf10b8-2408-45b7-a149-e769a9425175" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" Sep 17 23:16:42.051929 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.051903 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfv6t\" (UniqueName: \"kubernetes.io/projected/9bcd6410-0446-4ec1-b3e9-ec03c208a4d8-kube-api-access-rfv6t\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9shqs\" (UID: \"9bcd6410-0446-4ec1-b3e9-ec03c208a4d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs" Sep 17 23:16:42.052019 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.051955 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/344908c0-c919-480e-82cf-9e4ab4451ae5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-dz4j5\" (UID: \"344908c0-c919-480e-82cf-9e4ab4451ae5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5" Sep 17 23:16:42.052098 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.052028 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9bcd6410-0446-4ec1-b3e9-ec03c208a4d8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9shqs\" (UID: \"9bcd6410-0446-4ec1-b3e9-ec03c208a4d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs" Sep 17 23:16:42.052098 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.052057 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2djz\" (UniqueName: \"kubernetes.io/projected/00282dde-9ace-4275-bc1e-08edf677ef20-kube-api-access-s2djz\") pod \"limitador-operator-controller-manager-74495c69f8-96vl8\" (UID: \"00282dde-9ace-4275-bc1e-08edf677ef20\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-96vl8" Sep 17 23:16:42.052210 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.052099 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm2hw\" (UniqueName: \"kubernetes.io/projected/344908c0-c919-480e-82cf-9e4ab4451ae5-kube-api-access-xm2hw\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-dz4j5\" (UID: \"344908c0-c919-480e-82cf-9e4ab4451ae5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5" Sep 17 23:16:42.125808 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.125782 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:42.128030 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:42.128001 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kgvgh\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="4a35567f-c261-4ccb-b734-ded284fc5ec9" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:42.129621 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:42.129598 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-m9dhn\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="d1cf10b8-2408-45b7-a149-e769a9425175" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" Sep 17 23:16:42.148731 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:42.148702 2582 log.go:32] "RunPodSandbox from runtime service failed" err=< Sep 17 23:16:42.148731 ip-10-0-143-82 kubenswrapper[2582]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-q4jb6_kuadrant-system_481b5fcb-59a6-4586-9776-ec6fff353a43_0(8b727df90c8f986094b822445c9369af52dece5ead4d8b2582a70da12698a7d6): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-q4jb6 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"8b727df90c8f986094b822445c9369af52dece5ead4d8b2582a70da12698a7d6" Netns:"/var/run/netns/9a067602-6d88-4a1c-8c14-514654dee04d" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-q4jb6;K8S_POD_INFRA_CONTAINER_ID=8b727df90c8f986094b822445c9369af52dece5ead4d8b2582a70da12698a7d6;K8S_POD_UID=481b5fcb-59a6-4586-9776-ec6fff353a43" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6/481b5fcb-59a6-4586-9776-ec6fff353a43:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6 8b727df90c8f986094b822445c9369af52dece5ead4d8b2582a70da12698a7d6 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:85:00:23 [10.133.0.35/23] Sep 17 23:16:42.148731 ip-10-0-143-82 kubenswrapper[2582]: ' Sep 17 23:16:42.148731 ip-10-0-143-82 kubenswrapper[2582]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 17 23:16:42.148731 ip-10-0-143-82 kubenswrapper[2582]: > Sep 17 23:16:42.148917 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:42.148754 2582 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Sep 17 23:16:42.148917 ip-10-0-143-82 kubenswrapper[2582]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-q4jb6_kuadrant-system_481b5fcb-59a6-4586-9776-ec6fff353a43_0(8b727df90c8f986094b822445c9369af52dece5ead4d8b2582a70da12698a7d6): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-q4jb6 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"8b727df90c8f986094b822445c9369af52dece5ead4d8b2582a70da12698a7d6" Netns:"/var/run/netns/9a067602-6d88-4a1c-8c14-514654dee04d" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-q4jb6;K8S_POD_INFRA_CONTAINER_ID=8b727df90c8f986094b822445c9369af52dece5ead4d8b2582a70da12698a7d6;K8S_POD_UID=481b5fcb-59a6-4586-9776-ec6fff353a43" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6/481b5fcb-59a6-4586-9776-ec6fff353a43:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6 8b727df90c8f986094b822445c9369af52dece5ead4d8b2582a70da12698a7d6 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:85:00:23 [10.133.0.35/23] Sep 17 23:16:42.148917 ip-10-0-143-82 kubenswrapper[2582]: ' Sep 17 23:16:42.148917 ip-10-0-143-82 kubenswrapper[2582]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 17 23:16:42.148917 ip-10-0-143-82 kubenswrapper[2582]: > pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6" Sep 17 23:16:42.153098 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.153074 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfv6t\" (UniqueName: \"kubernetes.io/projected/9bcd6410-0446-4ec1-b3e9-ec03c208a4d8-kube-api-access-rfv6t\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9shqs\" (UID: \"9bcd6410-0446-4ec1-b3e9-ec03c208a4d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs" Sep 17 23:16:42.153224 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.153116 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/344908c0-c919-480e-82cf-9e4ab4451ae5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-dz4j5\" (UID: \"344908c0-c919-480e-82cf-9e4ab4451ae5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5" Sep 17 23:16:42.153224 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.153195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9bcd6410-0446-4ec1-b3e9-ec03c208a4d8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9shqs\" (UID: \"9bcd6410-0446-4ec1-b3e9-ec03c208a4d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs" Sep 17 23:16:42.153224 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.153220 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2djz\" (UniqueName: \"kubernetes.io/projected/00282dde-9ace-4275-bc1e-08edf677ef20-kube-api-access-s2djz\") pod \"limitador-operator-controller-manager-74495c69f8-96vl8\" (UID: \"00282dde-9ace-4275-bc1e-08edf677ef20\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-96vl8" Sep 17 23:16:42.153380 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.153249 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm2hw\" (UniqueName: \"kubernetes.io/projected/344908c0-c919-480e-82cf-9e4ab4451ae5-kube-api-access-xm2hw\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-dz4j5\" (UID: \"344908c0-c919-480e-82cf-9e4ab4451ae5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5" Sep 17 23:16:42.153644 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.153620 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/344908c0-c919-480e-82cf-9e4ab4451ae5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-dz4j5\" (UID: \"344908c0-c919-480e-82cf-9e4ab4451ae5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5" Sep 17 23:16:42.153724 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.153626 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9bcd6410-0446-4ec1-b3e9-ec03c208a4d8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9shqs\" (UID: \"9bcd6410-0446-4ec1-b3e9-ec03c208a4d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs" Sep 17 23:16:42.165547 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.165524 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfv6t\" (UniqueName: \"kubernetes.io/projected/9bcd6410-0446-4ec1-b3e9-ec03c208a4d8-kube-api-access-rfv6t\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9shqs\" (UID: \"9bcd6410-0446-4ec1-b3e9-ec03c208a4d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs" Sep 17 23:16:42.165846 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.165826 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm2hw\" (UniqueName: \"kubernetes.io/projected/344908c0-c919-480e-82cf-9e4ab4451ae5-kube-api-access-xm2hw\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-dz4j5\" (UID: \"344908c0-c919-480e-82cf-9e4ab4451ae5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5" Sep 17 23:16:42.165922 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.165906 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2djz\" (UniqueName: \"kubernetes.io/projected/00282dde-9ace-4275-bc1e-08edf677ef20-kube-api-access-s2djz\") pod \"limitador-operator-controller-manager-74495c69f8-96vl8\" (UID: \"00282dde-9ace-4275-bc1e-08edf677ef20\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-96vl8" Sep 17 23:16:42.195605 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.195582 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" Sep 17 23:16:42.197463 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:42.197440 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kgvgh\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="4a35567f-c261-4ccb-b734-ded284fc5ec9" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:42.199172 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:42.199150 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-m9dhn\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="d1cf10b8-2408-45b7-a149-e769a9425175" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" Sep 17 23:16:42.254601 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.254577 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlfvz\" (UniqueName: \"kubernetes.io/projected/4a35567f-c261-4ccb-b734-ded284fc5ec9-kube-api-access-tlfvz\") pod \"4a35567f-c261-4ccb-b734-ded284fc5ec9\" (UID: \"4a35567f-c261-4ccb-b734-ded284fc5ec9\") " Sep 17 23:16:42.254711 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.254674 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww5zb\" (UniqueName: \"kubernetes.io/projected/d1cf10b8-2408-45b7-a149-e769a9425175-kube-api-access-ww5zb\") pod \"d1cf10b8-2408-45b7-a149-e769a9425175\" (UID: \"d1cf10b8-2408-45b7-a149-e769a9425175\") " Sep 17 23:16:42.254773 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.254749 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a35567f-c261-4ccb-b734-ded284fc5ec9-extensions-socket-volume\") pod \"4a35567f-c261-4ccb-b734-ded284fc5ec9\" (UID: \"4a35567f-c261-4ccb-b734-ded284fc5ec9\") " Sep 17 23:16:42.255226 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.255196 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a35567f-c261-4ccb-b734-ded284fc5ec9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4a35567f-c261-4ccb-b734-ded284fc5ec9" (UID: "4a35567f-c261-4ccb-b734-ded284fc5ec9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:16:42.256786 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.256760 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1cf10b8-2408-45b7-a149-e769a9425175-kube-api-access-ww5zb" (OuterVolumeSpecName: "kube-api-access-ww5zb") pod "d1cf10b8-2408-45b7-a149-e769a9425175" (UID: "d1cf10b8-2408-45b7-a149-e769a9425175"). InnerVolumeSpecName "kube-api-access-ww5zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:16:42.256786 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.256775 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a35567f-c261-4ccb-b734-ded284fc5ec9-kube-api-access-tlfvz" (OuterVolumeSpecName: "kube-api-access-tlfvz") pod "4a35567f-c261-4ccb-b734-ded284fc5ec9" (UID: "4a35567f-c261-4ccb-b734-ded284fc5ec9"). InnerVolumeSpecName "kube-api-access-tlfvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:16:42.263948 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.263926 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs" Sep 17 23:16:42.339874 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.339840 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5" Sep 17 23:16:42.345871 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.345848 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-96vl8" Sep 17 23:16:42.356996 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.356966 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww5zb\" (UniqueName: \"kubernetes.io/projected/d1cf10b8-2408-45b7-a149-e769a9425175-kube-api-access-ww5zb\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:16:42.357096 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.357001 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a35567f-c261-4ccb-b734-ded284fc5ec9-extensions-socket-volume\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:16:42.357096 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.357017 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlfvz\" (UniqueName: \"kubernetes.io/projected/4a35567f-c261-4ccb-b734-ded284fc5ec9-kube-api-access-tlfvz\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:16:42.552155 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.552135 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs"] Sep 17 23:16:42.554515 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:16:42.554472 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bcd6410_0446_4ec1_b3e9_ec03c208a4d8.slice/crio-96bcd249ca1a8e0c838178c18249d9f12ced8085261379f95e41c20f3462c95d WatchSource:0}: Error finding container 96bcd249ca1a8e0c838178c18249d9f12ced8085261379f95e41c20f3462c95d: Status 404 returned error can't find the container with id 96bcd249ca1a8e0c838178c18249d9f12ced8085261379f95e41c20f3462c95d Sep 17 23:16:42.621669 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.621646 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5"] Sep 17 23:16:42.623485 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:16:42.623456 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344908c0_c919_480e_82cf_9e4ab4451ae5.slice/crio-0ccd76c9ab704859f028f9f6ece2e0c5dcf6c0806ca9756d7f6c2278d82d9f3c WatchSource:0}: Error finding container 0ccd76c9ab704859f028f9f6ece2e0c5dcf6c0806ca9756d7f6c2278d82d9f3c: Status 404 returned error can't find the container with id 0ccd76c9ab704859f028f9f6ece2e0c5dcf6c0806ca9756d7f6c2278d82d9f3c Sep 17 23:16:42.632181 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.632162 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-96vl8"] Sep 17 23:16:42.639252 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:16:42.637124 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00282dde_9ace_4275_bc1e_08edf677ef20.slice/crio-53666750fa70eecd3f8703167d38355e4ff98899c0f1850be0e3a247e0788ef4 WatchSource:0}: Error finding container 53666750fa70eecd3f8703167d38355e4ff98899c0f1850be0e3a247e0788ef4: Status 404 returned error can't find the container with id 53666750fa70eecd3f8703167d38355e4ff98899c0f1850be0e3a247e0788ef4 Sep 17 23:16:42.866582 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.866485 2582 generic.go:356] "Generic (PLEG): container finished" podID="d1cf10b8-2408-45b7-a149-e769a9425175" containerID="6c805be9c6ef6ad5bbb321f673e38ec196663cff7268a906d914b5fdc818d705" exitCode=0 Sep 17 23:16:42.866582 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.866557 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" Sep 17 23:16:42.866786 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.866586 2582 scope.go:117] "RemoveContainer" containerID="6c805be9c6ef6ad5bbb321f673e38ec196663cff7268a906d914b5fdc818d705" Sep 17 23:16:42.869651 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.868610 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5" event={"ID":"344908c0-c919-480e-82cf-9e4ab4451ae5","Type":"ContainerStarted","Data":"b894412c7e041c295aba52f4c615176acd0caa0c859306c40a54f0531d40d929"} Sep 17 23:16:42.869651 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.868663 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5" event={"ID":"344908c0-c919-480e-82cf-9e4ab4451ae5","Type":"ContainerStarted","Data":"0ccd76c9ab704859f028f9f6ece2e0c5dcf6c0806ca9756d7f6c2278d82d9f3c"} Sep 17 23:16:42.869651 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.868706 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5" Sep 17 23:16:42.869651 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:42.869441 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kgvgh\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="4a35567f-c261-4ccb-b734-ded284fc5ec9" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:42.871334 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:42.871249 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-m9dhn\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="d1cf10b8-2408-45b7-a149-e769a9425175" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" Sep 17 23:16:42.873120 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.871898 2582 generic.go:356] "Generic (PLEG): container finished" podID="4a35567f-c261-4ccb-b734-ded284fc5ec9" containerID="ac771c29c72348dcca5ffac2aeccf1f82f1ca625e7862ee390c99a4bcd525d51" exitCode=0 Sep 17 23:16:42.873120 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.872068 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:42.877915 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.877089 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-96vl8" event={"ID":"00282dde-9ace-4275-bc1e-08edf677ef20","Type":"ContainerStarted","Data":"16306d892b99ecf429ae76132325435e20ef94bc273cad908386a509639e0762"} Sep 17 23:16:42.877915 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.877145 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-96vl8" event={"ID":"00282dde-9ace-4275-bc1e-08edf677ef20","Type":"ContainerStarted","Data":"53666750fa70eecd3f8703167d38355e4ff98899c0f1850be0e3a247e0788ef4"} Sep 17 23:16:42.877915 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.877174 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-96vl8" Sep 17 23:16:42.879783 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.879760 2582 scope.go:117] "RemoveContainer" containerID="6c805be9c6ef6ad5bbb321f673e38ec196663cff7268a906d914b5fdc818d705" Sep 17 23:16:42.880117 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:42.880091 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c805be9c6ef6ad5bbb321f673e38ec196663cff7268a906d914b5fdc818d705\": container with ID starting with 6c805be9c6ef6ad5bbb321f673e38ec196663cff7268a906d914b5fdc818d705 not found: ID does not exist" containerID="6c805be9c6ef6ad5bbb321f673e38ec196663cff7268a906d914b5fdc818d705" Sep 17 23:16:42.880199 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.880130 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c805be9c6ef6ad5bbb321f673e38ec196663cff7268a906d914b5fdc818d705"} err="failed to get container status \"6c805be9c6ef6ad5bbb321f673e38ec196663cff7268a906d914b5fdc818d705\": rpc error: code = NotFound desc = could not find container \"6c805be9c6ef6ad5bbb321f673e38ec196663cff7268a906d914b5fdc818d705\": container with ID starting with 6c805be9c6ef6ad5bbb321f673e38ec196663cff7268a906d914b5fdc818d705 not found: ID does not exist" Sep 17 23:16:42.880199 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.880153 2582 scope.go:117] "RemoveContainer" containerID="ac771c29c72348dcca5ffac2aeccf1f82f1ca625e7862ee390c99a4bcd525d51" Sep 17 23:16:42.881686 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.881102 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6" Sep 17 23:16:42.881686 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.881192 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs" event={"ID":"9bcd6410-0446-4ec1-b3e9-ec03c208a4d8","Type":"ContainerStarted","Data":"9b4f64d2c8452ee1eb130a5530dd9b88257b9b319c130dff7a255deb6c5a6816"} Sep 17 23:16:42.881686 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.881240 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs" event={"ID":"9bcd6410-0446-4ec1-b3e9-ec03c208a4d8","Type":"ContainerStarted","Data":"96bcd249ca1a8e0c838178c18249d9f12ced8085261379f95e41c20f3462c95d"} Sep 17 23:16:42.889884 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.889859 2582 scope.go:117] "RemoveContainer" containerID="ac771c29c72348dcca5ffac2aeccf1f82f1ca625e7862ee390c99a4bcd525d51" Sep 17 23:16:42.890171 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:42.890148 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac771c29c72348dcca5ffac2aeccf1f82f1ca625e7862ee390c99a4bcd525d51\": container with ID starting with ac771c29c72348dcca5ffac2aeccf1f82f1ca625e7862ee390c99a4bcd525d51 not found: ID does not exist" containerID="ac771c29c72348dcca5ffac2aeccf1f82f1ca625e7862ee390c99a4bcd525d51" Sep 17 23:16:42.890264 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.890175 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac771c29c72348dcca5ffac2aeccf1f82f1ca625e7862ee390c99a4bcd525d51"} err="failed to get container status \"ac771c29c72348dcca5ffac2aeccf1f82f1ca625e7862ee390c99a4bcd525d51\": rpc error: code = NotFound desc = could not find container \"ac771c29c72348dcca5ffac2aeccf1f82f1ca625e7862ee390c99a4bcd525d51\": container with ID starting with ac771c29c72348dcca5ffac2aeccf1f82f1ca625e7862ee390c99a4bcd525d51 not found: ID does not exist" Sep 17 23:16:42.890635 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.890621 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6" Sep 17 23:16:42.893353 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:42.893329 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kgvgh\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="4a35567f-c261-4ccb-b734-ded284fc5ec9" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:42.893563 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.893526 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5" podStartSLOduration=1.893486241 podStartE2EDuration="1.893486241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:42.891335106 +0000 UTC m=+641.825242242" watchObservedRunningTime="2026-09-17 23:16:42.893486241 +0000 UTC m=+641.827393377" Sep 17 23:16:42.895061 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:42.895036 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-m9dhn\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="d1cf10b8-2408-45b7-a149-e769a9425175" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" Sep 17 23:16:42.896651 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:42.896632 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kgvgh\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="4a35567f-c261-4ccb-b734-ded284fc5ec9" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:42.898070 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:42.898040 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-q4jb6\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="481b5fcb-59a6-4586-9776-ec6fff353a43" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6" Sep 17 23:16:42.899609 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:42.899587 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-m9dhn\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="d1cf10b8-2408-45b7-a149-e769a9425175" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" Sep 17 23:16:42.925036 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.924997 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs" podStartSLOduration=1.9249842830000001 podStartE2EDuration="1.924984283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:42.922582246 +0000 UTC m=+641.856489382" watchObservedRunningTime="2026-09-17 23:16:42.924984283 +0000 UTC m=+641.858891419" Sep 17 23:16:42.941337 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.941295 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-96vl8" podStartSLOduration=1.941284618 podStartE2EDuration="1.941284618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:42.938944482 +0000 UTC m=+641.872851614" watchObservedRunningTime="2026-09-17 23:16:42.941284618 +0000 UTC m=+641.875191755" Sep 17 23:16:42.966714 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.966695 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgk7w\" (UniqueName: \"kubernetes.io/projected/481b5fcb-59a6-4586-9776-ec6fff353a43-kube-api-access-fgk7w\") pod \"481b5fcb-59a6-4586-9776-ec6fff353a43\" (UID: \"481b5fcb-59a6-4586-9776-ec6fff353a43\") " Sep 17 23:16:42.966806 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.966745 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/481b5fcb-59a6-4586-9776-ec6fff353a43-extensions-socket-volume\") pod \"481b5fcb-59a6-4586-9776-ec6fff353a43\" (UID: \"481b5fcb-59a6-4586-9776-ec6fff353a43\") " Sep 17 23:16:42.967018 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.966997 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481b5fcb-59a6-4586-9776-ec6fff353a43-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "481b5fcb-59a6-4586-9776-ec6fff353a43" (UID: "481b5fcb-59a6-4586-9776-ec6fff353a43"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:16:42.968838 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:42.968812 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481b5fcb-59a6-4586-9776-ec6fff353a43-kube-api-access-fgk7w" (OuterVolumeSpecName: "kube-api-access-fgk7w") pod "481b5fcb-59a6-4586-9776-ec6fff353a43" (UID: "481b5fcb-59a6-4586-9776-ec6fff353a43"). InnerVolumeSpecName "kube-api-access-fgk7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:16:43.067535 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:43.067488 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgk7w\" (UniqueName: \"kubernetes.io/projected/481b5fcb-59a6-4586-9776-ec6fff353a43-kube-api-access-fgk7w\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:16:43.067535 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:43.067532 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/481b5fcb-59a6-4586-9776-ec6fff353a43-extensions-socket-volume\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:16:43.292660 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:43.292616 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kgvgh\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="4a35567f-c261-4ccb-b734-ded284fc5ec9" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kgvgh" Sep 17 23:16:43.294398 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:43.294375 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-q4jb6\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="481b5fcb-59a6-4586-9776-ec6fff353a43" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6" Sep 17 23:16:43.294732 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:43.294713 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="481b5fcb-59a6-4586-9776-ec6fff353a43" path="/var/lib/kubelet/pods/481b5fcb-59a6-4586-9776-ec6fff353a43/volumes" Sep 17 23:16:43.294966 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:43.294954 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a35567f-c261-4ccb-b734-ded284fc5ec9" path="/var/lib/kubelet/pods/4a35567f-c261-4ccb-b734-ded284fc5ec9/volumes" Sep 17 23:16:43.295239 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:43.295228 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1cf10b8-2408-45b7-a149-e769a9425175" path="/var/lib/kubelet/pods/d1cf10b8-2408-45b7-a149-e769a9425175/volumes" Sep 17 23:16:43.296244 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:43.296209 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-m9dhn\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="d1cf10b8-2408-45b7-a149-e769a9425175" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m9dhn" Sep 17 23:16:43.887372 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:43.887297 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6" Sep 17 23:16:43.887817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:43.887796 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs" Sep 17 23:16:43.891371 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:43.891343 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-q4jb6\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="481b5fcb-59a6-4586-9776-ec6fff353a43" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6" Sep 17 23:16:44.826990 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:44.826962 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pqwsr" Sep 17 23:16:44.829003 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:44.828975 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-q4jb6\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="481b5fcb-59a6-4586-9776-ec6fff353a43" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q4jb6" Sep 17 23:16:52.872160 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:52.872114 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65bdd89887-4xl2h" podUID="b4b06529-61f5-4617-8b85-152485d9f1ca" containerName="console" containerID="cri-o://4a95a124608e7ebed93227040ebc17c3825c8d1c03a55bb6baf491dc6a15a1e2" gracePeriod=15 Sep 17 23:16:53.116476 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.116451 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:16:53.257433 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.257402 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-console-config\") pod \"b4b06529-61f5-4617-8b85-152485d9f1ca\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " Sep 17 23:16:53.257616 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.257468 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-oauth-serving-cert\") pod \"b4b06529-61f5-4617-8b85-152485d9f1ca\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " Sep 17 23:16:53.257616 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.257485 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-trusted-ca-bundle\") pod \"b4b06529-61f5-4617-8b85-152485d9f1ca\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " Sep 17 23:16:53.257616 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.257567 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-service-ca\") pod \"b4b06529-61f5-4617-8b85-152485d9f1ca\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " Sep 17 23:16:53.257616 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.257601 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4b06529-61f5-4617-8b85-152485d9f1ca-console-oauth-config\") pod \"b4b06529-61f5-4617-8b85-152485d9f1ca\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " Sep 17 23:16:53.257616 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.257616 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4b06529-61f5-4617-8b85-152485d9f1ca-console-serving-cert\") pod \"b4b06529-61f5-4617-8b85-152485d9f1ca\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " Sep 17 23:16:53.257881 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.257637 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t9pr\" (UniqueName: \"kubernetes.io/projected/b4b06529-61f5-4617-8b85-152485d9f1ca-kube-api-access-8t9pr\") pod \"b4b06529-61f5-4617-8b85-152485d9f1ca\" (UID: \"b4b06529-61f5-4617-8b85-152485d9f1ca\") " Sep 17 23:16:53.258005 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.257972 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-console-config" (OuterVolumeSpecName: "console-config") pod "b4b06529-61f5-4617-8b85-152485d9f1ca" (UID: "b4b06529-61f5-4617-8b85-152485d9f1ca"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:16:53.258094 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.258021 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b4b06529-61f5-4617-8b85-152485d9f1ca" (UID: "b4b06529-61f5-4617-8b85-152485d9f1ca"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:16:53.258094 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.258016 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b4b06529-61f5-4617-8b85-152485d9f1ca" (UID: "b4b06529-61f5-4617-8b85-152485d9f1ca"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:16:53.258186 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.258108 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-service-ca" (OuterVolumeSpecName: "service-ca") pod "b4b06529-61f5-4617-8b85-152485d9f1ca" (UID: "b4b06529-61f5-4617-8b85-152485d9f1ca"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:16:53.259905 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.259875 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4b06529-61f5-4617-8b85-152485d9f1ca-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b4b06529-61f5-4617-8b85-152485d9f1ca" (UID: "b4b06529-61f5-4617-8b85-152485d9f1ca"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:16:53.260002 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.259902 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4b06529-61f5-4617-8b85-152485d9f1ca-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b4b06529-61f5-4617-8b85-152485d9f1ca" (UID: "b4b06529-61f5-4617-8b85-152485d9f1ca"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:16:53.260002 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.259967 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b06529-61f5-4617-8b85-152485d9f1ca-kube-api-access-8t9pr" (OuterVolumeSpecName: "kube-api-access-8t9pr") pod "b4b06529-61f5-4617-8b85-152485d9f1ca" (UID: "b4b06529-61f5-4617-8b85-152485d9f1ca"). InnerVolumeSpecName "kube-api-access-8t9pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:16:53.358239 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.358208 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:16:53.358239 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.358238 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4b06529-61f5-4617-8b85-152485d9f1ca-console-oauth-config\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:16:53.358453 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.358248 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4b06529-61f5-4617-8b85-152485d9f1ca-console-serving-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:16:53.358453 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.358259 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t9pr\" (UniqueName: \"kubernetes.io/projected/b4b06529-61f5-4617-8b85-152485d9f1ca-kube-api-access-8t9pr\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:16:53.358453 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.358267 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-console-config\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:16:53.358453 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.358275 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-oauth-serving-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:16:53.358453 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.358283 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4b06529-61f5-4617-8b85-152485d9f1ca-trusted-ca-bundle\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:16:53.889817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.889774 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-96vl8" Sep 17 23:16:53.890204 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.889853 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5" Sep 17 23:16:53.937585 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.937556 2582 generic.go:356] "Generic (PLEG): container finished" podID="b4b06529-61f5-4617-8b85-152485d9f1ca" containerID="4a95a124608e7ebed93227040ebc17c3825c8d1c03a55bb6baf491dc6a15a1e2" exitCode=0 Sep 17 23:16:53.937723 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.937590 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65bdd89887-4xl2h" event={"ID":"b4b06529-61f5-4617-8b85-152485d9f1ca","Type":"ContainerDied","Data":"4a95a124608e7ebed93227040ebc17c3825c8d1c03a55bb6baf491dc6a15a1e2"} Sep 17 23:16:53.937723 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.937594 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65bdd89887-4xl2h" Sep 17 23:16:53.937723 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.937620 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65bdd89887-4xl2h" event={"ID":"b4b06529-61f5-4617-8b85-152485d9f1ca","Type":"ContainerDied","Data":"ac7bcdc65e02ac4ae99bf910de782f61c6ab1b6485bcfc195503c30f64095de3"} Sep 17 23:16:53.937723 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.937642 2582 scope.go:117] "RemoveContainer" containerID="4a95a124608e7ebed93227040ebc17c3825c8d1c03a55bb6baf491dc6a15a1e2" Sep 17 23:16:53.946125 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.946098 2582 scope.go:117] "RemoveContainer" containerID="4a95a124608e7ebed93227040ebc17c3825c8d1c03a55bb6baf491dc6a15a1e2" Sep 17 23:16:53.946374 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:53.946355 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a95a124608e7ebed93227040ebc17c3825c8d1c03a55bb6baf491dc6a15a1e2\": container with ID starting with 4a95a124608e7ebed93227040ebc17c3825c8d1c03a55bb6baf491dc6a15a1e2 not found: ID does not exist" containerID="4a95a124608e7ebed93227040ebc17c3825c8d1c03a55bb6baf491dc6a15a1e2" Sep 17 23:16:53.946444 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:53.946380 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a95a124608e7ebed93227040ebc17c3825c8d1c03a55bb6baf491dc6a15a1e2"} err="failed to get container status \"4a95a124608e7ebed93227040ebc17c3825c8d1c03a55bb6baf491dc6a15a1e2\": rpc error: code = NotFound desc = could not find container \"4a95a124608e7ebed93227040ebc17c3825c8d1c03a55bb6baf491dc6a15a1e2\": container with ID starting with 4a95a124608e7ebed93227040ebc17c3825c8d1c03a55bb6baf491dc6a15a1e2 not found: ID does not exist" Sep 17 23:16:54.004297 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.004271 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65bdd89887-4xl2h"] Sep 17 23:16:54.007463 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.007442 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs"] Sep 17 23:16:54.007728 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.007696 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs" podUID="9bcd6410-0446-4ec1-b3e9-ec03c208a4d8" containerName="manager" containerID="cri-o://9b4f64d2c8452ee1eb130a5530dd9b88257b9b319c130dff7a255deb6c5a6816" gracePeriod=10 Sep 17 23:16:54.009921 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.009897 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs" Sep 17 23:16:54.014520 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.014434 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65bdd89887-4xl2h"] Sep 17 23:16:54.251103 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.251069 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs" Sep 17 23:16:54.263393 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.263367 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg"] Sep 17 23:16:54.263738 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.263725 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4b06529-61f5-4617-8b85-152485d9f1ca" containerName="console" Sep 17 23:16:54.263806 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.263739 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4b06529-61f5-4617-8b85-152485d9f1ca" containerName="console" Sep 17 23:16:54.263806 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.263758 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bcd6410-0446-4ec1-b3e9-ec03c208a4d8" containerName="manager" Sep 17 23:16:54.263806 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.263764 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bcd6410-0446-4ec1-b3e9-ec03c208a4d8" containerName="manager" Sep 17 23:16:54.263915 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.263817 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4b06529-61f5-4617-8b85-152485d9f1ca" containerName="console" Sep 17 23:16:54.263915 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.263824 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bcd6410-0446-4ec1-b3e9-ec03c208a4d8" containerName="manager" Sep 17 23:16:54.265726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.265707 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg" Sep 17 23:16:54.284015 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.283985 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg"] Sep 17 23:16:54.366269 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.366240 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9bcd6410-0446-4ec1-b3e9-ec03c208a4d8-extensions-socket-volume\") pod \"9bcd6410-0446-4ec1-b3e9-ec03c208a4d8\" (UID: \"9bcd6410-0446-4ec1-b3e9-ec03c208a4d8\") " Sep 17 23:16:54.366442 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.366283 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfv6t\" (UniqueName: \"kubernetes.io/projected/9bcd6410-0446-4ec1-b3e9-ec03c208a4d8-kube-api-access-rfv6t\") pod \"9bcd6410-0446-4ec1-b3e9-ec03c208a4d8\" (UID: \"9bcd6410-0446-4ec1-b3e9-ec03c208a4d8\") " Sep 17 23:16:54.366543 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.366462 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55bc62a6-86f7-46e3-8208-03132e1d11ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kd6gg\" (UID: \"55bc62a6-86f7-46e3-8208-03132e1d11ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg" Sep 17 23:16:54.366620 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.366552 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nrwl\" (UniqueName: \"kubernetes.io/projected/55bc62a6-86f7-46e3-8208-03132e1d11ac-kube-api-access-6nrwl\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kd6gg\" (UID: \"55bc62a6-86f7-46e3-8208-03132e1d11ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg" Sep 17 23:16:54.366676 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.366658 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcd6410-0446-4ec1-b3e9-ec03c208a4d8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9bcd6410-0446-4ec1-b3e9-ec03c208a4d8" (UID: "9bcd6410-0446-4ec1-b3e9-ec03c208a4d8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:16:54.368370 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.368343 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bcd6410-0446-4ec1-b3e9-ec03c208a4d8-kube-api-access-rfv6t" (OuterVolumeSpecName: "kube-api-access-rfv6t") pod "9bcd6410-0446-4ec1-b3e9-ec03c208a4d8" (UID: "9bcd6410-0446-4ec1-b3e9-ec03c208a4d8"). InnerVolumeSpecName "kube-api-access-rfv6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:16:54.467673 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.467605 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nrwl\" (UniqueName: \"kubernetes.io/projected/55bc62a6-86f7-46e3-8208-03132e1d11ac-kube-api-access-6nrwl\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kd6gg\" (UID: \"55bc62a6-86f7-46e3-8208-03132e1d11ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg" Sep 17 23:16:54.467673 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.467665 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55bc62a6-86f7-46e3-8208-03132e1d11ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kd6gg\" (UID: \"55bc62a6-86f7-46e3-8208-03132e1d11ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg" Sep 17 23:16:54.467826 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.467696 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9bcd6410-0446-4ec1-b3e9-ec03c208a4d8-extensions-socket-volume\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:16:54.467826 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.467706 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfv6t\" (UniqueName: \"kubernetes.io/projected/9bcd6410-0446-4ec1-b3e9-ec03c208a4d8-kube-api-access-rfv6t\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:16:54.467989 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.467974 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55bc62a6-86f7-46e3-8208-03132e1d11ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kd6gg\" (UID: \"55bc62a6-86f7-46e3-8208-03132e1d11ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg" Sep 17 23:16:54.478903 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.478882 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nrwl\" (UniqueName: \"kubernetes.io/projected/55bc62a6-86f7-46e3-8208-03132e1d11ac-kube-api-access-6nrwl\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kd6gg\" (UID: \"55bc62a6-86f7-46e3-8208-03132e1d11ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg" Sep 17 23:16:54.577581 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.577557 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg" Sep 17 23:16:54.851241 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.851210 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg"] Sep 17 23:16:54.854460 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:16:54.854437 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55bc62a6_86f7_46e3_8208_03132e1d11ac.slice/crio-7259dd5405c75803da19c7649728e375f91f8f0e10427ad6859124cd3390bfdb WatchSource:0}: Error finding container 7259dd5405c75803da19c7649728e375f91f8f0e10427ad6859124cd3390bfdb: Status 404 returned error can't find the container with id 7259dd5405c75803da19c7649728e375f91f8f0e10427ad6859124cd3390bfdb Sep 17 23:16:54.945194 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.945147 2582 generic.go:356] "Generic (PLEG): container finished" podID="9bcd6410-0446-4ec1-b3e9-ec03c208a4d8" containerID="9b4f64d2c8452ee1eb130a5530dd9b88257b9b319c130dff7a255deb6c5a6816" exitCode=0 Sep 17 23:16:54.945549 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.945204 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs" Sep 17 23:16:54.945549 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.945244 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs" event={"ID":"9bcd6410-0446-4ec1-b3e9-ec03c208a4d8","Type":"ContainerDied","Data":"9b4f64d2c8452ee1eb130a5530dd9b88257b9b319c130dff7a255deb6c5a6816"} Sep 17 23:16:54.945549 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.945293 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs" event={"ID":"9bcd6410-0446-4ec1-b3e9-ec03c208a4d8","Type":"ContainerDied","Data":"96bcd249ca1a8e0c838178c18249d9f12ced8085261379f95e41c20f3462c95d"} Sep 17 23:16:54.945549 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.945316 2582 scope.go:117] "RemoveContainer" containerID="9b4f64d2c8452ee1eb130a5530dd9b88257b9b319c130dff7a255deb6c5a6816" Sep 17 23:16:54.947289 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.947269 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg" event={"ID":"55bc62a6-86f7-46e3-8208-03132e1d11ac","Type":"ContainerStarted","Data":"db0078821fabacdbf3d1c99ef48d52af57c2440a6cfa9f7630a0ae100e025520"} Sep 17 23:16:54.947387 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.947312 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg" event={"ID":"55bc62a6-86f7-46e3-8208-03132e1d11ac","Type":"ContainerStarted","Data":"7259dd5405c75803da19c7649728e375f91f8f0e10427ad6859124cd3390bfdb"} Sep 17 23:16:54.947440 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.947410 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg" Sep 17 23:16:54.954024 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.954008 2582 scope.go:117] "RemoveContainer" containerID="9b4f64d2c8452ee1eb130a5530dd9b88257b9b319c130dff7a255deb6c5a6816" Sep 17 23:16:54.954276 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:16:54.954259 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b4f64d2c8452ee1eb130a5530dd9b88257b9b319c130dff7a255deb6c5a6816\": container with ID starting with 9b4f64d2c8452ee1eb130a5530dd9b88257b9b319c130dff7a255deb6c5a6816 not found: ID does not exist" containerID="9b4f64d2c8452ee1eb130a5530dd9b88257b9b319c130dff7a255deb6c5a6816" Sep 17 23:16:54.954319 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.954284 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b4f64d2c8452ee1eb130a5530dd9b88257b9b319c130dff7a255deb6c5a6816"} err="failed to get container status \"9b4f64d2c8452ee1eb130a5530dd9b88257b9b319c130dff7a255deb6c5a6816\": rpc error: code = NotFound desc = could not find container \"9b4f64d2c8452ee1eb130a5530dd9b88257b9b319c130dff7a255deb6c5a6816\": container with ID starting with 9b4f64d2c8452ee1eb130a5530dd9b88257b9b319c130dff7a255deb6c5a6816 not found: ID does not exist" Sep 17 23:16:54.971037 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.970990 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg" podStartSLOduration=0.970974553 podStartE2EDuration="970.974553ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:54.969779829 +0000 UTC m=+653.903686981" watchObservedRunningTime="2026-09-17 23:16:54.970974553 +0000 UTC m=+653.904881690" Sep 17 23:16:54.989942 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.989913 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs"] Sep 17 23:16:54.999874 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:54.999852 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9shqs"] Sep 17 23:16:55.294042 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:55.294009 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bcd6410-0446-4ec1-b3e9-ec03c208a4d8" path="/var/lib/kubelet/pods/9bcd6410-0446-4ec1-b3e9-ec03c208a4d8/volumes" Sep 17 23:16:55.294396 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:16:55.294383 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4b06529-61f5-4617-8b85-152485d9f1ca" path="/var/lib/kubelet/pods/b4b06529-61f5-4617-8b85-152485d9f1ca/volumes" Sep 17 23:17:05.955935 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:05.955900 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg" Sep 17 23:17:06.003974 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:06.003938 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5"] Sep 17 23:17:06.004165 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:06.004126 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5" podUID="344908c0-c919-480e-82cf-9e4ab4451ae5" containerName="manager" containerID="cri-o://b894412c7e041c295aba52f4c615176acd0caa0c859306c40a54f0531d40d929" gracePeriod=10 Sep 17 23:17:06.241804 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:06.241781 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5" Sep 17 23:17:06.363632 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:06.363600 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/344908c0-c919-480e-82cf-9e4ab4451ae5-extensions-socket-volume\") pod \"344908c0-c919-480e-82cf-9e4ab4451ae5\" (UID: \"344908c0-c919-480e-82cf-9e4ab4451ae5\") " Sep 17 23:17:06.363773 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:06.363699 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm2hw\" (UniqueName: \"kubernetes.io/projected/344908c0-c919-480e-82cf-9e4ab4451ae5-kube-api-access-xm2hw\") pod \"344908c0-c919-480e-82cf-9e4ab4451ae5\" (UID: \"344908c0-c919-480e-82cf-9e4ab4451ae5\") " Sep 17 23:17:06.364015 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:06.363994 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344908c0-c919-480e-82cf-9e4ab4451ae5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "344908c0-c919-480e-82cf-9e4ab4451ae5" (UID: "344908c0-c919-480e-82cf-9e4ab4451ae5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:17:06.365701 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:06.365679 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344908c0-c919-480e-82cf-9e4ab4451ae5-kube-api-access-xm2hw" (OuterVolumeSpecName: "kube-api-access-xm2hw") pod "344908c0-c919-480e-82cf-9e4ab4451ae5" (UID: "344908c0-c919-480e-82cf-9e4ab4451ae5"). InnerVolumeSpecName "kube-api-access-xm2hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:17:06.464444 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:06.464414 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm2hw\" (UniqueName: \"kubernetes.io/projected/344908c0-c919-480e-82cf-9e4ab4451ae5-kube-api-access-xm2hw\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:17:06.464444 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:06.464439 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/344908c0-c919-480e-82cf-9e4ab4451ae5-extensions-socket-volume\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:17:07.006006 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:07.005967 2582 generic.go:356] "Generic (PLEG): container finished" podID="344908c0-c919-480e-82cf-9e4ab4451ae5" containerID="b894412c7e041c295aba52f4c615176acd0caa0c859306c40a54f0531d40d929" exitCode=0 Sep 17 23:17:07.006493 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:07.006016 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5" Sep 17 23:17:07.006493 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:07.006051 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5" event={"ID":"344908c0-c919-480e-82cf-9e4ab4451ae5","Type":"ContainerDied","Data":"b894412c7e041c295aba52f4c615176acd0caa0c859306c40a54f0531d40d929"} Sep 17 23:17:07.006493 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:07.006103 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5" event={"ID":"344908c0-c919-480e-82cf-9e4ab4451ae5","Type":"ContainerDied","Data":"0ccd76c9ab704859f028f9f6ece2e0c5dcf6c0806ca9756d7f6c2278d82d9f3c"} Sep 17 23:17:07.006493 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:07.006127 2582 scope.go:117] "RemoveContainer" containerID="b894412c7e041c295aba52f4c615176acd0caa0c859306c40a54f0531d40d929" Sep 17 23:17:07.015211 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:07.015192 2582 scope.go:117] "RemoveContainer" containerID="b894412c7e041c295aba52f4c615176acd0caa0c859306c40a54f0531d40d929" Sep 17 23:17:07.015491 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:17:07.015473 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b894412c7e041c295aba52f4c615176acd0caa0c859306c40a54f0531d40d929\": container with ID starting with b894412c7e041c295aba52f4c615176acd0caa0c859306c40a54f0531d40d929 not found: ID does not exist" containerID="b894412c7e041c295aba52f4c615176acd0caa0c859306c40a54f0531d40d929" Sep 17 23:17:07.015562 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:07.015529 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b894412c7e041c295aba52f4c615176acd0caa0c859306c40a54f0531d40d929"} err="failed to get container status \"b894412c7e041c295aba52f4c615176acd0caa0c859306c40a54f0531d40d929\": rpc error: code = NotFound desc = could not find container \"b894412c7e041c295aba52f4c615176acd0caa0c859306c40a54f0531d40d929\": container with ID starting with b894412c7e041c295aba52f4c615176acd0caa0c859306c40a54f0531d40d929 not found: ID does not exist" Sep 17 23:17:07.029961 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:07.029934 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5"] Sep 17 23:17:07.034535 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:07.034511 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dz4j5"] Sep 17 23:17:07.293117 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:07.293042 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="344908c0-c919-480e-82cf-9e4ab4451ae5" path="/var/lib/kubelet/pods/344908c0-c919-480e-82cf-9e4ab4451ae5/volumes" Sep 17 23:17:10.317763 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.317728 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc"] Sep 17 23:17:10.318136 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.318122 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="344908c0-c919-480e-82cf-9e4ab4451ae5" containerName="manager" Sep 17 23:17:10.318188 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.318138 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="344908c0-c919-480e-82cf-9e4ab4451ae5" containerName="manager" Sep 17 23:17:10.318222 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.318188 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="344908c0-c919-480e-82cf-9e4ab4451ae5" containerName="manager" Sep 17 23:17:10.321493 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.321472 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.325022 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.325002 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 17 23:17:10.325153 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.325004 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pn746\"" Sep 17 23:17:10.332811 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.332789 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc"] Sep 17 23:17:10.495805 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.495771 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b3c4835-32b2-46a3-8d2d-f64071d509fd-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.495805 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.495807 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b3c4835-32b2-46a3-8d2d-f64071d509fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.495998 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.495831 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b3c4835-32b2-46a3-8d2d-f64071d509fd-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.495998 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.495915 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b3c4835-32b2-46a3-8d2d-f64071d509fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.495998 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.495954 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxz7v\" (UniqueName: \"kubernetes.io/projected/7b3c4835-32b2-46a3-8d2d-f64071d509fd-kube-api-access-zxz7v\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.495998 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.495985 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3c4835-32b2-46a3-8d2d-f64071d509fd-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.496140 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.496010 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b3c4835-32b2-46a3-8d2d-f64071d509fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.496140 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.496035 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3c4835-32b2-46a3-8d2d-f64071d509fd-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.496140 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.496067 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b3c4835-32b2-46a3-8d2d-f64071d509fd-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.596735 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.596658 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b3c4835-32b2-46a3-8d2d-f64071d509fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.596735 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.596696 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxz7v\" (UniqueName: \"kubernetes.io/projected/7b3c4835-32b2-46a3-8d2d-f64071d509fd-kube-api-access-zxz7v\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.596735 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.596723 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3c4835-32b2-46a3-8d2d-f64071d509fd-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.596990 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.596749 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b3c4835-32b2-46a3-8d2d-f64071d509fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.596990 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.596773 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3c4835-32b2-46a3-8d2d-f64071d509fd-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.596990 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.596791 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b3c4835-32b2-46a3-8d2d-f64071d509fd-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.596990 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.596832 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b3c4835-32b2-46a3-8d2d-f64071d509fd-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.596990 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.596862 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b3c4835-32b2-46a3-8d2d-f64071d509fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.596990 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.596895 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b3c4835-32b2-46a3-8d2d-f64071d509fd-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.597320 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.597292 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3c4835-32b2-46a3-8d2d-f64071d509fd-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.597475 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.597453 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b3c4835-32b2-46a3-8d2d-f64071d509fd-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.597475 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.597466 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b3c4835-32b2-46a3-8d2d-f64071d509fd-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.597595 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.597523 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b3c4835-32b2-46a3-8d2d-f64071d509fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.597595 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.597533 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b3c4835-32b2-46a3-8d2d-f64071d509fd-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.599218 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.599200 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b3c4835-32b2-46a3-8d2d-f64071d509fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.599442 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.599424 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b3c4835-32b2-46a3-8d2d-f64071d509fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.605305 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.605282 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b3c4835-32b2-46a3-8d2d-f64071d509fd-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.605819 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.605801 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxz7v\" (UniqueName: \"kubernetes.io/projected/7b3c4835-32b2-46a3-8d2d-f64071d509fd-kube-api-access-zxz7v\") pod \"maas-default-gateway-openshift-default-78bb78ff78-b66bc\" (UID: \"7b3c4835-32b2-46a3-8d2d-f64071d509fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.632837 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.632814 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:10.904556 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:10.904534 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc"] Sep 17 23:17:10.906940 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:17:10.906914 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3c4835_32b2_46a3_8d2d_f64071d509fd.slice/crio-abd580ecdc21976b91c5e512580cc0f68a18b4e858b669ae4d4e488f75979d60 WatchSource:0}: Error finding container abd580ecdc21976b91c5e512580cc0f68a18b4e858b669ae4d4e488f75979d60: Status 404 returned error can't find the container with id abd580ecdc21976b91c5e512580cc0f68a18b4e858b669ae4d4e488f75979d60 Sep 17 23:17:11.028059 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:11.028028 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" event={"ID":"7b3c4835-32b2-46a3-8d2d-f64071d509fd","Type":"ContainerStarted","Data":"abd580ecdc21976b91c5e512580cc0f68a18b4e858b669ae4d4e488f75979d60"} Sep 17 23:17:17.548818 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:17.548774 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:17:17.549082 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:17.548850 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:17:17.549082 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:17.548877 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:17:18.066012 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:18.065978 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" event={"ID":"7b3c4835-32b2-46a3-8d2d-f64071d509fd","Type":"ContainerStarted","Data":"3cdfbef3ea26d8fbf7266a1089068bfc95ca7879fbfac4a5664d17881f036867"} Sep 17 23:17:18.088043 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:18.087992 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" podStartSLOduration=1.448924559 podStartE2EDuration="8.087976953s" totalImagesPullingTime="6.639052394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:17:18.085666779 +0000 UTC m=+677.019573927" watchObservedRunningTime="2026-09-17 23:17:18.087976953 +0000 UTC m=+677.021884161" Sep 17 23:17:18.633728 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:18.633694 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:18.638719 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:18.638689 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:19.071038 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:19.071001 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:19.071932 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:19.071913 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:17:27.641674 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:27.641639 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-x5mz7"] Sep 17 23:17:27.645175 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:27.645155 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-x5mz7" Sep 17 23:17:27.647365 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:27.647340 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 17 23:17:27.647483 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:27.647379 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-97v8q\"" Sep 17 23:17:27.656692 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:27.656670 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-x5mz7"] Sep 17 23:17:27.740461 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:27.740413 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-x5mz7"] Sep 17 23:17:27.741614 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:27.741576 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8ns7\" (UniqueName: \"kubernetes.io/projected/712ec0c0-4bab-4ac5-8329-f9fcf561deda-kube-api-access-k8ns7\") pod \"limitador-limitador-8588749977-x5mz7\" (UID: \"712ec0c0-4bab-4ac5-8329-f9fcf561deda\") " pod="kuadrant-system/limitador-limitador-8588749977-x5mz7" Sep 17 23:17:27.741775 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:27.741641 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/712ec0c0-4bab-4ac5-8329-f9fcf561deda-config-file\") pod \"limitador-limitador-8588749977-x5mz7\" (UID: \"712ec0c0-4bab-4ac5-8329-f9fcf561deda\") " pod="kuadrant-system/limitador-limitador-8588749977-x5mz7" Sep 17 23:17:27.842524 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:27.842462 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8ns7\" (UniqueName: \"kubernetes.io/projected/712ec0c0-4bab-4ac5-8329-f9fcf561deda-kube-api-access-k8ns7\") pod \"limitador-limitador-8588749977-x5mz7\" (UID: \"712ec0c0-4bab-4ac5-8329-f9fcf561deda\") " pod="kuadrant-system/limitador-limitador-8588749977-x5mz7" Sep 17 23:17:27.842524 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:27.842526 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/712ec0c0-4bab-4ac5-8329-f9fcf561deda-config-file\") pod \"limitador-limitador-8588749977-x5mz7\" (UID: \"712ec0c0-4bab-4ac5-8329-f9fcf561deda\") " pod="kuadrant-system/limitador-limitador-8588749977-x5mz7" Sep 17 23:17:27.843138 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:27.843120 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/712ec0c0-4bab-4ac5-8329-f9fcf561deda-config-file\") pod \"limitador-limitador-8588749977-x5mz7\" (UID: \"712ec0c0-4bab-4ac5-8329-f9fcf561deda\") " pod="kuadrant-system/limitador-limitador-8588749977-x5mz7" Sep 17 23:17:27.850109 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:27.850078 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8ns7\" (UniqueName: \"kubernetes.io/projected/712ec0c0-4bab-4ac5-8329-f9fcf561deda-kube-api-access-k8ns7\") pod \"limitador-limitador-8588749977-x5mz7\" (UID: \"712ec0c0-4bab-4ac5-8329-f9fcf561deda\") " pod="kuadrant-system/limitador-limitador-8588749977-x5mz7" Sep 17 23:17:27.956065 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:27.955983 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-x5mz7" Sep 17 23:17:28.231917 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:28.231884 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-x5mz7"] Sep 17 23:17:28.234964 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:17:28.234937 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712ec0c0_4bab_4ac5_8329_f9fcf561deda.slice/crio-1563f456a6a572ce140acd4cb1b4f2daaf6bfea3c183731e416b19a6ba65b1bb WatchSource:0}: Error finding container 1563f456a6a572ce140acd4cb1b4f2daaf6bfea3c183731e416b19a6ba65b1bb: Status 404 returned error can't find the container with id 1563f456a6a572ce140acd4cb1b4f2daaf6bfea3c183731e416b19a6ba65b1bb Sep 17 23:17:28.236926 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:28.236910 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 23:17:28.423709 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:28.423663 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7kxv5"] Sep 17 23:17:28.430074 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:28.430041 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-7kxv5" Sep 17 23:17:28.435107 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:28.435076 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cmz9j\"" Sep 17 23:17:28.438290 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:28.438260 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7kxv5"] Sep 17 23:17:28.550083 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:28.549997 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mb2f\" (UniqueName: \"kubernetes.io/projected/6830ef77-3cb3-46a6-b93c-d9e831f1e0fd-kube-api-access-8mb2f\") pod \"authorino-7bcfd8f9cd-7kxv5\" (UID: \"6830ef77-3cb3-46a6-b93c-d9e831f1e0fd\") " pod="kuadrant-system/authorino-7bcfd8f9cd-7kxv5" Sep 17 23:17:28.650887 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:28.650852 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mb2f\" (UniqueName: \"kubernetes.io/projected/6830ef77-3cb3-46a6-b93c-d9e831f1e0fd-kube-api-access-8mb2f\") pod \"authorino-7bcfd8f9cd-7kxv5\" (UID: \"6830ef77-3cb3-46a6-b93c-d9e831f1e0fd\") " pod="kuadrant-system/authorino-7bcfd8f9cd-7kxv5" Sep 17 23:17:28.658946 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:28.658921 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mb2f\" (UniqueName: \"kubernetes.io/projected/6830ef77-3cb3-46a6-b93c-d9e831f1e0fd-kube-api-access-8mb2f\") pod \"authorino-7bcfd8f9cd-7kxv5\" (UID: \"6830ef77-3cb3-46a6-b93c-d9e831f1e0fd\") " pod="kuadrant-system/authorino-7bcfd8f9cd-7kxv5" Sep 17 23:17:28.744693 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:28.744659 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-7kxv5" Sep 17 23:17:29.014201 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:29.014070 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7kxv5"] Sep 17 23:17:29.016593 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:17:29.016568 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6830ef77_3cb3_46a6_b93c_d9e831f1e0fd.slice/crio-197276a3f76c5bf4503157565f9e07f9e1dffb7fab50456fad23fcb9e7ad3872 WatchSource:0}: Error finding container 197276a3f76c5bf4503157565f9e07f9e1dffb7fab50456fad23fcb9e7ad3872: Status 404 returned error can't find the container with id 197276a3f76c5bf4503157565f9e07f9e1dffb7fab50456fad23fcb9e7ad3872 Sep 17 23:17:29.120643 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:29.120608 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-7kxv5" event={"ID":"6830ef77-3cb3-46a6-b93c-d9e831f1e0fd","Type":"ContainerStarted","Data":"197276a3f76c5bf4503157565f9e07f9e1dffb7fab50456fad23fcb9e7ad3872"} Sep 17 23:17:29.121765 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:29.121743 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-x5mz7" event={"ID":"712ec0c0-4bab-4ac5-8329-f9fcf561deda","Type":"ContainerStarted","Data":"1563f456a6a572ce140acd4cb1b4f2daaf6bfea3c183731e416b19a6ba65b1bb"} Sep 17 23:17:30.129481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:30.129447 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-x5mz7" event={"ID":"712ec0c0-4bab-4ac5-8329-f9fcf561deda","Type":"ContainerStarted","Data":"8aad28353bf1008562b508158b5b98f11a4c1f95b72c549742c9fc747dccc535"} Sep 17 23:17:30.129923 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:30.129536 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-x5mz7" Sep 17 23:17:30.146569 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:30.146522 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-x5mz7" podStartSLOduration=1.7268846230000001 podStartE2EDuration="3.14649015s" totalImagesPullingTime="1.419605527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:17:30.143813918 +0000 UTC m=+689.077721058" watchObservedRunningTime="2026-09-17 23:17:30.14649015 +0000 UTC m=+689.080397284" Sep 17 23:17:31.135988 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:31.135957 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-7kxv5" event={"ID":"6830ef77-3cb3-46a6-b93c-d9e831f1e0fd","Type":"ContainerStarted","Data":"8ba0d015b78aee0908662d0770c97c2181ccd1cfc2135e8672b5331cbb908767"} Sep 17 23:17:31.151207 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:31.151162 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-7kxv5" podStartSLOduration=1.487600145 podStartE2EDuration="3.15114655s" totalImagesPullingTime="1.663546405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:17:31.149248773 +0000 UTC m=+690.083155910" watchObservedRunningTime="2026-09-17 23:17:31.15114655 +0000 UTC m=+690.085053685" Sep 17 23:17:32.406891 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:32.406859 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7kxv5"] Sep 17 23:17:33.145965 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:33.145927 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-7kxv5" podUID="6830ef77-3cb3-46a6-b93c-d9e831f1e0fd" containerName="authorino" containerID="cri-o://8ba0d015b78aee0908662d0770c97c2181ccd1cfc2135e8672b5331cbb908767" gracePeriod=30 Sep 17 23:17:33.400974 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:33.400914 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-7kxv5" Sep 17 23:17:33.496254 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:33.496220 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mb2f\" (UniqueName: \"kubernetes.io/projected/6830ef77-3cb3-46a6-b93c-d9e831f1e0fd-kube-api-access-8mb2f\") pod \"6830ef77-3cb3-46a6-b93c-d9e831f1e0fd\" (UID: \"6830ef77-3cb3-46a6-b93c-d9e831f1e0fd\") " Sep 17 23:17:33.498303 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:33.498283 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6830ef77-3cb3-46a6-b93c-d9e831f1e0fd-kube-api-access-8mb2f" (OuterVolumeSpecName: "kube-api-access-8mb2f") pod "6830ef77-3cb3-46a6-b93c-d9e831f1e0fd" (UID: "6830ef77-3cb3-46a6-b93c-d9e831f1e0fd"). InnerVolumeSpecName "kube-api-access-8mb2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:17:33.597165 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:33.597128 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mb2f\" (UniqueName: \"kubernetes.io/projected/6830ef77-3cb3-46a6-b93c-d9e831f1e0fd-kube-api-access-8mb2f\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:17:34.153783 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:34.153749 2582 generic.go:356] "Generic (PLEG): container finished" podID="6830ef77-3cb3-46a6-b93c-d9e831f1e0fd" containerID="8ba0d015b78aee0908662d0770c97c2181ccd1cfc2135e8672b5331cbb908767" exitCode=0 Sep 17 23:17:34.153946 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:34.153786 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-7kxv5" Sep 17 23:17:34.153946 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:34.153821 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-7kxv5" event={"ID":"6830ef77-3cb3-46a6-b93c-d9e831f1e0fd","Type":"ContainerDied","Data":"8ba0d015b78aee0908662d0770c97c2181ccd1cfc2135e8672b5331cbb908767"} Sep 17 23:17:34.153946 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:34.153850 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-7kxv5" event={"ID":"6830ef77-3cb3-46a6-b93c-d9e831f1e0fd","Type":"ContainerDied","Data":"197276a3f76c5bf4503157565f9e07f9e1dffb7fab50456fad23fcb9e7ad3872"} Sep 17 23:17:34.153946 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:34.153866 2582 scope.go:117] "RemoveContainer" containerID="8ba0d015b78aee0908662d0770c97c2181ccd1cfc2135e8672b5331cbb908767" Sep 17 23:17:34.163094 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:34.163076 2582 scope.go:117] "RemoveContainer" containerID="8ba0d015b78aee0908662d0770c97c2181ccd1cfc2135e8672b5331cbb908767" Sep 17 23:17:34.163363 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:17:34.163345 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba0d015b78aee0908662d0770c97c2181ccd1cfc2135e8672b5331cbb908767\": container with ID starting with 8ba0d015b78aee0908662d0770c97c2181ccd1cfc2135e8672b5331cbb908767 not found: ID does not exist" containerID="8ba0d015b78aee0908662d0770c97c2181ccd1cfc2135e8672b5331cbb908767" Sep 17 23:17:34.163419 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:34.163371 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba0d015b78aee0908662d0770c97c2181ccd1cfc2135e8672b5331cbb908767"} err="failed to get container status \"8ba0d015b78aee0908662d0770c97c2181ccd1cfc2135e8672b5331cbb908767\": rpc error: code = NotFound desc = could not find container \"8ba0d015b78aee0908662d0770c97c2181ccd1cfc2135e8672b5331cbb908767\": container with ID starting with 8ba0d015b78aee0908662d0770c97c2181ccd1cfc2135e8672b5331cbb908767 not found: ID does not exist" Sep 17 23:17:34.173776 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:34.173755 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7kxv5"] Sep 17 23:17:34.177512 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:34.177485 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7kxv5"] Sep 17 23:17:35.293876 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:35.293846 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6830ef77-3cb3-46a6-b93c-d9e831f1e0fd" path="/var/lib/kubelet/pods/6830ef77-3cb3-46a6-b93c-d9e831f1e0fd/volumes" Sep 17 23:17:41.137356 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:41.137319 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-x5mz7" Sep 17 23:17:43.416933 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:43.416899 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-x5mz7"] Sep 17 23:17:43.417375 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:43.417111 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-x5mz7" podUID="712ec0c0-4bab-4ac5-8329-f9fcf561deda" containerName="limitador" containerID="cri-o://8aad28353bf1008562b508158b5b98f11a4c1f95b72c549742c9fc747dccc535" gracePeriod=30 Sep 17 23:17:44.204925 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:44.204838 2582 generic.go:356] "Generic (PLEG): container finished" podID="712ec0c0-4bab-4ac5-8329-f9fcf561deda" containerID="8aad28353bf1008562b508158b5b98f11a4c1f95b72c549742c9fc747dccc535" exitCode=0 Sep 17 23:17:44.205062 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:44.204917 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-x5mz7" event={"ID":"712ec0c0-4bab-4ac5-8329-f9fcf561deda","Type":"ContainerDied","Data":"8aad28353bf1008562b508158b5b98f11a4c1f95b72c549742c9fc747dccc535"} Sep 17 23:17:44.370416 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:44.370389 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-x5mz7" Sep 17 23:17:44.499437 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:44.499407 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/712ec0c0-4bab-4ac5-8329-f9fcf561deda-config-file\") pod \"712ec0c0-4bab-4ac5-8329-f9fcf561deda\" (UID: \"712ec0c0-4bab-4ac5-8329-f9fcf561deda\") " Sep 17 23:17:44.499934 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:44.499457 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8ns7\" (UniqueName: \"kubernetes.io/projected/712ec0c0-4bab-4ac5-8329-f9fcf561deda-kube-api-access-k8ns7\") pod \"712ec0c0-4bab-4ac5-8329-f9fcf561deda\" (UID: \"712ec0c0-4bab-4ac5-8329-f9fcf561deda\") " Sep 17 23:17:44.499934 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:44.499870 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712ec0c0-4bab-4ac5-8329-f9fcf561deda-config-file" (OuterVolumeSpecName: "config-file") pod "712ec0c0-4bab-4ac5-8329-f9fcf561deda" (UID: "712ec0c0-4bab-4ac5-8329-f9fcf561deda"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:17:44.501595 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:44.501561 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712ec0c0-4bab-4ac5-8329-f9fcf561deda-kube-api-access-k8ns7" (OuterVolumeSpecName: "kube-api-access-k8ns7") pod "712ec0c0-4bab-4ac5-8329-f9fcf561deda" (UID: "712ec0c0-4bab-4ac5-8329-f9fcf561deda"). InnerVolumeSpecName "kube-api-access-k8ns7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:17:44.600536 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:44.600472 2582 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/712ec0c0-4bab-4ac5-8329-f9fcf561deda-config-file\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:17:44.600536 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:44.600533 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8ns7\" (UniqueName: \"kubernetes.io/projected/712ec0c0-4bab-4ac5-8329-f9fcf561deda-kube-api-access-k8ns7\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:17:45.211835 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:45.211801 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-x5mz7" event={"ID":"712ec0c0-4bab-4ac5-8329-f9fcf561deda","Type":"ContainerDied","Data":"1563f456a6a572ce140acd4cb1b4f2daaf6bfea3c183731e416b19a6ba65b1bb"} Sep 17 23:17:45.212033 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:45.211864 2582 scope.go:117] "RemoveContainer" containerID="8aad28353bf1008562b508158b5b98f11a4c1f95b72c549742c9fc747dccc535" Sep 17 23:17:45.212033 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:45.211811 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-x5mz7" Sep 17 23:17:45.232295 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:45.232264 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-x5mz7"] Sep 17 23:17:45.236655 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:45.236629 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-x5mz7"] Sep 17 23:17:45.293934 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:45.293902 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="712ec0c0-4bab-4ac5-8329-f9fcf561deda" path="/var/lib/kubelet/pods/712ec0c0-4bab-4ac5-8329-f9fcf561deda/volumes" Sep 17 23:17:45.875229 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:45.875200 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-z5h2z"] Sep 17 23:17:45.875622 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:45.875609 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6830ef77-3cb3-46a6-b93c-d9e831f1e0fd" containerName="authorino" Sep 17 23:17:45.875661 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:45.875623 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6830ef77-3cb3-46a6-b93c-d9e831f1e0fd" containerName="authorino" Sep 17 23:17:45.875661 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:45.875642 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="712ec0c0-4bab-4ac5-8329-f9fcf561deda" containerName="limitador" Sep 17 23:17:45.875661 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:45.875649 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="712ec0c0-4bab-4ac5-8329-f9fcf561deda" containerName="limitador" Sep 17 23:17:45.875779 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:45.875707 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="712ec0c0-4bab-4ac5-8329-f9fcf561deda" containerName="limitador" Sep 17 23:17:45.875779 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:45.875717 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6830ef77-3cb3-46a6-b93c-d9e831f1e0fd" containerName="authorino" Sep 17 23:17:45.879928 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:45.879889 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5h2z" Sep 17 23:17:45.882490 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:45.882466 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-5r9x6\"" Sep 17 23:17:45.882656 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:45.882466 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 23:17:45.882656 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:45.882476 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 23:17:45.883156 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:45.883136 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 17 23:17:45.891604 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:45.891566 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-z5h2z"] Sep 17 23:17:46.014925 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:46.014885 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/402d4843-f631-4d80-b8ec-c5999f751f53-data\") pod \"postgres-69fb4594cf-z5h2z\" (UID: \"402d4843-f631-4d80-b8ec-c5999f751f53\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5h2z" Sep 17 23:17:46.015100 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:46.014976 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zvs2\" (UniqueName: \"kubernetes.io/projected/402d4843-f631-4d80-b8ec-c5999f751f53-kube-api-access-9zvs2\") pod \"postgres-69fb4594cf-z5h2z\" (UID: \"402d4843-f631-4d80-b8ec-c5999f751f53\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5h2z" Sep 17 23:17:46.116203 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:46.116172 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zvs2\" (UniqueName: \"kubernetes.io/projected/402d4843-f631-4d80-b8ec-c5999f751f53-kube-api-access-9zvs2\") pod \"postgres-69fb4594cf-z5h2z\" (UID: \"402d4843-f631-4d80-b8ec-c5999f751f53\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5h2z" Sep 17 23:17:46.116341 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:46.116232 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/402d4843-f631-4d80-b8ec-c5999f751f53-data\") pod \"postgres-69fb4594cf-z5h2z\" (UID: \"402d4843-f631-4d80-b8ec-c5999f751f53\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5h2z" Sep 17 23:17:46.116634 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:46.116617 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/402d4843-f631-4d80-b8ec-c5999f751f53-data\") pod \"postgres-69fb4594cf-z5h2z\" (UID: \"402d4843-f631-4d80-b8ec-c5999f751f53\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5h2z" Sep 17 23:17:46.124093 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:46.124068 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zvs2\" (UniqueName: \"kubernetes.io/projected/402d4843-f631-4d80-b8ec-c5999f751f53-kube-api-access-9zvs2\") pod \"postgres-69fb4594cf-z5h2z\" (UID: \"402d4843-f631-4d80-b8ec-c5999f751f53\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5h2z" Sep 17 23:17:46.192343 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:46.192250 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5h2z" Sep 17 23:17:46.483817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:46.483571 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-z5h2z"] Sep 17 23:17:46.486639 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:17:46.486607 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402d4843_f631_4d80_b8ec_c5999f751f53.slice/crio-f3510433a89c3463dbe2bfc220e0336341b320e3c68496dbcd8583b5a5fb1272 WatchSource:0}: Error finding container f3510433a89c3463dbe2bfc220e0336341b320e3c68496dbcd8583b5a5fb1272: Status 404 returned error can't find the container with id f3510433a89c3463dbe2bfc220e0336341b320e3c68496dbcd8583b5a5fb1272 Sep 17 23:17:47.226554 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:47.226491 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5h2z" event={"ID":"402d4843-f631-4d80-b8ec-c5999f751f53","Type":"ContainerStarted","Data":"f3510433a89c3463dbe2bfc220e0336341b320e3c68496dbcd8583b5a5fb1272"} Sep 17 23:17:52.256863 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:52.256817 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5h2z" event={"ID":"402d4843-f631-4d80-b8ec-c5999f751f53","Type":"ContainerStarted","Data":"576855447fa89d8128c79cd52906e1c2c51a9cb3962590b43ee8a6d41a2695ef"} Sep 17 23:17:52.257284 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:52.256902 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5h2z" Sep 17 23:17:52.272057 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:52.271997 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5h2z" podStartSLOduration=1.930804968 podStartE2EDuration="7.271977514s" totalImagesPullingTime="5.341172546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:17:52.27141825 +0000 UTC m=+711.205325389" watchObservedRunningTime="2026-09-17 23:17:52.271977514 +0000 UTC m=+711.205884651" Sep 17 23:17:58.290873 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:58.290843 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-z5h2z" Sep 17 23:17:58.825630 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:58.825597 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-tzhx5"] Sep 17 23:17:58.829006 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:58.828989 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-tzhx5" Sep 17 23:17:58.831106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:58.831084 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cmz9j\"" Sep 17 23:17:58.835313 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:58.835274 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-tzhx5"] Sep 17 23:17:58.925223 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:58.925168 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbz9b\" (UniqueName: \"kubernetes.io/projected/9ec2fd3b-9ef6-46cd-b7b9-22402ece2ee8-kube-api-access-zbz9b\") pod \"authorino-557f876f69-tzhx5\" (UID: \"9ec2fd3b-9ef6-46cd-b7b9-22402ece2ee8\") " pod="kuadrant-system/authorino-557f876f69-tzhx5" Sep 17 23:17:58.956985 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:58.956946 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-tzhx5"] Sep 17 23:17:58.957283 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:17:58.957259 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-zbz9b], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-tzhx5" podUID="9ec2fd3b-9ef6-46cd-b7b9-22402ece2ee8" Sep 17 23:17:58.986111 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:58.986075 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-w48pn"] Sep 17 23:17:58.989667 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:58.989647 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-w48pn" Sep 17 23:17:58.991899 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:58.991876 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 17 23:17:58.998183 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:58.998159 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-w48pn"] Sep 17 23:17:59.026220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.026193 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbz9b\" (UniqueName: \"kubernetes.io/projected/9ec2fd3b-9ef6-46cd-b7b9-22402ece2ee8-kube-api-access-zbz9b\") pod \"authorino-557f876f69-tzhx5\" (UID: \"9ec2fd3b-9ef6-46cd-b7b9-22402ece2ee8\") " pod="kuadrant-system/authorino-557f876f69-tzhx5" Sep 17 23:17:59.026358 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.026254 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ba33b77-e273-4f3a-a03f-10fcd27a80dc-tls-cert\") pod \"authorino-6d4dfc649f-w48pn\" (UID: \"9ba33b77-e273-4f3a-a03f-10fcd27a80dc\") " pod="kuadrant-system/authorino-6d4dfc649f-w48pn" Sep 17 23:17:59.026358 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.026307 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5kpc\" (UniqueName: \"kubernetes.io/projected/9ba33b77-e273-4f3a-a03f-10fcd27a80dc-kube-api-access-l5kpc\") pod \"authorino-6d4dfc649f-w48pn\" (UID: \"9ba33b77-e273-4f3a-a03f-10fcd27a80dc\") " pod="kuadrant-system/authorino-6d4dfc649f-w48pn" Sep 17 23:17:59.036083 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.036056 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbz9b\" (UniqueName: \"kubernetes.io/projected/9ec2fd3b-9ef6-46cd-b7b9-22402ece2ee8-kube-api-access-zbz9b\") pod \"authorino-557f876f69-tzhx5\" (UID: \"9ec2fd3b-9ef6-46cd-b7b9-22402ece2ee8\") " pod="kuadrant-system/authorino-557f876f69-tzhx5" Sep 17 23:17:59.065239 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.065215 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-w48pn"] Sep 17 23:17:59.065524 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:17:59.065488 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-l5kpc tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-w48pn" podUID="9ba33b77-e273-4f3a-a03f-10fcd27a80dc" Sep 17 23:17:59.093755 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.093694 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-85654986b8-22pdn"] Sep 17 23:17:59.097322 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.097303 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85654986b8-22pdn" Sep 17 23:17:59.107518 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.107473 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85654986b8-22pdn"] Sep 17 23:17:59.127767 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.127742 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpd9t\" (UniqueName: \"kubernetes.io/projected/635f5734-34fe-48ca-b175-86342fba02d3-kube-api-access-hpd9t\") pod \"authorino-85654986b8-22pdn\" (UID: \"635f5734-34fe-48ca-b175-86342fba02d3\") " pod="kuadrant-system/authorino-85654986b8-22pdn" Sep 17 23:17:59.127911 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.127776 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ba33b77-e273-4f3a-a03f-10fcd27a80dc-tls-cert\") pod \"authorino-6d4dfc649f-w48pn\" (UID: \"9ba33b77-e273-4f3a-a03f-10fcd27a80dc\") " pod="kuadrant-system/authorino-6d4dfc649f-w48pn" Sep 17 23:17:59.127911 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.127816 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5kpc\" (UniqueName: \"kubernetes.io/projected/9ba33b77-e273-4f3a-a03f-10fcd27a80dc-kube-api-access-l5kpc\") pod \"authorino-6d4dfc649f-w48pn\" (UID: \"9ba33b77-e273-4f3a-a03f-10fcd27a80dc\") " pod="kuadrant-system/authorino-6d4dfc649f-w48pn" Sep 17 23:17:59.127911 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.127884 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/635f5734-34fe-48ca-b175-86342fba02d3-tls-cert\") pod \"authorino-85654986b8-22pdn\" (UID: \"635f5734-34fe-48ca-b175-86342fba02d3\") " pod="kuadrant-system/authorino-85654986b8-22pdn" Sep 17 23:17:59.130254 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.130227 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ba33b77-e273-4f3a-a03f-10fcd27a80dc-tls-cert\") pod \"authorino-6d4dfc649f-w48pn\" (UID: \"9ba33b77-e273-4f3a-a03f-10fcd27a80dc\") " pod="kuadrant-system/authorino-6d4dfc649f-w48pn" Sep 17 23:17:59.136231 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.136211 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5kpc\" (UniqueName: \"kubernetes.io/projected/9ba33b77-e273-4f3a-a03f-10fcd27a80dc-kube-api-access-l5kpc\") pod \"authorino-6d4dfc649f-w48pn\" (UID: \"9ba33b77-e273-4f3a-a03f-10fcd27a80dc\") " pod="kuadrant-system/authorino-6d4dfc649f-w48pn" Sep 17 23:17:59.228928 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.228895 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/635f5734-34fe-48ca-b175-86342fba02d3-tls-cert\") pod \"authorino-85654986b8-22pdn\" (UID: \"635f5734-34fe-48ca-b175-86342fba02d3\") " pod="kuadrant-system/authorino-85654986b8-22pdn" Sep 17 23:17:59.229091 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.228987 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpd9t\" (UniqueName: \"kubernetes.io/projected/635f5734-34fe-48ca-b175-86342fba02d3-kube-api-access-hpd9t\") pod \"authorino-85654986b8-22pdn\" (UID: \"635f5734-34fe-48ca-b175-86342fba02d3\") " pod="kuadrant-system/authorino-85654986b8-22pdn" Sep 17 23:17:59.231288 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.231269 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/635f5734-34fe-48ca-b175-86342fba02d3-tls-cert\") pod \"authorino-85654986b8-22pdn\" (UID: \"635f5734-34fe-48ca-b175-86342fba02d3\") " pod="kuadrant-system/authorino-85654986b8-22pdn" Sep 17 23:17:59.236989 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.236960 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpd9t\" (UniqueName: \"kubernetes.io/projected/635f5734-34fe-48ca-b175-86342fba02d3-kube-api-access-hpd9t\") pod \"authorino-85654986b8-22pdn\" (UID: \"635f5734-34fe-48ca-b175-86342fba02d3\") " pod="kuadrant-system/authorino-85654986b8-22pdn" Sep 17 23:17:59.291954 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.291932 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-tzhx5" Sep 17 23:17:59.292388 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.291932 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-w48pn" Sep 17 23:17:59.299550 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.299527 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-tzhx5" Sep 17 23:17:59.302440 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.302425 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-w48pn" Sep 17 23:17:59.329865 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.329838 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5kpc\" (UniqueName: \"kubernetes.io/projected/9ba33b77-e273-4f3a-a03f-10fcd27a80dc-kube-api-access-l5kpc\") pod \"9ba33b77-e273-4f3a-a03f-10fcd27a80dc\" (UID: \"9ba33b77-e273-4f3a-a03f-10fcd27a80dc\") " Sep 17 23:17:59.330000 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.329873 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbz9b\" (UniqueName: \"kubernetes.io/projected/9ec2fd3b-9ef6-46cd-b7b9-22402ece2ee8-kube-api-access-zbz9b\") pod \"9ec2fd3b-9ef6-46cd-b7b9-22402ece2ee8\" (UID: \"9ec2fd3b-9ef6-46cd-b7b9-22402ece2ee8\") " Sep 17 23:17:59.330000 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.329919 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ba33b77-e273-4f3a-a03f-10fcd27a80dc-tls-cert\") pod \"9ba33b77-e273-4f3a-a03f-10fcd27a80dc\" (UID: \"9ba33b77-e273-4f3a-a03f-10fcd27a80dc\") " Sep 17 23:17:59.332065 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.332039 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba33b77-e273-4f3a-a03f-10fcd27a80dc-kube-api-access-l5kpc" (OuterVolumeSpecName: "kube-api-access-l5kpc") pod "9ba33b77-e273-4f3a-a03f-10fcd27a80dc" (UID: "9ba33b77-e273-4f3a-a03f-10fcd27a80dc"). InnerVolumeSpecName "kube-api-access-l5kpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:17:59.332169 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.332065 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ba33b77-e273-4f3a-a03f-10fcd27a80dc-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9ba33b77-e273-4f3a-a03f-10fcd27a80dc" (UID: "9ba33b77-e273-4f3a-a03f-10fcd27a80dc"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:17:59.332169 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.332080 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec2fd3b-9ef6-46cd-b7b9-22402ece2ee8-kube-api-access-zbz9b" (OuterVolumeSpecName: "kube-api-access-zbz9b") pod "9ec2fd3b-9ef6-46cd-b7b9-22402ece2ee8" (UID: "9ec2fd3b-9ef6-46cd-b7b9-22402ece2ee8"). InnerVolumeSpecName "kube-api-access-zbz9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:17:59.408325 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.408257 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85654986b8-22pdn" Sep 17 23:17:59.430747 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.430717 2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ba33b77-e273-4f3a-a03f-10fcd27a80dc-tls-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:17:59.430747 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.430747 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5kpc\" (UniqueName: \"kubernetes.io/projected/9ba33b77-e273-4f3a-a03f-10fcd27a80dc-kube-api-access-l5kpc\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:17:59.430882 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.430759 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbz9b\" (UniqueName: \"kubernetes.io/projected/9ec2fd3b-9ef6-46cd-b7b9-22402ece2ee8-kube-api-access-zbz9b\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:17:59.678553 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:17:59.676340 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85654986b8-22pdn"] Sep 17 23:18:00.298645 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:00.298545 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85654986b8-22pdn" event={"ID":"635f5734-34fe-48ca-b175-86342fba02d3","Type":"ContainerStarted","Data":"11b73048344a125bd8c3a831de71237d500937ec44642df10fa8994f54a8d983"} Sep 17 23:18:00.298645 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:00.298601 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85654986b8-22pdn" event={"ID":"635f5734-34fe-48ca-b175-86342fba02d3","Type":"ContainerStarted","Data":"4467c60dc0f241316aa52509f53d677e9dc1e76643cb2c6b999cebfd8a9593a3"} Sep 17 23:18:00.299138 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:00.298718 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-w48pn" Sep 17 23:18:00.299138 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:00.298890 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-tzhx5" Sep 17 23:18:00.313320 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:00.313267 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-85654986b8-22pdn" podStartSLOduration=0.855252932 podStartE2EDuration="1.313248971s" totalImagesPullingTime="457.996039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:00.31322315 +0000 UTC m=+719.247130287" watchObservedRunningTime="2026-09-17 23:18:00.313248971 +0000 UTC m=+719.247156109" Sep 17 23:18:00.335981 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:00.335950 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-tzhx5"] Sep 17 23:18:00.345588 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:00.345559 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-tzhx5"] Sep 17 23:18:00.375332 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:00.375299 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-w48pn"] Sep 17 23:18:00.376993 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:00.376968 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-w48pn"] Sep 17 23:18:01.295550 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:01.295491 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ba33b77-e273-4f3a-a03f-10fcd27a80dc" path="/var/lib/kubelet/pods/9ba33b77-e273-4f3a-a03f-10fcd27a80dc/volumes" Sep 17 23:18:01.295781 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:01.295768 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ec2fd3b-9ef6-46cd-b7b9-22402ece2ee8" path="/var/lib/kubelet/pods/9ec2fd3b-9ef6-46cd-b7b9-22402ece2ee8/volumes" Sep 17 23:18:05.299613 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:05.299584 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7cd595f459-2wdxz"] Sep 17 23:18:05.305773 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:05.305750 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd595f459-2wdxz" Sep 17 23:18:05.307995 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:05.307970 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 17 23:18:05.308820 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:05.308794 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 17 23:18:05.308919 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:05.308834 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 17 23:18:05.308919 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:05.308834 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2mntz\"" Sep 17 23:18:05.317339 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:05.317315 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7cd595f459-2wdxz"] Sep 17 23:18:05.383394 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:05.383366 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ed809b5-b370-4deb-8b01-69513dabaab3-webhook-cert\") pod \"maas-controller-7cd595f459-2wdxz\" (UID: \"7ed809b5-b370-4deb-8b01-69513dabaab3\") " pod="opendatahub/maas-controller-7cd595f459-2wdxz" Sep 17 23:18:05.383602 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:05.383410 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed809b5-b370-4deb-8b01-69513dabaab3-metrics-certs\") pod \"maas-controller-7cd595f459-2wdxz\" (UID: \"7ed809b5-b370-4deb-8b01-69513dabaab3\") " pod="opendatahub/maas-controller-7cd595f459-2wdxz" Sep 17 23:18:05.383602 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:05.383441 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkd8f\" (UniqueName: \"kubernetes.io/projected/7ed809b5-b370-4deb-8b01-69513dabaab3-kube-api-access-gkd8f\") pod \"maas-controller-7cd595f459-2wdxz\" (UID: \"7ed809b5-b370-4deb-8b01-69513dabaab3\") " pod="opendatahub/maas-controller-7cd595f459-2wdxz" Sep 17 23:18:05.484734 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:05.484698 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed809b5-b370-4deb-8b01-69513dabaab3-metrics-certs\") pod \"maas-controller-7cd595f459-2wdxz\" (UID: \"7ed809b5-b370-4deb-8b01-69513dabaab3\") " pod="opendatahub/maas-controller-7cd595f459-2wdxz" Sep 17 23:18:05.484892 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:05.484755 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkd8f\" (UniqueName: \"kubernetes.io/projected/7ed809b5-b370-4deb-8b01-69513dabaab3-kube-api-access-gkd8f\") pod \"maas-controller-7cd595f459-2wdxz\" (UID: \"7ed809b5-b370-4deb-8b01-69513dabaab3\") " pod="opendatahub/maas-controller-7cd595f459-2wdxz" Sep 17 23:18:05.484892 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:05.484814 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ed809b5-b370-4deb-8b01-69513dabaab3-webhook-cert\") pod \"maas-controller-7cd595f459-2wdxz\" (UID: \"7ed809b5-b370-4deb-8b01-69513dabaab3\") " pod="opendatahub/maas-controller-7cd595f459-2wdxz" Sep 17 23:18:05.485004 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:18:05.484924 2582 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 17 23:18:05.485043 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:18:05.485011 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ed809b5-b370-4deb-8b01-69513dabaab3-webhook-cert podName:7ed809b5-b370-4deb-8b01-69513dabaab3 nodeName:}" failed. No retries permitted until 2026-09-17 23:18:05.984990146 +0000 UTC m=+724.918897263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/7ed809b5-b370-4deb-8b01-69513dabaab3-webhook-cert") pod "maas-controller-7cd595f459-2wdxz" (UID: "7ed809b5-b370-4deb-8b01-69513dabaab3") : secret "maas-controller-webhook-cert" not found Sep 17 23:18:05.487178 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:05.487156 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed809b5-b370-4deb-8b01-69513dabaab3-metrics-certs\") pod \"maas-controller-7cd595f459-2wdxz\" (UID: \"7ed809b5-b370-4deb-8b01-69513dabaab3\") " pod="opendatahub/maas-controller-7cd595f459-2wdxz" Sep 17 23:18:05.492894 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:05.492869 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkd8f\" (UniqueName: \"kubernetes.io/projected/7ed809b5-b370-4deb-8b01-69513dabaab3-kube-api-access-gkd8f\") pod \"maas-controller-7cd595f459-2wdxz\" (UID: \"7ed809b5-b370-4deb-8b01-69513dabaab3\") " pod="opendatahub/maas-controller-7cd595f459-2wdxz" Sep 17 23:18:05.989698 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:05.989668 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ed809b5-b370-4deb-8b01-69513dabaab3-webhook-cert\") pod \"maas-controller-7cd595f459-2wdxz\" (UID: \"7ed809b5-b370-4deb-8b01-69513dabaab3\") " pod="opendatahub/maas-controller-7cd595f459-2wdxz" Sep 17 23:18:05.992038 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:05.992014 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ed809b5-b370-4deb-8b01-69513dabaab3-webhook-cert\") pod \"maas-controller-7cd595f459-2wdxz\" (UID: \"7ed809b5-b370-4deb-8b01-69513dabaab3\") " pod="opendatahub/maas-controller-7cd595f459-2wdxz" Sep 17 23:18:06.215366 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:06.215327 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd595f459-2wdxz" Sep 17 23:18:06.485274 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:06.485248 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7cd595f459-2wdxz"] Sep 17 23:18:06.487029 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:18:06.487001 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed809b5_b370_4deb_8b01_69513dabaab3.slice/crio-18ad73c6464dada3d4d3a0c2852ae5a8f2c7261f959544bc13ddff518119bc20 WatchSource:0}: Error finding container 18ad73c6464dada3d4d3a0c2852ae5a8f2c7261f959544bc13ddff518119bc20: Status 404 returned error can't find the container with id 18ad73c6464dada3d4d3a0c2852ae5a8f2c7261f959544bc13ddff518119bc20 Sep 17 23:18:07.342952 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:07.342912 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd595f459-2wdxz" event={"ID":"7ed809b5-b370-4deb-8b01-69513dabaab3","Type":"ContainerStarted","Data":"18ad73c6464dada3d4d3a0c2852ae5a8f2c7261f959544bc13ddff518119bc20"} Sep 17 23:18:09.354804 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:09.354766 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd595f459-2wdxz" event={"ID":"7ed809b5-b370-4deb-8b01-69513dabaab3","Type":"ContainerStarted","Data":"3b122221a8a1ebfb195396a4e36cec0e8edd9c836396140ff07d52c69ee9b9c9"} Sep 17 23:18:09.355182 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:09.354850 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7cd595f459-2wdxz" Sep 17 23:18:09.372124 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:09.372072 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7cd595f459-2wdxz" podStartSLOduration=2.147551362 podStartE2EDuration="4.372057169s" totalImagesPullingTime="2.224505807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:09.370098684 +0000 UTC m=+728.304005821" watchObservedRunningTime="2026-09-17 23:18:09.372057169 +0000 UTC m=+728.305964305" Sep 17 23:18:20.362364 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:20.362331 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7cd595f459-2wdxz" Sep 17 23:18:42.490229 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:42.490190 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-pxpjd"] Sep 17 23:18:42.516370 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:42.516333 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-pxpjd"] Sep 17 23:18:42.516576 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:42.516412 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-pxpjd" Sep 17 23:18:42.518674 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:42.518645 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 17 23:18:42.518909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:42.518892 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-wlnsl\"" Sep 17 23:18:42.614265 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:42.614230 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6287a30-1142-48b6-a7ea-50b426e7b598-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-pxpjd\" (UID: \"b6287a30-1142-48b6-a7ea-50b426e7b598\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-pxpjd" Sep 17 23:18:42.614265 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:42.614272 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wx9l\" (UniqueName: \"kubernetes.io/projected/b6287a30-1142-48b6-a7ea-50b426e7b598-kube-api-access-8wx9l\") pod \"payload-processing-5b54b7fbb7-pxpjd\" (UID: \"b6287a30-1142-48b6-a7ea-50b426e7b598\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-pxpjd" Sep 17 23:18:42.715577 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:42.715538 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6287a30-1142-48b6-a7ea-50b426e7b598-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-pxpjd\" (UID: \"b6287a30-1142-48b6-a7ea-50b426e7b598\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-pxpjd" Sep 17 23:18:42.715761 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:42.715611 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wx9l\" (UniqueName: \"kubernetes.io/projected/b6287a30-1142-48b6-a7ea-50b426e7b598-kube-api-access-8wx9l\") pod \"payload-processing-5b54b7fbb7-pxpjd\" (UID: \"b6287a30-1142-48b6-a7ea-50b426e7b598\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-pxpjd" Sep 17 23:18:42.716420 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:42.716393 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6287a30-1142-48b6-a7ea-50b426e7b598-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-pxpjd\" (UID: \"b6287a30-1142-48b6-a7ea-50b426e7b598\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-pxpjd" Sep 17 23:18:42.724779 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:42.724754 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wx9l\" (UniqueName: \"kubernetes.io/projected/b6287a30-1142-48b6-a7ea-50b426e7b598-kube-api-access-8wx9l\") pod \"payload-processing-5b54b7fbb7-pxpjd\" (UID: \"b6287a30-1142-48b6-a7ea-50b426e7b598\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-pxpjd" Sep 17 23:18:42.828675 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:42.828596 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-pxpjd" Sep 17 23:18:43.111160 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:43.111094 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-pxpjd"] Sep 17 23:18:43.114051 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:18:43.114025 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6287a30_1142_48b6_a7ea_50b426e7b598.slice/crio-5b8e248fafa91fcff04ec03d7c9f4dc6fbcae4ce0cff789997733e40064c0ad9 WatchSource:0}: Error finding container 5b8e248fafa91fcff04ec03d7c9f4dc6fbcae4ce0cff789997733e40064c0ad9: Status 404 returned error can't find the container with id 5b8e248fafa91fcff04ec03d7c9f4dc6fbcae4ce0cff789997733e40064c0ad9 Sep 17 23:18:43.528249 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:43.528213 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-pxpjd" event={"ID":"b6287a30-1142-48b6-a7ea-50b426e7b598","Type":"ContainerStarted","Data":"5b8e248fafa91fcff04ec03d7c9f4dc6fbcae4ce0cff789997733e40064c0ad9"} Sep 17 23:18:45.540036 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:45.540005 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-pxpjd" event={"ID":"b6287a30-1142-48b6-a7ea-50b426e7b598","Type":"ContainerStarted","Data":"828875edb9d99ceef9f3b0db61e3fc55d40147d4560bc8771b4d70c7faca0d1b"} Sep 17 23:18:45.540423 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:45.540067 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-pxpjd" Sep 17 23:18:45.562120 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:45.556867 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-pxpjd" podStartSLOduration=2.062860108 podStartE2EDuration="3.55684382s" totalImagesPullingTime="1.493983712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:45.553034711 +0000 UTC m=+764.486941850" watchObservedRunningTime="2026-09-17 23:18:45.55684382 +0000 UTC m=+764.490750959" Sep 17 23:18:50.112722 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:50.112682 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-b66bc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 17 23:18:50.113217 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:50.112763 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" podUID="7b3c4835-32b2-46a3-8d2d-f64071d509fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 17 23:18:52.583046 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:52.583010 2582 generic.go:356] "Generic (PLEG): container finished" podID="7b3c4835-32b2-46a3-8d2d-f64071d509fd" containerID="3cdfbef3ea26d8fbf7266a1089068bfc95ca7879fbfac4a5664d17881f036867" exitCode=0 Sep 17 23:18:52.583454 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:52.583080 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" event={"ID":"7b3c4835-32b2-46a3-8d2d-f64071d509fd","Type":"ContainerDied","Data":"3cdfbef3ea26d8fbf7266a1089068bfc95ca7879fbfac4a5664d17881f036867"} Sep 17 23:18:52.583552 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:52.583489 2582 scope.go:117] "RemoveContainer" containerID="3cdfbef3ea26d8fbf7266a1089068bfc95ca7879fbfac4a5664d17881f036867" Sep 17 23:18:52.584553 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:52.584525 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:18:52.584617 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:52.584604 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:18:52.584661 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:52.584650 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:18:53.595027 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:53.594977 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" event={"ID":"7b3c4835-32b2-46a3-8d2d-f64071d509fd","Type":"ContainerStarted","Data":"af9ab02dce149f6f8e949dc6bdb6f7a9db43b6ac6961c45075061ba8a7fb64fb"} Sep 17 23:18:53.633828 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:53.633792 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:18:53.634095 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:53.634073 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-b66bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 17 23:18:53.634145 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:53.634124 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" podUID="7b3c4835-32b2-46a3-8d2d-f64071d509fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 17 23:18:54.633339 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:54.633299 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-b66bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 17 23:18:54.633830 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:54.633359 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" podUID="7b3c4835-32b2-46a3-8d2d-f64071d509fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 17 23:18:55.633550 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:55.633494 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-b66bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 17 23:18:55.634009 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:55.633578 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" podUID="7b3c4835-32b2-46a3-8d2d-f64071d509fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 17 23:18:56.546824 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:56.546792 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-pxpjd" Sep 17 23:18:56.633394 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:56.633359 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-b66bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 17 23:18:56.633589 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:56.633413 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" podUID="7b3c4835-32b2-46a3-8d2d-f64071d509fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 17 23:18:57.633942 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:57.633913 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-b66bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 17 23:18:57.634318 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:57.633973 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" podUID="7b3c4835-32b2-46a3-8d2d-f64071d509fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 17 23:18:58.637321 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:58.637290 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:18:58.637805 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:58.637688 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:18:58.638381 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:18:58.638363 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-b66bc" Sep 17 23:19:07.027091 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.027053 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc"] Sep 17 23:19:07.030591 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.030567 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.033111 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.033090 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 17 23:19:07.033225 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.033113 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 17 23:19:07.033293 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.033169 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jks5l\"" Sep 17 23:19:07.034189 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.034170 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 17 23:19:07.040923 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.040900 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc"] Sep 17 23:19:07.131162 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.131129 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3a82f97-c23a-4088-8baa-a95e6ef32ecb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkfxc\" (UID: \"c3a82f97-c23a-4088-8baa-a95e6ef32ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.131162 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.131163 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jcwv\" (UniqueName: \"kubernetes.io/projected/c3a82f97-c23a-4088-8baa-a95e6ef32ecb-kube-api-access-8jcwv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkfxc\" (UID: \"c3a82f97-c23a-4088-8baa-a95e6ef32ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.131351 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.131206 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3a82f97-c23a-4088-8baa-a95e6ef32ecb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkfxc\" (UID: \"c3a82f97-c23a-4088-8baa-a95e6ef32ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.131351 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.131221 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3a82f97-c23a-4088-8baa-a95e6ef32ecb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkfxc\" (UID: \"c3a82f97-c23a-4088-8baa-a95e6ef32ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.131351 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.131241 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3a82f97-c23a-4088-8baa-a95e6ef32ecb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkfxc\" (UID: \"c3a82f97-c23a-4088-8baa-a95e6ef32ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.131351 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.131261 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3a82f97-c23a-4088-8baa-a95e6ef32ecb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkfxc\" (UID: \"c3a82f97-c23a-4088-8baa-a95e6ef32ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.232476 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.232443 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3a82f97-c23a-4088-8baa-a95e6ef32ecb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkfxc\" (UID: \"c3a82f97-c23a-4088-8baa-a95e6ef32ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.232476 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.232477 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3a82f97-c23a-4088-8baa-a95e6ef32ecb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkfxc\" (UID: \"c3a82f97-c23a-4088-8baa-a95e6ef32ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.232709 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.232532 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3a82f97-c23a-4088-8baa-a95e6ef32ecb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkfxc\" (UID: \"c3a82f97-c23a-4088-8baa-a95e6ef32ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.232709 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.232557 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3a82f97-c23a-4088-8baa-a95e6ef32ecb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkfxc\" (UID: \"c3a82f97-c23a-4088-8baa-a95e6ef32ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.232709 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.232605 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3a82f97-c23a-4088-8baa-a95e6ef32ecb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkfxc\" (UID: \"c3a82f97-c23a-4088-8baa-a95e6ef32ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.232709 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.232622 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jcwv\" (UniqueName: \"kubernetes.io/projected/c3a82f97-c23a-4088-8baa-a95e6ef32ecb-kube-api-access-8jcwv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkfxc\" (UID: \"c3a82f97-c23a-4088-8baa-a95e6ef32ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.232903 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.232860 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3a82f97-c23a-4088-8baa-a95e6ef32ecb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkfxc\" (UID: \"c3a82f97-c23a-4088-8baa-a95e6ef32ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.232958 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.232936 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3a82f97-c23a-4088-8baa-a95e6ef32ecb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkfxc\" (UID: \"c3a82f97-c23a-4088-8baa-a95e6ef32ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.233014 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.232984 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3a82f97-c23a-4088-8baa-a95e6ef32ecb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkfxc\" (UID: \"c3a82f97-c23a-4088-8baa-a95e6ef32ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.234893 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.234867 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3a82f97-c23a-4088-8baa-a95e6ef32ecb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkfxc\" (UID: \"c3a82f97-c23a-4088-8baa-a95e6ef32ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.235239 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.235220 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3a82f97-c23a-4088-8baa-a95e6ef32ecb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkfxc\" (UID: \"c3a82f97-c23a-4088-8baa-a95e6ef32ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.240165 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.240146 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jcwv\" (UniqueName: \"kubernetes.io/projected/c3a82f97-c23a-4088-8baa-a95e6ef32ecb-kube-api-access-8jcwv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkfxc\" (UID: \"c3a82f97-c23a-4088-8baa-a95e6ef32ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.342854 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.342784 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:07.620009 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.619985 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc"] Sep 17 23:19:07.622359 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:19:07.622333 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a82f97_c23a_4088_8baa_a95e6ef32ecb.slice/crio-bad7b36c9bf6302d51c3ed6a688dfa52c29001999ab4d44764e6ac414ed1407a WatchSource:0}: Error finding container bad7b36c9bf6302d51c3ed6a688dfa52c29001999ab4d44764e6ac414ed1407a: Status 404 returned error can't find the container with id bad7b36c9bf6302d51c3ed6a688dfa52c29001999ab4d44764e6ac414ed1407a Sep 17 23:19:07.666509 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:07.666466 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" event={"ID":"c3a82f97-c23a-4088-8baa-a95e6ef32ecb","Type":"ContainerStarted","Data":"bad7b36c9bf6302d51c3ed6a688dfa52c29001999ab4d44764e6ac414ed1407a"} Sep 17 23:19:09.686862 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:09.686831 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" event={"ID":"c3a82f97-c23a-4088-8baa-a95e6ef32ecb","Type":"ContainerStarted","Data":"695736e244bdff0bba711224d54d2e807d6b67421cad0264c98caed46d53a0be"} Sep 17 23:19:09.705153 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:09.705101 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" podStartSLOduration=1.6818435699999998 podStartE2EDuration="2.705086241s" totalImagesPullingTime="1.023242671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:09.703596693 +0000 UTC m=+788.637503841" watchObservedRunningTime="2026-09-17 23:19:09.705086241 +0000 UTC m=+788.638993375" Sep 17 23:19:17.343923 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:17.343803 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:17.343923 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:17.343902 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:17.356150 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:17.356121 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:19:17.736899 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:19:17.736871 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkfxc" Sep 17 23:21:14.897106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:14.897071 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76b64f6977-5hc4l"] Sep 17 23:21:14.900581 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:14.900563 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76b64f6977-5hc4l" Sep 17 23:21:14.908169 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:14.908146 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76b64f6977-5hc4l"] Sep 17 23:21:15.030868 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:15.030839 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc430d25-8f1f-4a68-a409-91c5768c9746-tls-cert\") pod \"authorino-76b64f6977-5hc4l\" (UID: \"cc430d25-8f1f-4a68-a409-91c5768c9746\") " pod="kuadrant-system/authorino-76b64f6977-5hc4l" Sep 17 23:21:15.031047 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:15.030896 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gkms\" (UniqueName: \"kubernetes.io/projected/cc430d25-8f1f-4a68-a409-91c5768c9746-kube-api-access-8gkms\") pod \"authorino-76b64f6977-5hc4l\" (UID: \"cc430d25-8f1f-4a68-a409-91c5768c9746\") " pod="kuadrant-system/authorino-76b64f6977-5hc4l" Sep 17 23:21:15.132327 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:15.132295 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gkms\" (UniqueName: \"kubernetes.io/projected/cc430d25-8f1f-4a68-a409-91c5768c9746-kube-api-access-8gkms\") pod \"authorino-76b64f6977-5hc4l\" (UID: \"cc430d25-8f1f-4a68-a409-91c5768c9746\") " pod="kuadrant-system/authorino-76b64f6977-5hc4l" Sep 17 23:21:15.132461 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:15.132385 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc430d25-8f1f-4a68-a409-91c5768c9746-tls-cert\") pod \"authorino-76b64f6977-5hc4l\" (UID: \"cc430d25-8f1f-4a68-a409-91c5768c9746\") " pod="kuadrant-system/authorino-76b64f6977-5hc4l" Sep 17 23:21:15.134873 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:15.134853 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc430d25-8f1f-4a68-a409-91c5768c9746-tls-cert\") pod \"authorino-76b64f6977-5hc4l\" (UID: \"cc430d25-8f1f-4a68-a409-91c5768c9746\") " pod="kuadrant-system/authorino-76b64f6977-5hc4l" Sep 17 23:21:15.140226 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:15.140201 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gkms\" (UniqueName: \"kubernetes.io/projected/cc430d25-8f1f-4a68-a409-91c5768c9746-kube-api-access-8gkms\") pod \"authorino-76b64f6977-5hc4l\" (UID: \"cc430d25-8f1f-4a68-a409-91c5768c9746\") " pod="kuadrant-system/authorino-76b64f6977-5hc4l" Sep 17 23:21:15.211036 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:15.210983 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76b64f6977-5hc4l" Sep 17 23:21:15.477649 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:15.477625 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76b64f6977-5hc4l"] Sep 17 23:21:15.479679 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:21:15.479650 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc430d25_8f1f_4a68_a409_91c5768c9746.slice/crio-4a3680d04ecad24d77d910dbfa827eda6d3cd00c7b7a2dc9b4a0d867c9b58669 WatchSource:0}: Error finding container 4a3680d04ecad24d77d910dbfa827eda6d3cd00c7b7a2dc9b4a0d867c9b58669: Status 404 returned error can't find the container with id 4a3680d04ecad24d77d910dbfa827eda6d3cd00c7b7a2dc9b4a0d867c9b58669 Sep 17 23:21:16.340964 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:16.340924 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76b64f6977-5hc4l" event={"ID":"cc430d25-8f1f-4a68-a409-91c5768c9746","Type":"ContainerStarted","Data":"f21a4b4ba327d5a82401da25937c5abfa9276db7a4ddbab4d6dd0d2c00602e36"} Sep 17 23:21:16.341278 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:16.340983 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76b64f6977-5hc4l" event={"ID":"cc430d25-8f1f-4a68-a409-91c5768c9746","Type":"ContainerStarted","Data":"4a3680d04ecad24d77d910dbfa827eda6d3cd00c7b7a2dc9b4a0d867c9b58669"} Sep 17 23:21:16.360172 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:16.360129 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-76b64f6977-5hc4l" podStartSLOduration=1.588578985 podStartE2EDuration="2.360114574s" totalImagesPullingTime="771.535589ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:16.358118131 +0000 UTC m=+915.292025266" watchObservedRunningTime="2026-09-17 23:21:16.360114574 +0000 UTC m=+915.294021707" Sep 17 23:21:16.393242 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:16.393173 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85654986b8-22pdn"] Sep 17 23:21:16.393394 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:16.393348 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-85654986b8-22pdn" podUID="635f5734-34fe-48ca-b175-86342fba02d3" containerName="authorino" containerID="cri-o://11b73048344a125bd8c3a831de71237d500937ec44642df10fa8994f54a8d983" gracePeriod=30 Sep 17 23:21:16.649293 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:16.649269 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85654986b8-22pdn" Sep 17 23:21:16.745952 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:16.745920 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpd9t\" (UniqueName: \"kubernetes.io/projected/635f5734-34fe-48ca-b175-86342fba02d3-kube-api-access-hpd9t\") pod \"635f5734-34fe-48ca-b175-86342fba02d3\" (UID: \"635f5734-34fe-48ca-b175-86342fba02d3\") " Sep 17 23:21:16.745952 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:16.745956 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/635f5734-34fe-48ca-b175-86342fba02d3-tls-cert\") pod \"635f5734-34fe-48ca-b175-86342fba02d3\" (UID: \"635f5734-34fe-48ca-b175-86342fba02d3\") " Sep 17 23:21:16.748543 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:16.748518 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635f5734-34fe-48ca-b175-86342fba02d3-kube-api-access-hpd9t" (OuterVolumeSpecName: "kube-api-access-hpd9t") pod "635f5734-34fe-48ca-b175-86342fba02d3" (UID: "635f5734-34fe-48ca-b175-86342fba02d3"). InnerVolumeSpecName "kube-api-access-hpd9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:21:16.757624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:16.757596 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/635f5734-34fe-48ca-b175-86342fba02d3-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "635f5734-34fe-48ca-b175-86342fba02d3" (UID: "635f5734-34fe-48ca-b175-86342fba02d3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:21:16.847076 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:16.846992 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpd9t\" (UniqueName: \"kubernetes.io/projected/635f5734-34fe-48ca-b175-86342fba02d3-kube-api-access-hpd9t\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:21:16.847076 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:16.847041 2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/635f5734-34fe-48ca-b175-86342fba02d3-tls-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:21:17.347796 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:17.347761 2582 generic.go:356] "Generic (PLEG): container finished" podID="635f5734-34fe-48ca-b175-86342fba02d3" containerID="11b73048344a125bd8c3a831de71237d500937ec44642df10fa8994f54a8d983" exitCode=0 Sep 17 23:21:17.348223 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:17.347805 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85654986b8-22pdn" Sep 17 23:21:17.348223 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:17.347843 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85654986b8-22pdn" event={"ID":"635f5734-34fe-48ca-b175-86342fba02d3","Type":"ContainerDied","Data":"11b73048344a125bd8c3a831de71237d500937ec44642df10fa8994f54a8d983"} Sep 17 23:21:17.348223 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:17.347887 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85654986b8-22pdn" event={"ID":"635f5734-34fe-48ca-b175-86342fba02d3","Type":"ContainerDied","Data":"4467c60dc0f241316aa52509f53d677e9dc1e76643cb2c6b999cebfd8a9593a3"} Sep 17 23:21:17.348223 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:17.347908 2582 scope.go:117] "RemoveContainer" containerID="11b73048344a125bd8c3a831de71237d500937ec44642df10fa8994f54a8d983" Sep 17 23:21:17.357109 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:17.357083 2582 scope.go:117] "RemoveContainer" containerID="11b73048344a125bd8c3a831de71237d500937ec44642df10fa8994f54a8d983" Sep 17 23:21:17.357509 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:21:17.357476 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b73048344a125bd8c3a831de71237d500937ec44642df10fa8994f54a8d983\": container with ID starting with 11b73048344a125bd8c3a831de71237d500937ec44642df10fa8994f54a8d983 not found: ID does not exist" containerID="11b73048344a125bd8c3a831de71237d500937ec44642df10fa8994f54a8d983" Sep 17 23:21:17.357598 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:17.357517 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b73048344a125bd8c3a831de71237d500937ec44642df10fa8994f54a8d983"} err="failed to get container status \"11b73048344a125bd8c3a831de71237d500937ec44642df10fa8994f54a8d983\": rpc error: code = NotFound desc = could not find container \"11b73048344a125bd8c3a831de71237d500937ec44642df10fa8994f54a8d983\": container with ID starting with 11b73048344a125bd8c3a831de71237d500937ec44642df10fa8994f54a8d983 not found: ID does not exist" Sep 17 23:21:17.363513 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:17.363476 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85654986b8-22pdn"] Sep 17 23:21:17.368239 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:17.368215 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-85654986b8-22pdn"] Sep 17 23:21:19.294589 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:19.294556 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="635f5734-34fe-48ca-b175-86342fba02d3" path="/var/lib/kubelet/pods/635f5734-34fe-48ca-b175-86342fba02d3/volumes" Sep 17 23:21:34.153364 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.153331 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx"] Sep 17 23:21:34.153985 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.153814 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635f5734-34fe-48ca-b175-86342fba02d3" containerName="authorino" Sep 17 23:21:34.153985 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.153828 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="635f5734-34fe-48ca-b175-86342fba02d3" containerName="authorino" Sep 17 23:21:34.153985 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.153917 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="635f5734-34fe-48ca-b175-86342fba02d3" containerName="authorino" Sep 17 23:21:34.158328 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.158308 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.161154 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.161125 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0c363f-openshift-default-dockercfg-rrq2p\"" Sep 17 23:21:34.168536 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.168141 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx"] Sep 17 23:21:34.193173 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.193144 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-podinfo\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.193319 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.193179 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6khl\" (UniqueName: \"kubernetes.io/projected/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-kube-api-access-b6khl\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.193319 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.193197 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-workload-certs\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.193319 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.193262 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istiod-ca-cert\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.193560 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.193341 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-credential-socket\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.193560 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.193391 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-envoy\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.193560 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.193420 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-workload-socket\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.193560 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.193449 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-data\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.193692 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.193559 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-token\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.223418 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.223382 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7"] Sep 17 23:21:34.227168 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.227147 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.229543 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.229524 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-983490-openshift-default-dockercfg-hzl85\"" Sep 17 23:21:34.240728 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.240706 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7"] Sep 17 23:21:34.294673 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.294637 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-data\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.294816 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.294679 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istiod-ca-cert\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.294816 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.294750 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-credential-socket\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.294816 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.294793 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-envoy\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.295177 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.295146 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-workload-socket\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.295268 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.295198 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-data\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.295268 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.295256 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-token\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.295384 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.295294 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-credential-socket\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.295384 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.295326 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713a3a60-185a-4ee3-8e53-803ee57e08a9-istiod-ca-cert\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.295384 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.295354 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-envoy\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.295547 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.295384 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-workload-certs\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.295547 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.295415 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-token\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.295547 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.295478 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-podinfo\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.295547 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.295529 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6khl\" (UniqueName: \"kubernetes.io/projected/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-kube-api-access-b6khl\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.295731 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.295677 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istiod-ca-cert\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.295790 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.295767 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-workload-certs\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.295990 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.295972 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-workload-socket\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.296103 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.296075 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-credential-socket\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.296103 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.296090 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jq6w\" (UniqueName: \"kubernetes.io/projected/713a3a60-185a-4ee3-8e53-803ee57e08a9-kube-api-access-2jq6w\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.296233 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.296169 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-podinfo\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.296682 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.296636 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-workload-socket\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.296881 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.296855 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-workload-certs\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.297132 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.297110 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-data\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.297640 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.297619 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-envoy\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.298684 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.298633 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-podinfo\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.309342 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.309320 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r"] Sep 17 23:21:34.313133 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.313112 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-token\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.313303 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.313153 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6khl\" (UniqueName: \"kubernetes.io/projected/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-kube-api-access-b6khl\") pod \"e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.314756 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.314737 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.317107 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.317076 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-861522-openshift-default-dockercfg-kwnnq\"" Sep 17 23:21:34.323731 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.323712 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r"] Sep 17 23:21:34.397025 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.396999 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9vhx\" (UniqueName: \"kubernetes.io/projected/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-kube-api-access-b9vhx\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.397162 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.397035 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-podinfo\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.397162 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.397072 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-podinfo\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.397162 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.397145 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-envoy\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.397280 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.397191 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713a3a60-185a-4ee3-8e53-803ee57e08a9-istiod-ca-cert\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.397280 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.397223 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-token\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.397280 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.397256 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-token\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.397439 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.397291 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jq6w\" (UniqueName: \"kubernetes.io/projected/713a3a60-185a-4ee3-8e53-803ee57e08a9-kube-api-access-2jq6w\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.397439 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.397333 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-data\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.397439 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.397368 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-data\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.397439 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.397392 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-workload-socket\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.397724 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.397441 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-envoy\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.397724 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.397487 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-credential-socket\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.397724 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.397529 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-workload-certs\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.397724 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.397560 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-workload-certs\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.397724 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.397599 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-credential-socket\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.397724 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.397639 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istiod-ca-cert\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.397724 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.397666 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-workload-socket\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.398088 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.397783 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-data\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.398088 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.398012 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-workload-certs\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.398088 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.398044 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-credential-socket\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.398241 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.398217 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713a3a60-185a-4ee3-8e53-803ee57e08a9-istiod-ca-cert\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.398292 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.398274 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-workload-socket\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.399791 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.399761 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-envoy\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.399988 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.399972 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-podinfo\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.409773 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.409728 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-token\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.418256 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.418010 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jq6w\" (UniqueName: \"kubernetes.io/projected/713a3a60-185a-4ee3-8e53-803ee57e08a9-kube-api-access-2jq6w\") pod \"e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.473160 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.473126 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:34.498464 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.498439 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-podinfo\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.498602 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.498484 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-envoy\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.498602 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.498560 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-token\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.498733 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.498704 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-data\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.498783 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.498761 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-workload-socket\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.498840 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.498810 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-workload-certs\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.498890 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.498854 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-credential-socket\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.498890 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.498883 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istiod-ca-cert\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.498997 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.498916 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9vhx\" (UniqueName: \"kubernetes.io/projected/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-kube-api-access-b9vhx\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.499366 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.499102 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-data\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.499366 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.499229 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-workload-socket\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.499366 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.499280 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-credential-socket\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.499366 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.499365 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-workload-certs\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.499795 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.499779 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istiod-ca-cert\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.500921 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.500905 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-podinfo\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.501260 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.501245 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-envoy\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.509051 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.509017 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-token\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.509214 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.509197 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9vhx\" (UniqueName: \"kubernetes.io/projected/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-kube-api-access-b9vhx\") pod \"e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.539678 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.539576 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:34.635422 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.635391 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:34.751626 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.751600 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx"] Sep 17 23:21:34.753398 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:21:34.753376 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eae828a_0d27_457f_b49c_2f4cb9c4b4bc.slice/crio-432cc15aaa5ae5a10ef34744f42b70c60594a641ee4d231ffdb48b25fb9dcdc8 WatchSource:0}: Error finding container 432cc15aaa5ae5a10ef34744f42b70c60594a641ee4d231ffdb48b25fb9dcdc8: Status 404 returned error can't find the container with id 432cc15aaa5ae5a10ef34744f42b70c60594a641ee4d231ffdb48b25fb9dcdc8 Sep 17 23:21:34.755603 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.755575 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:34.755675 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.755633 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:34.755675 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.755662 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:34.825644 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.825619 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7"] Sep 17 23:21:34.827758 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:21:34.827680 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713a3a60_185a_4ee3_8e53_803ee57e08a9.slice/crio-f639d214a06795324dc323215c046459e62698053c4db8b745005dc0f0dda902 WatchSource:0}: Error finding container f639d214a06795324dc323215c046459e62698053c4db8b745005dc0f0dda902: Status 404 returned error can't find the container with id f639d214a06795324dc323215c046459e62698053c4db8b745005dc0f0dda902 Sep 17 23:21:34.829991 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.829955 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:34.830092 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.830042 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:34.830092 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.830082 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:34.913877 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.913851 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r"] Sep 17 23:21:34.916000 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:21:34.915974 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c70e8e6_75d0_4559_8819_36d5dc9d4a4b.slice/crio-72a55d582a1b573ca94a1aa03c99fc7b5b613024eb92d8ac4dc09b8847ea713c WatchSource:0}: Error finding container 72a55d582a1b573ca94a1aa03c99fc7b5b613024eb92d8ac4dc09b8847ea713c: Status 404 returned error can't find the container with id 72a55d582a1b573ca94a1aa03c99fc7b5b613024eb92d8ac4dc09b8847ea713c Sep 17 23:21:34.918694 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.918658 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:34.918790 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.918741 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:34.918856 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:34.918807 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:35.438677 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:35.438641 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" event={"ID":"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc","Type":"ContainerStarted","Data":"3d22c2d2fc58f164326f53875036c8b989c435ee9e1cfd953adb30c252069441"} Sep 17 23:21:35.439134 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:35.438703 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" event={"ID":"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc","Type":"ContainerStarted","Data":"432cc15aaa5ae5a10ef34744f42b70c60594a641ee4d231ffdb48b25fb9dcdc8"} Sep 17 23:21:35.440434 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:35.440404 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" event={"ID":"713a3a60-185a-4ee3-8e53-803ee57e08a9","Type":"ContainerStarted","Data":"c0ad59aee86c9a801468b85acbc478c2bf69d57e457decbbba4aebf86be87cfa"} Sep 17 23:21:35.440608 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:35.440448 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" event={"ID":"713a3a60-185a-4ee3-8e53-803ee57e08a9","Type":"ContainerStarted","Data":"f639d214a06795324dc323215c046459e62698053c4db8b745005dc0f0dda902"} Sep 17 23:21:35.442116 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:35.442095 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" event={"ID":"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b","Type":"ContainerStarted","Data":"3d4e2b3f85d4f51db19daa635eed96255284a9a763459ce48066139def54650a"} Sep 17 23:21:35.442221 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:35.442136 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" event={"ID":"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b","Type":"ContainerStarted","Data":"72a55d582a1b573ca94a1aa03c99fc7b5b613024eb92d8ac4dc09b8847ea713c"} Sep 17 23:21:35.461905 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:35.461855 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" podStartSLOduration=1.461838179 podStartE2EDuration="1.461838179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:35.460736651 +0000 UTC m=+934.394643782" watchObservedRunningTime="2026-09-17 23:21:35.461838179 +0000 UTC m=+934.395745314" Sep 17 23:21:35.473554 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:35.473531 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:35.478747 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:35.478727 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:35.485479 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:35.485443 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" podStartSLOduration=1.485430621 podStartE2EDuration="1.485430621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:35.484480981 +0000 UTC m=+934.418388138" watchObservedRunningTime="2026-09-17 23:21:35.485430621 +0000 UTC m=+934.419337747" Sep 17 23:21:35.511834 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:35.511792 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" podStartSLOduration=1.511778417 podStartE2EDuration="1.511778417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:35.509859619 +0000 UTC m=+934.443766754" watchObservedRunningTime="2026-09-17 23:21:35.511778417 +0000 UTC m=+934.445685552" Sep 17 23:21:35.540346 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:35.540328 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:35.541481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:35.541461 2582 patch_prober.go:28] interesting pod/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 17 23:21:35.541566 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:35.541529 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" podUID="713a3a60-185a-4ee3-8e53-803ee57e08a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 17 23:21:35.636738 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:35.636702 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:35.646540 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:35.646509 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:36.447118 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:36.447089 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:36.447576 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:36.447145 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:36.448039 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:36.448016 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:21:36.448144 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:36.448077 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:21:36.544476 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:36.544447 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:37.452999 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:37.452958 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:37.454006 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:37.453982 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:21:40.392750 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:40.392719 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7861-c7c348"] Sep 17 23:21:40.397750 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:40.397730 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7861-c7c348" Sep 17 23:21:40.400771 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:40.400752 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m6p2p\"" Sep 17 23:21:40.416028 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:40.416004 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7861-c7c348"] Sep 17 23:21:40.460199 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:40.460166 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65xs4\" (UniqueName: \"kubernetes.io/projected/73791af2-91cd-4121-a999-ac62b3c0c000-kube-api-access-65xs4\") pod \"test-curl-7861-c7c348\" (UID: \"73791af2-91cd-4121-a999-ac62b3c0c000\") " pod="opendatahub/test-curl-7861-c7c348" Sep 17 23:21:40.561472 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:40.561438 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65xs4\" (UniqueName: \"kubernetes.io/projected/73791af2-91cd-4121-a999-ac62b3c0c000-kube-api-access-65xs4\") pod \"test-curl-7861-c7c348\" (UID: \"73791af2-91cd-4121-a999-ac62b3c0c000\") " pod="opendatahub/test-curl-7861-c7c348" Sep 17 23:21:40.569870 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:40.569846 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65xs4\" (UniqueName: \"kubernetes.io/projected/73791af2-91cd-4121-a999-ac62b3c0c000-kube-api-access-65xs4\") pod \"test-curl-7861-c7c348\" (UID: \"73791af2-91cd-4121-a999-ac62b3c0c000\") " pod="opendatahub/test-curl-7861-c7c348" Sep 17 23:21:40.706966 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:40.706880 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7861-c7c348" Sep 17 23:21:40.977387 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:40.977363 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7861-c7c348"] Sep 17 23:21:40.979362 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:21:40.979337 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73791af2_91cd_4121_a999_ac62b3c0c000.slice/crio-03f2220a5e35124e05f314ed18263cd9161e0cf1406193f40eb272fb519b37e5 WatchSource:0}: Error finding container 03f2220a5e35124e05f314ed18263cd9161e0cf1406193f40eb272fb519b37e5: Status 404 returned error can't find the container with id 03f2220a5e35124e05f314ed18263cd9161e0cf1406193f40eb272fb519b37e5 Sep 17 23:21:41.475891 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:41.475860 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7861-c7c348" event={"ID":"73791af2-91cd-4121-a999-ac62b3c0c000","Type":"ContainerStarted","Data":"03f2220a5e35124e05f314ed18263cd9161e0cf1406193f40eb272fb519b37e5"} Sep 17 23:21:41.707835 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:41.707786 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s"] Sep 17 23:21:41.719585 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:41.719556 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s" Sep 17 23:21:41.721988 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:41.721958 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s"] Sep 17 23:21:41.722119 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:41.721960 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0c363f\"" Sep 17 23:21:41.722119 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:41.722038 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0c363f-dockercfg-v78dp\"" Sep 17 23:21:41.773298 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:41.773266 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2pzz\" (UniqueName: \"kubernetes.io/projected/5866863a-2c2f-4436-92b7-17278e4f05fb-kube-api-access-s2pzz\") pod \"payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s\" (UID: \"5866863a-2c2f-4436-92b7-17278e4f05fb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s" Sep 17 23:21:41.773516 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:41.773476 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5866863a-2c2f-4436-92b7-17278e4f05fb-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s\" (UID: \"5866863a-2c2f-4436-92b7-17278e4f05fb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s" Sep 17 23:21:41.875088 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:41.875048 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2pzz\" (UniqueName: \"kubernetes.io/projected/5866863a-2c2f-4436-92b7-17278e4f05fb-kube-api-access-s2pzz\") pod \"payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s\" (UID: \"5866863a-2c2f-4436-92b7-17278e4f05fb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s" Sep 17 23:21:41.875244 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:41.875116 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5866863a-2c2f-4436-92b7-17278e4f05fb-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s\" (UID: \"5866863a-2c2f-4436-92b7-17278e4f05fb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s" Sep 17 23:21:41.876050 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:41.876026 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5866863a-2c2f-4436-92b7-17278e4f05fb-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s\" (UID: \"5866863a-2c2f-4436-92b7-17278e4f05fb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s" Sep 17 23:21:41.883138 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:41.883109 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2pzz\" (UniqueName: \"kubernetes.io/projected/5866863a-2c2f-4436-92b7-17278e4f05fb-kube-api-access-s2pzz\") pod \"payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s\" (UID: \"5866863a-2c2f-4436-92b7-17278e4f05fb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s" Sep 17 23:21:42.037976 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:42.037897 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s" Sep 17 23:21:42.311113 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:42.311085 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s"] Sep 17 23:21:42.313189 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:21:42.313161 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5866863a_2c2f_4436_92b7_17278e4f05fb.slice/crio-e99366963ff31bf78a8422e7dd4f26ac89e1a3932bcd6c1c49b6e1a557f88e68 WatchSource:0}: Error finding container e99366963ff31bf78a8422e7dd4f26ac89e1a3932bcd6c1c49b6e1a557f88e68: Status 404 returned error can't find the container with id e99366963ff31bf78a8422e7dd4f26ac89e1a3932bcd6c1c49b6e1a557f88e68 Sep 17 23:21:42.483802 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:42.483767 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s" event={"ID":"5866863a-2c2f-4436-92b7-17278e4f05fb","Type":"ContainerStarted","Data":"3bb2ae575cfb3578b647a17ede0aed4bb438a809bf80eb8c9e0540bb949f8d0f"} Sep 17 23:21:42.484200 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:42.483833 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s" Sep 17 23:21:42.484200 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:42.483857 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s" event={"ID":"5866863a-2c2f-4436-92b7-17278e4f05fb","Type":"ContainerStarted","Data":"e99366963ff31bf78a8422e7dd4f26ac89e1a3932bcd6c1c49b6e1a557f88e68"} Sep 17 23:21:42.500524 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:42.500442 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s" podStartSLOduration=1.5004224000000002 podStartE2EDuration="1.5004224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:42.499661627 +0000 UTC m=+941.433568762" watchObservedRunningTime="2026-09-17 23:21:42.5004224 +0000 UTC m=+941.434329535" Sep 17 23:21:46.516644 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:46.516606 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7861-c7c348" event={"ID":"73791af2-91cd-4121-a999-ac62b3c0c000","Type":"ContainerStarted","Data":"d9a9399b939f4715f5e10dd87df5869f18750e368047c0118b84905fef866f8c"} Sep 17 23:21:47.259041 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:47.258929 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7861-c7c348" podStartSLOduration=2.354824449 podStartE2EDuration="7.258880232s" totalImagesPullingTime="4.904055783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:46.532569977 +0000 UTC m=+945.466477115" watchObservedRunningTime="2026-09-17 23:21:47.258880232 +0000 UTC m=+946.192787366" Sep 17 23:21:47.260837 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:47.260805 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7861-c7c348"] Sep 17 23:21:47.264621 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:47.264595 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7861-c7c348"] Sep 17 23:21:47.525068 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:21:47.524974 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7861-c7c348" container="test-curl-7861-c7c348" Sep 17 23:21:48.463866 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.461568 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf"] Sep 17 23:21:48.463866 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.462312 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73791af2-91cd-4121-a999-ac62b3c0c000" containerName="test-curl-7861-c7c348" Sep 17 23:21:48.463866 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.462338 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="73791af2-91cd-4121-a999-ac62b3c0c000" containerName="test-curl-7861-c7c348" Sep 17 23:21:48.463866 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.462456 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="73791af2-91cd-4121-a999-ac62b3c0c000" containerName="test-curl-7861-c7c348" Sep 17 23:21:48.480555 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.479876 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf"] Sep 17 23:21:48.480555 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.480017 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.508525 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.507963 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-63b144-openshift-default-dockercfg-mfzfj\"" Sep 17 23:21:48.556778 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.556326 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-credential-socket\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.556778 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.556395 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-workload-socket\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.556778 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.556436 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-data\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.556778 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.556464 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n86d\" (UniqueName: \"kubernetes.io/projected/869ba654-13cf-4bfe-9c6f-68bcf59df02f-kube-api-access-7n86d\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.556778 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.556492 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-workload-certs\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.556778 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.556539 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-podinfo\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.556778 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.556582 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-token\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.556778 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.556619 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istiod-ca-cert\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.556778 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.556666 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-envoy\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.580710 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.571300 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7861-c7c348" podUID="73791af2-91cd-4121-a999-ac62b3c0c000" containerName="test-curl-7861-c7c348" containerID="cri-o://d9a9399b939f4715f5e10dd87df5869f18750e368047c0118b84905fef866f8c" gracePeriod=2 Sep 17 23:21:48.660652 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.659009 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-workload-certs\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.660652 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.659078 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-podinfo\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.660652 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.659126 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-token\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.660652 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.659169 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istiod-ca-cert\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.660652 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.659224 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-envoy\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.660652 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.659295 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-credential-socket\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.660652 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.659351 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-workload-socket\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.660652 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.659396 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-data\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.660652 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.659427 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n86d\" (UniqueName: \"kubernetes.io/projected/869ba654-13cf-4bfe-9c6f-68bcf59df02f-kube-api-access-7n86d\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.660652 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.660188 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-workload-certs\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.661442 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.661163 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-workload-socket\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.661519 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.661441 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-credential-socket\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.661742 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.661720 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-data\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.664530 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.662534 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istiod-ca-cert\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.689591 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.689007 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-token\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.689591 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.689487 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-podinfo\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.690652 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.690625 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-envoy\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.698588 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.697553 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n86d\" (UniqueName: \"kubernetes.io/projected/869ba654-13cf-4bfe-9c6f-68bcf59df02f-kube-api-access-7n86d\") pod \"e2e-autores-63b144-openshift-default-54764d8c46-l9cdf\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:48.884787 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:48.884745 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:49.325540 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:21:49.322602 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869ba654_13cf_4bfe_9c6f_68bcf59df02f.slice/crio-ea5b8a33c36d887a36911436748edc21893157a72578f0b2a5d493b04c16499a WatchSource:0}: Error finding container ea5b8a33c36d887a36911436748edc21893157a72578f0b2a5d493b04c16499a: Status 404 returned error can't find the container with id ea5b8a33c36d887a36911436748edc21893157a72578f0b2a5d493b04c16499a Sep 17 23:21:49.325540 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:49.323660 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf"] Sep 17 23:21:49.350688 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:49.349266 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:49.350688 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:49.349359 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:49.350688 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:49.349401 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:49.582220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:49.582118 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" event={"ID":"869ba654-13cf-4bfe-9c6f-68bcf59df02f","Type":"ContainerStarted","Data":"ed54d655403db9afad45db8d56508624a213fa7ad4a776a9dfaa11bc51b033a0"} Sep 17 23:21:49.582220 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:49.582200 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" event={"ID":"869ba654-13cf-4bfe-9c6f-68bcf59df02f","Type":"ContainerStarted","Data":"ea5b8a33c36d887a36911436748edc21893157a72578f0b2a5d493b04c16499a"} Sep 17 23:21:49.604766 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:49.604690 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" podStartSLOduration=1.6046675179999998 podStartE2EDuration="1.604667518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:49.603436072 +0000 UTC m=+948.537343207" watchObservedRunningTime="2026-09-17 23:21:49.604667518 +0000 UTC m=+948.538574657" Sep 17 23:21:49.885590 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:49.885483 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:50.911528 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:50.910816 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:51.071258 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:51.071233 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7861-c7c348" Sep 17 23:21:51.099226 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:51.099132 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65xs4\" (UniqueName: \"kubernetes.io/projected/73791af2-91cd-4121-a999-ac62b3c0c000-kube-api-access-65xs4\") pod \"73791af2-91cd-4121-a999-ac62b3c0c000\" (UID: \"73791af2-91cd-4121-a999-ac62b3c0c000\") " Sep 17 23:21:51.104235 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:51.104204 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73791af2-91cd-4121-a999-ac62b3c0c000-kube-api-access-65xs4" (OuterVolumeSpecName: "kube-api-access-65xs4") pod "73791af2-91cd-4121-a999-ac62b3c0c000" (UID: "73791af2-91cd-4121-a999-ac62b3c0c000"). InnerVolumeSpecName "kube-api-access-65xs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:21:51.201588 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:51.201547 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65xs4\" (UniqueName: \"kubernetes.io/projected/73791af2-91cd-4121-a999-ac62b3c0c000-kube-api-access-65xs4\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:21:51.299279 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:51.299243 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73791af2-91cd-4121-a999-ac62b3c0c000" path="/var/lib/kubelet/pods/73791af2-91cd-4121-a999-ac62b3c0c000/volumes" Sep 17 23:21:51.605472 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:51.603448 2582 generic.go:356] "Generic (PLEG): container finished" podID="73791af2-91cd-4121-a999-ac62b3c0c000" containerID="d9a9399b939f4715f5e10dd87df5869f18750e368047c0118b84905fef866f8c" exitCode=137 Sep 17 23:21:51.605472 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:51.604674 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7861-c7c348" Sep 17 23:21:51.605472 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:51.605069 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:51.605472 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:51.605113 2582 scope.go:117] "RemoveContainer" containerID="d9a9399b939f4715f5e10dd87df5869f18750e368047c0118b84905fef866f8c" Sep 17 23:21:51.613832 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:51.613611 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:21:51.622852 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:51.622758 2582 scope.go:117] "RemoveContainer" containerID="d9a9399b939f4715f5e10dd87df5869f18750e368047c0118b84905fef866f8c" Sep 17 23:21:51.623326 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:21:51.623257 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9a9399b939f4715f5e10dd87df5869f18750e368047c0118b84905fef866f8c\": container with ID starting with d9a9399b939f4715f5e10dd87df5869f18750e368047c0118b84905fef866f8c not found: ID does not exist" containerID="d9a9399b939f4715f5e10dd87df5869f18750e368047c0118b84905fef866f8c" Sep 17 23:21:51.623326 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:51.623291 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9a9399b939f4715f5e10dd87df5869f18750e368047c0118b84905fef866f8c"} err="failed to get container status \"d9a9399b939f4715f5e10dd87df5869f18750e368047c0118b84905fef866f8c\": rpc error: code = NotFound desc = could not find container \"d9a9399b939f4715f5e10dd87df5869f18750e368047c0118b84905fef866f8c\": container with ID starting with d9a9399b939f4715f5e10dd87df5869f18750e368047c0118b84905fef866f8c not found: ID does not exist" Sep 17 23:21:53.099992 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.097862 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx"] Sep 17 23:21:53.104132 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.102632 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.107731 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.107480 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-307a7b-openshift-default-dockercfg-vb6hd\"" Sep 17 23:21:53.116491 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.116450 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx"] Sep 17 23:21:53.146408 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.146364 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-podinfo\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.146584 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.146423 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-workload-socket\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.146584 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.146447 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-credential-socket\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.146584 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.146491 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-envoy\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.146584 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.146543 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxkbg\" (UniqueName: \"kubernetes.io/projected/4392fd08-44e2-4bb0-8833-eea74a376d48-kube-api-access-cxkbg\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.146584 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.146573 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-data\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.146870 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.146642 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4392fd08-44e2-4bb0-8833-eea74a376d48-istiod-ca-cert\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.146870 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.146674 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-workload-certs\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.146870 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.146758 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-token\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.247791 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.247753 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-token\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.248090 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.247833 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-podinfo\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.248090 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.247876 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-workload-socket\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.248090 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.247902 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-credential-socket\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.248090 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.247950 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-envoy\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.248090 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.247987 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxkbg\" (UniqueName: \"kubernetes.io/projected/4392fd08-44e2-4bb0-8833-eea74a376d48-kube-api-access-cxkbg\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.248090 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.248017 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-data\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.248440 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.248095 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4392fd08-44e2-4bb0-8833-eea74a376d48-istiod-ca-cert\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.248440 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.248128 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-workload-certs\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.250574 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.248643 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-workload-certs\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.250574 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.248784 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-workload-socket\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.250574 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.249863 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-data\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.250574 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.250002 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-credential-socket\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.250574 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.250323 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4392fd08-44e2-4bb0-8833-eea74a376d48-istiod-ca-cert\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.251897 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.251851 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-podinfo\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.252750 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.252708 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-envoy\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.257035 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.256992 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxkbg\" (UniqueName: \"kubernetes.io/projected/4392fd08-44e2-4bb0-8833-eea74a376d48-kube-api-access-cxkbg\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.257998 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.257959 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-token\") pod \"e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.420298 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.420191 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:53.494193 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.494159 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s" Sep 17 23:21:53.729289 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.728746 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx"] Sep 17 23:21:53.739524 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.739053 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:53.739524 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.739135 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:53.739524 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:53.739177 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:54.630175 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:54.629969 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" event={"ID":"4392fd08-44e2-4bb0-8833-eea74a376d48","Type":"ContainerStarted","Data":"3748961b33628336df8aa9e5bd3d7d5e20a3b625897b3a92db411fcf9ca881f0"} Sep 17 23:21:54.630175 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:54.630027 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" event={"ID":"4392fd08-44e2-4bb0-8833-eea74a376d48","Type":"ContainerStarted","Data":"c15f2ec92eb015e5b379d8d5b52230f639214a00784e9fa41e364799e226cc93"} Sep 17 23:21:54.651516 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:54.651458 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" podStartSLOduration=1.651437553 podStartE2EDuration="1.651437553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:54.649076722 +0000 UTC m=+953.582983868" watchObservedRunningTime="2026-09-17 23:21:54.651437553 +0000 UTC m=+953.585344687" Sep 17 23:21:55.420723 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:55.420687 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:55.426832 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:55.426808 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:55.638270 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:55.638225 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:55.639341 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:55.639313 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:21:57.393518 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:57.393467 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq"] Sep 17 23:21:57.669811 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:57.426753 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq"] Sep 17 23:21:57.669811 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:57.426873 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq" Sep 17 23:21:57.669811 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:57.429404 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-63b144-dockercfg-fttdq\"" Sep 17 23:21:57.669811 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:57.429722 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-63b144\"" Sep 17 23:21:57.669811 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:57.496543 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc97a7d-1203-43b1-840b-1a2fd6a24a01-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq\" (UID: \"4fc97a7d-1203-43b1-840b-1a2fd6a24a01\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq" Sep 17 23:21:57.669811 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:57.496620 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlqvd\" (UniqueName: \"kubernetes.io/projected/4fc97a7d-1203-43b1-840b-1a2fd6a24a01-kube-api-access-qlqvd\") pod \"payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq\" (UID: \"4fc97a7d-1203-43b1-840b-1a2fd6a24a01\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq" Sep 17 23:21:57.669811 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:57.597794 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlqvd\" (UniqueName: \"kubernetes.io/projected/4fc97a7d-1203-43b1-840b-1a2fd6a24a01-kube-api-access-qlqvd\") pod \"payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq\" (UID: \"4fc97a7d-1203-43b1-840b-1a2fd6a24a01\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq" Sep 17 23:21:57.669811 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:57.597897 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc97a7d-1203-43b1-840b-1a2fd6a24a01-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq\" (UID: \"4fc97a7d-1203-43b1-840b-1a2fd6a24a01\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq" Sep 17 23:21:57.669811 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:57.598514 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc97a7d-1203-43b1-840b-1a2fd6a24a01-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq\" (UID: \"4fc97a7d-1203-43b1-840b-1a2fd6a24a01\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq" Sep 17 23:21:57.669811 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:57.612781 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlqvd\" (UniqueName: \"kubernetes.io/projected/4fc97a7d-1203-43b1-840b-1a2fd6a24a01-kube-api-access-qlqvd\") pod \"payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq\" (UID: \"4fc97a7d-1203-43b1-840b-1a2fd6a24a01\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq" Sep 17 23:21:57.819402 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:57.739883 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq" Sep 17 23:21:58.034546 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.034515 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq"] Sep 17 23:21:58.035178 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:21:58.035148 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc97a7d_1203_43b1_840b_1a2fd6a24a01.slice/crio-b554a437adf2662984c1420f425d5dce04a7c47c6039729cc795fb7ecb2bc9b9 WatchSource:0}: Error finding container b554a437adf2662984c1420f425d5dce04a7c47c6039729cc795fb7ecb2bc9b9: Status 404 returned error can't find the container with id b554a437adf2662984c1420f425d5dce04a7c47c6039729cc795fb7ecb2bc9b9 Sep 17 23:21:58.659737 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.659705 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq" event={"ID":"4fc97a7d-1203-43b1-840b-1a2fd6a24a01","Type":"ContainerStarted","Data":"92642818bae26be2203aaf75a9f1f09e605365f2153a8aab826abde4542b08f0"} Sep 17 23:21:58.660143 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.659767 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq" event={"ID":"4fc97a7d-1203-43b1-840b-1a2fd6a24a01","Type":"ContainerStarted","Data":"b554a437adf2662984c1420f425d5dce04a7c47c6039729cc795fb7ecb2bc9b9"} Sep 17 23:21:58.660143 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.659806 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq" Sep 17 23:21:58.661314 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.661292 2582 generic.go:356] "Generic (PLEG): container finished" podID="2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" containerID="3d4e2b3f85d4f51db19daa635eed96255284a9a763459ce48066139def54650a" exitCode=0 Sep 17 23:21:58.661474 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.661362 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" event={"ID":"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b","Type":"ContainerDied","Data":"3d4e2b3f85d4f51db19daa635eed96255284a9a763459ce48066139def54650a"} Sep 17 23:21:58.661795 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.661775 2582 scope.go:117] "RemoveContainer" containerID="3d4e2b3f85d4f51db19daa635eed96255284a9a763459ce48066139def54650a" Sep 17 23:21:58.663097 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.663068 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:58.663191 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.663137 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:58.663191 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.663159 2582 generic.go:356] "Generic (PLEG): container finished" podID="8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" containerID="3d22c2d2fc58f164326f53875036c8b989c435ee9e1cfd953adb30c252069441" exitCode=0 Sep 17 23:21:58.663191 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.663170 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:58.663305 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.663237 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" event={"ID":"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc","Type":"ContainerDied","Data":"3d22c2d2fc58f164326f53875036c8b989c435ee9e1cfd953adb30c252069441"} Sep 17 23:21:58.663728 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.663605 2582 scope.go:117] "RemoveContainer" containerID="3d22c2d2fc58f164326f53875036c8b989c435ee9e1cfd953adb30c252069441" Sep 17 23:21:58.664717 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.664695 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:58.664811 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.664766 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:58.664874 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.664814 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:58.664983 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.664964 2582 generic.go:356] "Generic (PLEG): container finished" podID="713a3a60-185a-4ee3-8e53-803ee57e08a9" containerID="c0ad59aee86c9a801468b85acbc478c2bf69d57e457decbbba4aebf86be87cfa" exitCode=0 Sep 17 23:21:58.665057 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.665026 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" event={"ID":"713a3a60-185a-4ee3-8e53-803ee57e08a9","Type":"ContainerDied","Data":"c0ad59aee86c9a801468b85acbc478c2bf69d57e457decbbba4aebf86be87cfa"} Sep 17 23:21:58.665428 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.665411 2582 scope.go:117] "RemoveContainer" containerID="c0ad59aee86c9a801468b85acbc478c2bf69d57e457decbbba4aebf86be87cfa" Sep 17 23:21:58.666459 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.666433 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:58.666551 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.666520 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:58.666604 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.666565 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:21:58.678762 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:58.678716 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq" podStartSLOduration=1.678699078 podStartE2EDuration="1.678699078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:58.675864612 +0000 UTC m=+957.609771750" watchObservedRunningTime="2026-09-17 23:21:58.678699078 +0000 UTC m=+957.612606213" Sep 17 23:21:59.677104 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:59.677061 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" event={"ID":"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b","Type":"ContainerStarted","Data":"713f0ecede515e7327180da3b6a30c618815d72a0ea2eeff9d315e4165d00ddf"} Sep 17 23:21:59.680693 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:59.680659 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" event={"ID":"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc","Type":"ContainerStarted","Data":"7d55c243fd72a2c6c6fc444195cf96d813197406bbd5fec4b4a70a77cae06f86"} Sep 17 23:21:59.695827 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:21:59.695793 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" event={"ID":"713a3a60-185a-4ee3-8e53-803ee57e08a9","Type":"ContainerStarted","Data":"e49f1349273e4ea2c9cb72e7d0548df8dbb641755b09ed90f4ee451a34838c24"} Sep 17 23:22:00.478534 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:00.478207 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:22:00.478534 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:00.478362 2582 patch_prober.go:28] interesting pod/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Sep 17 23:22:00.478534 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:00.478412 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" podUID="8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Sep 17 23:22:00.541967 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:00.541935 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:22:00.542299 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:00.542275 2582 patch_prober.go:28] interesting pod/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 17 23:22:00.542468 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:00.542442 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" podUID="713a3a60-185a-4ee3-8e53-803ee57e08a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 17 23:22:00.647927 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:00.647896 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:22:00.648277 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:00.648191 2582 patch_prober.go:28] interesting pod/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 17 23:22:00.648428 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:00.648406 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" podUID="2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 17 23:22:01.480342 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:01.474110 2582 patch_prober.go:28] interesting pod/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Sep 17 23:22:01.480342 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:01.474256 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" podUID="8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Sep 17 23:22:01.540588 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:01.540543 2582 patch_prober.go:28] interesting pod/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 17 23:22:01.540766 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:01.540650 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" podUID="713a3a60-185a-4ee3-8e53-803ee57e08a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 17 23:22:01.636372 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:01.636326 2582 patch_prober.go:28] interesting pod/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 17 23:22:01.636619 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:01.636396 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" podUID="2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 17 23:22:02.474381 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:02.474343 2582 patch_prober.go:28] interesting pod/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Sep 17 23:22:02.474683 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:02.474651 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" podUID="8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Sep 17 23:22:02.540757 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:02.540720 2582 patch_prober.go:28] interesting pod/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 17 23:22:02.541386 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:02.541335 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" podUID="713a3a60-185a-4ee3-8e53-803ee57e08a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 17 23:22:02.636917 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:02.636884 2582 patch_prober.go:28] interesting pod/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 17 23:22:02.637215 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:02.637171 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" podUID="2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 17 23:22:03.473620 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:03.473585 2582 patch_prober.go:28] interesting pod/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Sep 17 23:22:03.473822 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:03.473646 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" podUID="8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Sep 17 23:22:03.540968 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:03.540866 2582 patch_prober.go:28] interesting pod/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 17 23:22:03.541583 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:03.541539 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" podUID="713a3a60-185a-4ee3-8e53-803ee57e08a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 17 23:22:03.636224 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:03.636181 2582 patch_prober.go:28] interesting pod/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 17 23:22:03.636427 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:03.636249 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" podUID="2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 17 23:22:04.474388 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:04.474338 2582 patch_prober.go:28] interesting pod/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Sep 17 23:22:04.474660 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:04.474411 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" podUID="8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Sep 17 23:22:04.541002 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:04.540956 2582 patch_prober.go:28] interesting pod/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 17 23:22:04.541489 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:04.541027 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" podUID="713a3a60-185a-4ee3-8e53-803ee57e08a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 17 23:22:04.636542 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:04.636474 2582 patch_prober.go:28] interesting pod/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 17 23:22:04.636720 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:04.636572 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" podUID="2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 17 23:22:05.478970 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:05.478940 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:22:05.479238 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:05.479216 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:22:05.479996 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:05.479974 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:22:05.544703 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:05.544674 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:22:05.545054 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:05.544951 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:22:05.545701 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:05.545679 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:22:05.640575 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:05.640549 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:22:05.640850 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:05.640833 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:22:05.641523 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:05.641479 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:22:09.698688 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:09.698644 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq" Sep 17 23:22:11.737458 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:11.737424 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021e6520-4g8jf"] Sep 17 23:22:11.744554 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:11.744531 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021e6520-4g8jf" Sep 17 23:22:11.746908 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:11.746884 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:22:11.748250 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:11.748226 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021e6520-4g8jf"] Sep 17 23:22:11.934211 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:11.934174 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf73fe7-6aa1-4948-b12c-00284b8f5852-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-021e6520-4g8jf\" (UID: \"5cf73fe7-6aa1-4948-b12c-00284b8f5852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021e6520-4g8jf" Sep 17 23:22:12.034836 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:12.034746 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf73fe7-6aa1-4948-b12c-00284b8f5852-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-021e6520-4g8jf\" (UID: \"5cf73fe7-6aa1-4948-b12c-00284b8f5852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021e6520-4g8jf" Sep 17 23:22:12.035462 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:12.035438 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf73fe7-6aa1-4948-b12c-00284b8f5852-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-021e6520-4g8jf\" (UID: \"5cf73fe7-6aa1-4948-b12c-00284b8f5852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021e6520-4g8jf" Sep 17 23:22:12.057424 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:12.057397 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021e6520-4g8jf" Sep 17 23:22:12.381900 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:12.381631 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021e6520-4g8jf"] Sep 17 23:22:12.852635 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:12.852603 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021e6520-4g8jf" event={"ID":"5cf73fe7-6aa1-4948-b12c-00284b8f5852","Type":"ContainerStarted","Data":"c2f67847b81b9a84f019e3fbe378151a5cb5a30d369af229f4ba237f4f238023"} Sep 17 23:22:15.879977 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:15.879583 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021e6520-4g8jf" event={"ID":"5cf73fe7-6aa1-4948-b12c-00284b8f5852","Type":"ContainerStarted","Data":"6fa12bb9cbbaf2ab616610c02bec0ef6922dec73a0a5d93da622e6584b9bb231"} Sep 17 23:22:16.894955 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:16.894813 2582 generic.go:356] "Generic (PLEG): container finished" podID="5cf73fe7-6aa1-4948-b12c-00284b8f5852" containerID="6fa12bb9cbbaf2ab616610c02bec0ef6922dec73a0a5d93da622e6584b9bb231" exitCode=0 Sep 17 23:22:16.894955 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:16.894903 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021e6520-4g8jf" event={"ID":"5cf73fe7-6aa1-4948-b12c-00284b8f5852","Type":"ContainerDied","Data":"6fa12bb9cbbaf2ab616610c02bec0ef6922dec73a0a5d93da622e6584b9bb231"} Sep 17 23:22:17.902784 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:17.902753 2582 generic.go:356] "Generic (PLEG): container finished" podID="869ba654-13cf-4bfe-9c6f-68bcf59df02f" containerID="ed54d655403db9afad45db8d56508624a213fa7ad4a776a9dfaa11bc51b033a0" exitCode=0 Sep 17 23:22:17.903173 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:17.902827 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" event={"ID":"869ba654-13cf-4bfe-9c6f-68bcf59df02f","Type":"ContainerDied","Data":"ed54d655403db9afad45db8d56508624a213fa7ad4a776a9dfaa11bc51b033a0"} Sep 17 23:22:17.903312 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:17.903292 2582 scope.go:117] "RemoveContainer" containerID="ed54d655403db9afad45db8d56508624a213fa7ad4a776a9dfaa11bc51b033a0" Sep 17 23:22:17.904541 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:17.904487 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:22:17.904636 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:17.904579 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:22:17.904636 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:17.904575 2582 generic.go:356] "Generic (PLEG): container finished" podID="4392fd08-44e2-4bb0-8833-eea74a376d48" containerID="3748961b33628336df8aa9e5bd3d7d5e20a3b625897b3a92db411fcf9ca881f0" exitCode=0 Sep 17 23:22:17.904636 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:17.904620 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:22:17.904800 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:17.904656 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" event={"ID":"4392fd08-44e2-4bb0-8833-eea74a376d48","Type":"ContainerDied","Data":"3748961b33628336df8aa9e5bd3d7d5e20a3b625897b3a92db411fcf9ca881f0"} Sep 17 23:22:17.905116 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:17.905034 2582 scope.go:117] "RemoveContainer" containerID="3748961b33628336df8aa9e5bd3d7d5e20a3b625897b3a92db411fcf9ca881f0" Sep 17 23:22:17.906274 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:17.906242 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:22:17.906379 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:17.906322 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:22:17.906379 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:17.906368 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:22:18.085665 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:18.085416 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021e6520-4g8jf" Sep 17 23:22:18.098627 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:18.098161 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf73fe7-6aa1-4948-b12c-00284b8f5852-maas-api-service-ca\") pod \"5cf73fe7-6aa1-4948-b12c-00284b8f5852\" (UID: \"5cf73fe7-6aa1-4948-b12c-00284b8f5852\") " Sep 17 23:22:18.098950 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:18.098914 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf73fe7-6aa1-4948-b12c-00284b8f5852-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5cf73fe7-6aa1-4948-b12c-00284b8f5852" (UID: "5cf73fe7-6aa1-4948-b12c-00284b8f5852"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:22:18.199559 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:18.199432 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cf73fe7-6aa1-4948-b12c-00284b8f5852-maas-api-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:22:18.919877 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:18.919038 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" event={"ID":"869ba654-13cf-4bfe-9c6f-68bcf59df02f","Type":"ContainerStarted","Data":"2fe465fa67abb9c0a958b6eb2762b03b276d0f93726708bf788c1798c58ddf15"} Sep 17 23:22:18.923670 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:18.923639 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" event={"ID":"4392fd08-44e2-4bb0-8833-eea74a376d48","Type":"ContainerStarted","Data":"42f05f83b9ca2ee610dccd5b904630d80b22470eca7b6a56c3bc35067035f6c6"} Sep 17 23:22:18.925817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:18.925787 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021e6520-4g8jf" event={"ID":"5cf73fe7-6aa1-4948-b12c-00284b8f5852","Type":"ContainerDied","Data":"c2f67847b81b9a84f019e3fbe378151a5cb5a30d369af229f4ba237f4f238023"} Sep 17 23:22:18.925949 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:18.925838 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2f67847b81b9a84f019e3fbe378151a5cb5a30d369af229f4ba237f4f238023" Sep 17 23:22:18.925949 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:18.925893 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021e6520-4g8jf" Sep 17 23:22:19.423567 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:19.423455 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:22:19.423776 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:19.423658 2582 patch_prober.go:28] interesting pod/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 17 23:22:19.423776 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:19.423706 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" podUID="4392fd08-44e2-4bb0-8833-eea74a376d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 17 23:22:19.886149 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:19.886116 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:22:19.887414 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:19.887383 2582 patch_prober.go:28] interesting pod/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 17 23:22:19.887558 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:19.887448 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" podUID="869ba654-13cf-4bfe-9c6f-68bcf59df02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 17 23:22:20.421288 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:20.421237 2582 patch_prober.go:28] interesting pod/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 17 23:22:20.421902 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:20.421869 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" podUID="4392fd08-44e2-4bb0-8833-eea74a376d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 17 23:22:20.886048 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:20.886010 2582 patch_prober.go:28] interesting pod/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 17 23:22:20.886349 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:20.886317 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" podUID="869ba654-13cf-4bfe-9c6f-68bcf59df02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 17 23:22:21.420967 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:21.420921 2582 patch_prober.go:28] interesting pod/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 17 23:22:21.421189 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:21.420991 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" podUID="4392fd08-44e2-4bb0-8833-eea74a376d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 17 23:22:21.605335 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:21.605298 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:22:21.886207 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:21.886170 2582 patch_prober.go:28] interesting pod/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 17 23:22:21.886515 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:21.886467 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" podUID="869ba654-13cf-4bfe-9c6f-68bcf59df02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 17 23:22:22.421466 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:22.421430 2582 patch_prober.go:28] interesting pod/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 17 23:22:22.421765 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:22.421735 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" podUID="4392fd08-44e2-4bb0-8833-eea74a376d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 17 23:22:22.886322 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:22.886282 2582 patch_prober.go:28] interesting pod/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 17 23:22:22.886915 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:22.886880 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" podUID="869ba654-13cf-4bfe-9c6f-68bcf59df02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 17 23:22:23.421292 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:23.421254 2582 patch_prober.go:28] interesting pod/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 17 23:22:23.421522 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:23.421330 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" podUID="4392fd08-44e2-4bb0-8833-eea74a376d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 17 23:22:23.885463 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:23.885423 2582 patch_prober.go:28] interesting pod/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 17 23:22:23.885672 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:23.885490 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" podUID="869ba654-13cf-4bfe-9c6f-68bcf59df02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 17 23:22:24.424811 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:24.424780 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:22:24.425266 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:24.425246 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:22:24.425804 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:24.425785 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:22:24.889581 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:24.889551 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:22:24.890453 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:24.890432 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:22:28.536404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:28.536370 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021e6520-4g8jf"] Sep 17 23:22:28.539456 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:28.539431 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-021e6520-4g8jf"] Sep 17 23:22:29.293569 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:29.293535 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cf73fe7-6aa1-4948-b12c-00284b8f5852" path="/var/lib/kubelet/pods/5cf73fe7-6aa1-4948-b12c-00284b8f5852/volumes" Sep 17 23:22:29.792163 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:29.792127 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5"] Sep 17 23:22:29.792576 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:29.792562 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf73fe7-6aa1-4948-b12c-00284b8f5852" containerName="revoke-keys" Sep 17 23:22:29.792576 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:29.792576 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf73fe7-6aa1-4948-b12c-00284b8f5852" containerName="revoke-keys" Sep 17 23:22:29.792672 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:29.792665 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf73fe7-6aa1-4948-b12c-00284b8f5852" containerName="revoke-keys" Sep 17 23:22:29.797900 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:29.797877 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:29.800809 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:29.800790 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-d683cd6a-gw-openshift-default-dockercfg-qfztm\"" Sep 17 23:22:29.829797 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:29.829774 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5"] Sep 17 23:22:29.936962 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:29.936921 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-istio-data\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:29.937163 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:29.937050 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-workload-certs\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:29.937163 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:29.937108 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt9p4\" (UniqueName: \"kubernetes.io/projected/1957d728-1267-4e86-a9c0-11eca18d34df-kube-api-access-wt9p4\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:29.937252 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:29.937187 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1957d728-1267-4e86-a9c0-11eca18d34df-istio-podinfo\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:29.937252 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:29.937239 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1957d728-1267-4e86-a9c0-11eca18d34df-istiod-ca-cert\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:29.937346 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:29.937324 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1957d728-1267-4e86-a9c0-11eca18d34df-istio-token\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:29.937418 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:29.937404 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-credential-socket\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:29.937474 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:29.937435 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-workload-socket\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:29.937549 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:29.937480 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-istio-envoy\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.038308 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.038274 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-workload-certs\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.038482 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.038319 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt9p4\" (UniqueName: \"kubernetes.io/projected/1957d728-1267-4e86-a9c0-11eca18d34df-kube-api-access-wt9p4\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.038482 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.038357 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1957d728-1267-4e86-a9c0-11eca18d34df-istio-podinfo\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.038482 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.038396 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1957d728-1267-4e86-a9c0-11eca18d34df-istiod-ca-cert\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.038482 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.038436 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1957d728-1267-4e86-a9c0-11eca18d34df-istio-token\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.038482 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.038472 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-credential-socket\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.038797 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.038527 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-workload-socket\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.038797 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.038575 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-istio-envoy\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.038797 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.038629 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-istio-data\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.038797 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.038770 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-workload-certs\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.039013 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.038844 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-credential-socket\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.039013 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.038919 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-workload-socket\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.039013 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.038999 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-istio-data\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.039236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.039214 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1957d728-1267-4e86-a9c0-11eca18d34df-istiod-ca-cert\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.040846 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.040816 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-istio-envoy\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.040974 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.040897 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1957d728-1267-4e86-a9c0-11eca18d34df-istio-podinfo\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.046035 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.045978 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1957d728-1267-4e86-a9c0-11eca18d34df-istio-token\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.046191 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.046175 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt9p4\" (UniqueName: \"kubernetes.io/projected/1957d728-1267-4e86-a9c0-11eca18d34df-kube-api-access-wt9p4\") pod \"e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.109016 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.108986 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:30.390657 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.390627 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5"] Sep 17 23:22:30.392443 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:22:30.392421 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1957d728_1267_4e86_a9c0_11eca18d34df.slice/crio-d37598a39744775678ec4a99c0494f5d0588c9fed7795d5580d76d7eed6f26e4 WatchSource:0}: Error finding container d37598a39744775678ec4a99c0494f5d0588c9fed7795d5580d76d7eed6f26e4: Status 404 returned error can't find the container with id d37598a39744775678ec4a99c0494f5d0588c9fed7795d5580d76d7eed6f26e4 Sep 17 23:22:30.395085 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.395053 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:22:30.395169 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.395117 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:22:30.395169 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:30.395149 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:22:31.000386 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:31.000341 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" event={"ID":"1957d728-1267-4e86-a9c0-11eca18d34df","Type":"ContainerStarted","Data":"6b2210bf1545ae3a2dc43d4d230dfcf29fecfd51e9f6efa3c253c23ae076fd4c"} Sep 17 23:22:31.000806 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:31.000407 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" event={"ID":"1957d728-1267-4e86-a9c0-11eca18d34df","Type":"ContainerStarted","Data":"d37598a39744775678ec4a99c0494f5d0588c9fed7795d5580d76d7eed6f26e4"} Sep 17 23:22:31.024776 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:31.024724 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" podStartSLOduration=2.024709953 podStartE2EDuration="2.024709953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:22:31.021622585 +0000 UTC m=+989.955529732" watchObservedRunningTime="2026-09-17 23:22:31.024709953 +0000 UTC m=+989.958617088" Sep 17 23:22:31.109656 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:31.109626 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:31.114694 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:31.114672 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:32.006458 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:32.006423 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:32.007251 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:32.007232 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:52.137092 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:52.137057 2582 generic.go:356] "Generic (PLEG): container finished" podID="1957d728-1267-4e86-a9c0-11eca18d34df" containerID="6b2210bf1545ae3a2dc43d4d230dfcf29fecfd51e9f6efa3c253c23ae076fd4c" exitCode=0 Sep 17 23:22:52.137572 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:52.137129 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" event={"ID":"1957d728-1267-4e86-a9c0-11eca18d34df","Type":"ContainerDied","Data":"6b2210bf1545ae3a2dc43d4d230dfcf29fecfd51e9f6efa3c253c23ae076fd4c"} Sep 17 23:22:52.137572 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:52.137560 2582 scope.go:117] "RemoveContainer" containerID="6b2210bf1545ae3a2dc43d4d230dfcf29fecfd51e9f6efa3c253c23ae076fd4c" Sep 17 23:22:52.138763 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:52.138733 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:22:52.138858 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:52.138798 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:22:52.138858 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:52.138831 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:22:53.145571 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:53.145534 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" event={"ID":"1957d728-1267-4e86-a9c0-11eca18d34df","Type":"ContainerStarted","Data":"88f2470ce99b1cd2e24776f1aa583d2e96bd7811e0cdb162e1bfaf73da07e7e1"} Sep 17 23:22:54.109375 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:54.109339 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:54.109590 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:54.109563 2582 patch_prober.go:28] interesting pod/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 17 23:22:54.109653 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:54.109626 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" podUID="1957d728-1267-4e86-a9c0-11eca18d34df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 17 23:22:55.110295 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:55.110257 2582 patch_prober.go:28] interesting pod/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 17 23:22:55.110698 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:55.110313 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" podUID="1957d728-1267-4e86-a9c0-11eca18d34df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 17 23:22:56.109832 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:56.109792 2582 patch_prober.go:28] interesting pod/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 17 23:22:56.110011 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:56.109853 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" podUID="1957d728-1267-4e86-a9c0-11eca18d34df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 17 23:22:57.110071 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:57.110042 2582 patch_prober.go:28] interesting pod/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 17 23:22:57.110546 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:57.110489 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" podUID="1957d728-1267-4e86-a9c0-11eca18d34df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 17 23:22:57.734928 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:57.734901 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d683cd6a-2cvnm"] Sep 17 23:22:57.739303 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:57.739287 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d683cd6a-2cvnm" Sep 17 23:22:57.741817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:57.741792 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:22:57.745973 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:57.745950 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d683cd6a-2cvnm"] Sep 17 23:22:57.848125 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:57.848086 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/705ad394-a71f-44ec-a6d6-13faf43cd3a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d683cd6a-2cvnm\" (UID: \"705ad394-a71f-44ec-a6d6-13faf43cd3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d683cd6a-2cvnm" Sep 17 23:22:57.948939 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:57.948901 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/705ad394-a71f-44ec-a6d6-13faf43cd3a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d683cd6a-2cvnm\" (UID: \"705ad394-a71f-44ec-a6d6-13faf43cd3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d683cd6a-2cvnm" Sep 17 23:22:57.949510 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:57.949477 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/705ad394-a71f-44ec-a6d6-13faf43cd3a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d683cd6a-2cvnm\" (UID: \"705ad394-a71f-44ec-a6d6-13faf43cd3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d683cd6a-2cvnm" Sep 17 23:22:58.052670 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:58.052585 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d683cd6a-2cvnm" Sep 17 23:22:58.116959 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:58.116935 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:58.117902 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:58.117884 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:58.118570 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:58.118553 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:22:58.329652 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:58.329619 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d683cd6a-2cvnm"] Sep 17 23:22:58.338096 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:22:58.338067 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705ad394_a71f_44ec_a6d6_13faf43cd3a4.slice/crio-65ba3ea03e68da03c86a612289a54e223c10cd49a02fbe25686eb2ba6563604a WatchSource:0}: Error finding container 65ba3ea03e68da03c86a612289a54e223c10cd49a02fbe25686eb2ba6563604a: Status 404 returned error can't find the container with id 65ba3ea03e68da03c86a612289a54e223c10cd49a02fbe25686eb2ba6563604a Sep 17 23:22:59.192326 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:59.192290 2582 generic.go:356] "Generic (PLEG): container finished" podID="705ad394-a71f-44ec-a6d6-13faf43cd3a4" containerID="d947b532307bd80c293b763db48326042fa286b5018a764d12bf8ccf52c29409" exitCode=0 Sep 17 23:22:59.192728 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:59.192376 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d683cd6a-2cvnm" event={"ID":"705ad394-a71f-44ec-a6d6-13faf43cd3a4","Type":"ContainerDied","Data":"d947b532307bd80c293b763db48326042fa286b5018a764d12bf8ccf52c29409"} Sep 17 23:22:59.192728 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:22:59.192425 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d683cd6a-2cvnm" event={"ID":"705ad394-a71f-44ec-a6d6-13faf43cd3a4","Type":"ContainerStarted","Data":"65ba3ea03e68da03c86a612289a54e223c10cd49a02fbe25686eb2ba6563604a"} Sep 17 23:23:00.344877 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:00.344855 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d683cd6a-2cvnm" Sep 17 23:23:00.477454 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:00.477422 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/705ad394-a71f-44ec-a6d6-13faf43cd3a4-maas-api-service-ca\") pod \"705ad394-a71f-44ec-a6d6-13faf43cd3a4\" (UID: \"705ad394-a71f-44ec-a6d6-13faf43cd3a4\") " Sep 17 23:23:00.477786 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:00.477767 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705ad394-a71f-44ec-a6d6-13faf43cd3a4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "705ad394-a71f-44ec-a6d6-13faf43cd3a4" (UID: "705ad394-a71f-44ec-a6d6-13faf43cd3a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:23:00.578553 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:00.578525 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/705ad394-a71f-44ec-a6d6-13faf43cd3a4-maas-api-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:23:01.205230 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:01.205204 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d683cd6a-2cvnm" Sep 17 23:23:01.205390 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:01.205241 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d683cd6a-2cvnm" event={"ID":"705ad394-a71f-44ec-a6d6-13faf43cd3a4","Type":"ContainerDied","Data":"65ba3ea03e68da03c86a612289a54e223c10cd49a02fbe25686eb2ba6563604a"} Sep 17 23:23:01.205390 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:01.205288 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65ba3ea03e68da03c86a612289a54e223c10cd49a02fbe25686eb2ba6563604a" Sep 17 23:23:04.109630 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:04.109596 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d683cd6a-2cvnm"] Sep 17 23:23:04.112831 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:04.112807 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d683cd6a-2cvnm"] Sep 17 23:23:04.247906 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:04.247865 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5"] Sep 17 23:23:04.248281 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:04.248225 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" podUID="1957d728-1267-4e86-a9c0-11eca18d34df" containerName="istio-proxy" containerID="cri-o://88f2470ce99b1cd2e24776f1aa583d2e96bd7811e0cdb162e1bfaf73da07e7e1" gracePeriod=30 Sep 17 23:23:05.294173 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:05.294141 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="705ad394-a71f-44ec-a6d6-13faf43cd3a4" path="/var/lib/kubelet/pods/705ad394-a71f-44ec-a6d6-13faf43cd3a4/volumes" Sep 17 23:23:09.515806 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.515744 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:23:09.667493 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.667462 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-credential-socket\") pod \"1957d728-1267-4e86-a9c0-11eca18d34df\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " Sep 17 23:23:09.667698 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.667545 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1957d728-1267-4e86-a9c0-11eca18d34df-istio-token\") pod \"1957d728-1267-4e86-a9c0-11eca18d34df\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " Sep 17 23:23:09.667698 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.667583 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-workload-certs\") pod \"1957d728-1267-4e86-a9c0-11eca18d34df\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " Sep 17 23:23:09.667698 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.667617 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1957d728-1267-4e86-a9c0-11eca18d34df-istio-podinfo\") pod \"1957d728-1267-4e86-a9c0-11eca18d34df\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " Sep 17 23:23:09.667868 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.667719 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1957d728-1267-4e86-a9c0-11eca18d34df" (UID: "1957d728-1267-4e86-a9c0-11eca18d34df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:23:09.667868 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.667727 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-workload-socket\") pod \"1957d728-1267-4e86-a9c0-11eca18d34df\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " Sep 17 23:23:09.667868 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.667788 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-istio-envoy\") pod \"1957d728-1267-4e86-a9c0-11eca18d34df\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " Sep 17 23:23:09.667868 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.667852 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1957d728-1267-4e86-a9c0-11eca18d34df-istiod-ca-cert\") pod \"1957d728-1267-4e86-a9c0-11eca18d34df\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " Sep 17 23:23:09.668079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.667878 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-istio-data\") pod \"1957d728-1267-4e86-a9c0-11eca18d34df\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " Sep 17 23:23:09.668079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.667897 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1957d728-1267-4e86-a9c0-11eca18d34df" (UID: "1957d728-1267-4e86-a9c0-11eca18d34df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:23:09.668079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.667939 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1957d728-1267-4e86-a9c0-11eca18d34df" (UID: "1957d728-1267-4e86-a9c0-11eca18d34df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:23:09.668079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.667945 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt9p4\" (UniqueName: \"kubernetes.io/projected/1957d728-1267-4e86-a9c0-11eca18d34df-kube-api-access-wt9p4\") pod \"1957d728-1267-4e86-a9c0-11eca18d34df\" (UID: \"1957d728-1267-4e86-a9c0-11eca18d34df\") " Sep 17 23:23:09.668281 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.668187 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-istio-data" (OuterVolumeSpecName: "istio-data") pod "1957d728-1267-4e86-a9c0-11eca18d34df" (UID: "1957d728-1267-4e86-a9c0-11eca18d34df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:23:09.668281 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.668227 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1957d728-1267-4e86-a9c0-11eca18d34df-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1957d728-1267-4e86-a9c0-11eca18d34df" (UID: "1957d728-1267-4e86-a9c0-11eca18d34df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:23:09.668398 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.668382 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-credential-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:23:09.668456 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.668399 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-workload-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:23:09.668456 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.668409 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-workload-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:23:09.668456 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.668417 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1957d728-1267-4e86-a9c0-11eca18d34df-istiod-ca-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:23:09.668456 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.668425 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-istio-data\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:23:09.670571 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.670534 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1957d728-1267-4e86-a9c0-11eca18d34df" (UID: "1957d728-1267-4e86-a9c0-11eca18d34df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:23:09.670679 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.670659 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1957d728-1267-4e86-a9c0-11eca18d34df-kube-api-access-wt9p4" (OuterVolumeSpecName: "kube-api-access-wt9p4") pod "1957d728-1267-4e86-a9c0-11eca18d34df" (UID: "1957d728-1267-4e86-a9c0-11eca18d34df"). InnerVolumeSpecName "kube-api-access-wt9p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:23:09.671062 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.671042 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1957d728-1267-4e86-a9c0-11eca18d34df-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1957d728-1267-4e86-a9c0-11eca18d34df" (UID: "1957d728-1267-4e86-a9c0-11eca18d34df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:23:09.671708 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.671683 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1957d728-1267-4e86-a9c0-11eca18d34df-istio-token" (OuterVolumeSpecName: "istio-token") pod "1957d728-1267-4e86-a9c0-11eca18d34df" (UID: "1957d728-1267-4e86-a9c0-11eca18d34df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:23:09.769022 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.768946 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1957d728-1267-4e86-a9c0-11eca18d34df-istio-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:23:09.769022 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.768978 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1957d728-1267-4e86-a9c0-11eca18d34df-istio-podinfo\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:23:09.769022 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.768990 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1957d728-1267-4e86-a9c0-11eca18d34df-istio-envoy\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:23:09.769022 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:09.769001 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt9p4\" (UniqueName: \"kubernetes.io/projected/1957d728-1267-4e86-a9c0-11eca18d34df-kube-api-access-wt9p4\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:23:10.182887 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.182812 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd"] Sep 17 23:23:10.183281 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.183268 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="705ad394-a71f-44ec-a6d6-13faf43cd3a4" containerName="revoke-keys" Sep 17 23:23:10.183324 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.183283 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="705ad394-a71f-44ec-a6d6-13faf43cd3a4" containerName="revoke-keys" Sep 17 23:23:10.183324 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.183300 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1957d728-1267-4e86-a9c0-11eca18d34df" containerName="istio-proxy" Sep 17 23:23:10.183324 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.183305 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1957d728-1267-4e86-a9c0-11eca18d34df" containerName="istio-proxy" Sep 17 23:23:10.183324 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.183316 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1957d728-1267-4e86-a9c0-11eca18d34df" containerName="istio-proxy" Sep 17 23:23:10.183324 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.183322 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1957d728-1267-4e86-a9c0-11eca18d34df" containerName="istio-proxy" Sep 17 23:23:10.183476 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.183379 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="705ad394-a71f-44ec-a6d6-13faf43cd3a4" containerName="revoke-keys" Sep 17 23:23:10.183476 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.183386 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1957d728-1267-4e86-a9c0-11eca18d34df" containerName="istio-proxy" Sep 17 23:23:10.183476 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.183394 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1957d728-1267-4e86-a9c0-11eca18d34df" containerName="istio-proxy" Sep 17 23:23:10.186842 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.186827 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd" Sep 17 23:23:10.189583 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.189557 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-cc9b429c\"" Sep 17 23:23:10.189704 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.189679 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-cc9b429c-dockercfg-dgcr8\"" Sep 17 23:23:10.198694 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.198671 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd"] Sep 17 23:23:10.266281 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.266247 2582 generic.go:356] "Generic (PLEG): container finished" podID="1957d728-1267-4e86-a9c0-11eca18d34df" containerID="88f2470ce99b1cd2e24776f1aa583d2e96bd7811e0cdb162e1bfaf73da07e7e1" exitCode=0 Sep 17 23:23:10.266656 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.266637 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" Sep 17 23:23:10.266904 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.266536 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" event={"ID":"1957d728-1267-4e86-a9c0-11eca18d34df","Type":"ContainerDied","Data":"88f2470ce99b1cd2e24776f1aa583d2e96bd7811e0cdb162e1bfaf73da07e7e1"} Sep 17 23:23:10.266985 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.266916 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5" event={"ID":"1957d728-1267-4e86-a9c0-11eca18d34df","Type":"ContainerDied","Data":"d37598a39744775678ec4a99c0494f5d0588c9fed7795d5580d76d7eed6f26e4"} Sep 17 23:23:10.266985 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.266969 2582 scope.go:117] "RemoveContainer" containerID="88f2470ce99b1cd2e24776f1aa583d2e96bd7811e0cdb162e1bfaf73da07e7e1" Sep 17 23:23:10.276644 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.276619 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cdab163-3c8a-45b8-b7a3-70777a714705-plugins-config-volume\") pod \"payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd\" (UID: \"5cdab163-3c8a-45b8-b7a3-70777a714705\") " pod="openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd" Sep 17 23:23:10.276744 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.276657 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7kq8\" (UniqueName: \"kubernetes.io/projected/5cdab163-3c8a-45b8-b7a3-70777a714705-kube-api-access-l7kq8\") pod \"payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd\" (UID: \"5cdab163-3c8a-45b8-b7a3-70777a714705\") " pod="openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd" Sep 17 23:23:10.277994 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.277973 2582 scope.go:117] "RemoveContainer" containerID="6b2210bf1545ae3a2dc43d4d230dfcf29fecfd51e9f6efa3c253c23ae076fd4c" Sep 17 23:23:10.286920 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.286900 2582 scope.go:117] "RemoveContainer" containerID="88f2470ce99b1cd2e24776f1aa583d2e96bd7811e0cdb162e1bfaf73da07e7e1" Sep 17 23:23:10.287188 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:23:10.287168 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f2470ce99b1cd2e24776f1aa583d2e96bd7811e0cdb162e1bfaf73da07e7e1\": container with ID starting with 88f2470ce99b1cd2e24776f1aa583d2e96bd7811e0cdb162e1bfaf73da07e7e1 not found: ID does not exist" containerID="88f2470ce99b1cd2e24776f1aa583d2e96bd7811e0cdb162e1bfaf73da07e7e1" Sep 17 23:23:10.287271 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.287199 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f2470ce99b1cd2e24776f1aa583d2e96bd7811e0cdb162e1bfaf73da07e7e1"} err="failed to get container status \"88f2470ce99b1cd2e24776f1aa583d2e96bd7811e0cdb162e1bfaf73da07e7e1\": rpc error: code = NotFound desc = could not find container \"88f2470ce99b1cd2e24776f1aa583d2e96bd7811e0cdb162e1bfaf73da07e7e1\": container with ID starting with 88f2470ce99b1cd2e24776f1aa583d2e96bd7811e0cdb162e1bfaf73da07e7e1 not found: ID does not exist" Sep 17 23:23:10.287271 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.287222 2582 scope.go:117] "RemoveContainer" containerID="6b2210bf1545ae3a2dc43d4d230dfcf29fecfd51e9f6efa3c253c23ae076fd4c" Sep 17 23:23:10.287468 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:23:10.287452 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b2210bf1545ae3a2dc43d4d230dfcf29fecfd51e9f6efa3c253c23ae076fd4c\": container with ID starting with 6b2210bf1545ae3a2dc43d4d230dfcf29fecfd51e9f6efa3c253c23ae076fd4c not found: ID does not exist" containerID="6b2210bf1545ae3a2dc43d4d230dfcf29fecfd51e9f6efa3c253c23ae076fd4c" Sep 17 23:23:10.287548 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.287477 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b2210bf1545ae3a2dc43d4d230dfcf29fecfd51e9f6efa3c253c23ae076fd4c"} err="failed to get container status \"6b2210bf1545ae3a2dc43d4d230dfcf29fecfd51e9f6efa3c253c23ae076fd4c\": rpc error: code = NotFound desc = could not find container \"6b2210bf1545ae3a2dc43d4d230dfcf29fecfd51e9f6efa3c253c23ae076fd4c\": container with ID starting with 6b2210bf1545ae3a2dc43d4d230dfcf29fecfd51e9f6efa3c253c23ae076fd4c not found: ID does not exist" Sep 17 23:23:10.295083 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.295064 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5"] Sep 17 23:23:10.297120 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.297097 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-d683cd6a-gw-openshift-default-76c6f7687b-566s5"] Sep 17 23:23:10.377225 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.377197 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cdab163-3c8a-45b8-b7a3-70777a714705-plugins-config-volume\") pod \"payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd\" (UID: \"5cdab163-3c8a-45b8-b7a3-70777a714705\") " pod="openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd" Sep 17 23:23:10.377357 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.377230 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7kq8\" (UniqueName: \"kubernetes.io/projected/5cdab163-3c8a-45b8-b7a3-70777a714705-kube-api-access-l7kq8\") pod \"payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd\" (UID: \"5cdab163-3c8a-45b8-b7a3-70777a714705\") " pod="openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd" Sep 17 23:23:10.378055 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.378033 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cdab163-3c8a-45b8-b7a3-70777a714705-plugins-config-volume\") pod \"payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd\" (UID: \"5cdab163-3c8a-45b8-b7a3-70777a714705\") " pod="openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd" Sep 17 23:23:10.385186 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.385165 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7kq8\" (UniqueName: \"kubernetes.io/projected/5cdab163-3c8a-45b8-b7a3-70777a714705-kube-api-access-l7kq8\") pod \"payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd\" (UID: \"5cdab163-3c8a-45b8-b7a3-70777a714705\") " pod="openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd" Sep 17 23:23:10.499101 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.499072 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd" Sep 17 23:23:10.780388 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:10.780367 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd"] Sep 17 23:23:10.782793 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:23:10.782750 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cdab163_3c8a_45b8_b7a3_70777a714705.slice/crio-c3fe7ee7dc71a66f27d10c22c27a9e6d210d878f4ae1a30b0754966b1972405c WatchSource:0}: Error finding container c3fe7ee7dc71a66f27d10c22c27a9e6d210d878f4ae1a30b0754966b1972405c: Status 404 returned error can't find the container with id c3fe7ee7dc71a66f27d10c22c27a9e6d210d878f4ae1a30b0754966b1972405c Sep 17 23:23:11.275713 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:11.275676 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd" event={"ID":"5cdab163-3c8a-45b8-b7a3-70777a714705","Type":"ContainerStarted","Data":"03d1409a57795c00dabc6a12115eb7affaa3bf810b52ccba74203cf61e92110e"} Sep 17 23:23:11.275927 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:11.275740 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd" event={"ID":"5cdab163-3c8a-45b8-b7a3-70777a714705","Type":"ContainerStarted","Data":"c3fe7ee7dc71a66f27d10c22c27a9e6d210d878f4ae1a30b0754966b1972405c"} Sep 17 23:23:11.276002 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:11.275956 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd" Sep 17 23:23:11.296611 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:11.296200 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd" podStartSLOduration=1.296179054 podStartE2EDuration="1.296179054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:11.29137268 +0000 UTC m=+1030.225279817" watchObservedRunningTime="2026-09-17 23:23:11.296179054 +0000 UTC m=+1030.230086181" Sep 17 23:23:11.299318 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:11.299287 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1957d728-1267-4e86-a9c0-11eca18d34df" path="/var/lib/kubelet/pods/1957d728-1267-4e86-a9c0-11eca18d34df/volumes" Sep 17 23:23:22.283840 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:22.283808 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd" Sep 17 23:23:33.485020 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:33.484985 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc9b429c-twpb6"] Sep 17 23:23:33.489356 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:33.489331 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc9b429c-twpb6" Sep 17 23:23:33.491958 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:33.491935 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:23:33.497525 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:33.497487 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc9b429c-twpb6"] Sep 17 23:23:33.613769 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:33.613737 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5949408a-6d3b-4661-aea8-effbd0d7fd89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc9b429c-twpb6\" (UID: \"5949408a-6d3b-4661-aea8-effbd0d7fd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc9b429c-twpb6" Sep 17 23:23:33.714330 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:33.714296 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5949408a-6d3b-4661-aea8-effbd0d7fd89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc9b429c-twpb6\" (UID: \"5949408a-6d3b-4661-aea8-effbd0d7fd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc9b429c-twpb6" Sep 17 23:23:33.714986 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:33.714966 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5949408a-6d3b-4661-aea8-effbd0d7fd89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc9b429c-twpb6\" (UID: \"5949408a-6d3b-4661-aea8-effbd0d7fd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc9b429c-twpb6" Sep 17 23:23:33.802948 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:33.802876 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc9b429c-twpb6" Sep 17 23:23:34.077273 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:34.077222 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc9b429c-twpb6"] Sep 17 23:23:34.082567 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:23:34.082490 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5949408a_6d3b_4661_aea8_effbd0d7fd89.slice/crio-dad19d76f6ae6728bb318bc12180c1f1748ccd757bc9d770ec209b485ca9dbbf WatchSource:0}: Error finding container dad19d76f6ae6728bb318bc12180c1f1748ccd757bc9d770ec209b485ca9dbbf: Status 404 returned error can't find the container with id dad19d76f6ae6728bb318bc12180c1f1748ccd757bc9d770ec209b485ca9dbbf Sep 17 23:23:34.410331 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:34.410303 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc9b429c-twpb6" event={"ID":"5949408a-6d3b-4661-aea8-effbd0d7fd89","Type":"ContainerStarted","Data":"dad19d76f6ae6728bb318bc12180c1f1748ccd757bc9d770ec209b485ca9dbbf"} Sep 17 23:23:35.417837 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:35.417805 2582 generic.go:356] "Generic (PLEG): container finished" podID="5949408a-6d3b-4661-aea8-effbd0d7fd89" containerID="ef0ea5ee4e45f7cc22ae125f2fb22796b640014ea5f7aca69f5edfc84cb68f55" exitCode=0 Sep 17 23:23:35.418213 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:35.417872 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc9b429c-twpb6" event={"ID":"5949408a-6d3b-4661-aea8-effbd0d7fd89","Type":"ContainerDied","Data":"ef0ea5ee4e45f7cc22ae125f2fb22796b640014ea5f7aca69f5edfc84cb68f55"} Sep 17 23:23:36.562511 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:36.562476 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc9b429c-twpb6" Sep 17 23:23:36.643792 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:36.643763 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5949408a-6d3b-4661-aea8-effbd0d7fd89-maas-api-service-ca\") pod \"5949408a-6d3b-4661-aea8-effbd0d7fd89\" (UID: \"5949408a-6d3b-4661-aea8-effbd0d7fd89\") " Sep 17 23:23:36.644062 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:36.644042 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5949408a-6d3b-4661-aea8-effbd0d7fd89-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5949408a-6d3b-4661-aea8-effbd0d7fd89" (UID: "5949408a-6d3b-4661-aea8-effbd0d7fd89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:23:36.745200 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:36.745138 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5949408a-6d3b-4661-aea8-effbd0d7fd89-maas-api-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:23:37.430400 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:37.430362 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc9b429c-twpb6" event={"ID":"5949408a-6d3b-4661-aea8-effbd0d7fd89","Type":"ContainerDied","Data":"dad19d76f6ae6728bb318bc12180c1f1748ccd757bc9d770ec209b485ca9dbbf"} Sep 17 23:23:37.430597 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:37.430422 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dad19d76f6ae6728bb318bc12180c1f1748ccd757bc9d770ec209b485ca9dbbf" Sep 17 23:23:37.430597 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:37.430374 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc9b429c-twpb6" Sep 17 23:23:39.298365 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:39.298331 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd"] Sep 17 23:23:39.300249 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:39.298574 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd" podUID="5cdab163-3c8a-45b8-b7a3-70777a714705" containerName="payload-processing" containerID="cri-o://03d1409a57795c00dabc6a12115eb7affaa3bf810b52ccba74203cf61e92110e" gracePeriod=30 Sep 17 23:23:39.443227 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:39.443188 2582 generic.go:356] "Generic (PLEG): container finished" podID="5cdab163-3c8a-45b8-b7a3-70777a714705" containerID="03d1409a57795c00dabc6a12115eb7affaa3bf810b52ccba74203cf61e92110e" exitCode=0 Sep 17 23:23:39.443395 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:39.443238 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd" event={"ID":"5cdab163-3c8a-45b8-b7a3-70777a714705","Type":"ContainerDied","Data":"03d1409a57795c00dabc6a12115eb7affaa3bf810b52ccba74203cf61e92110e"} Sep 17 23:23:39.484710 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:39.484679 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc9b429c-twpb6"] Sep 17 23:23:39.490285 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:39.490233 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc9b429c-twpb6"] Sep 17 23:23:39.564582 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:39.564515 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd" Sep 17 23:23:39.679259 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:39.679219 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7kq8\" (UniqueName: \"kubernetes.io/projected/5cdab163-3c8a-45b8-b7a3-70777a714705-kube-api-access-l7kq8\") pod \"5cdab163-3c8a-45b8-b7a3-70777a714705\" (UID: \"5cdab163-3c8a-45b8-b7a3-70777a714705\") " Sep 17 23:23:39.679455 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:39.679267 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cdab163-3c8a-45b8-b7a3-70777a714705-plugins-config-volume\") pod \"5cdab163-3c8a-45b8-b7a3-70777a714705\" (UID: \"5cdab163-3c8a-45b8-b7a3-70777a714705\") " Sep 17 23:23:39.679671 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:39.679635 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cdab163-3c8a-45b8-b7a3-70777a714705-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5cdab163-3c8a-45b8-b7a3-70777a714705" (UID: "5cdab163-3c8a-45b8-b7a3-70777a714705"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:23:39.679805 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:39.679790 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cdab163-3c8a-45b8-b7a3-70777a714705-plugins-config-volume\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:23:39.681434 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:39.681412 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cdab163-3c8a-45b8-b7a3-70777a714705-kube-api-access-l7kq8" (OuterVolumeSpecName: "kube-api-access-l7kq8") pod "5cdab163-3c8a-45b8-b7a3-70777a714705" (UID: "5cdab163-3c8a-45b8-b7a3-70777a714705"). InnerVolumeSpecName "kube-api-access-l7kq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:23:39.780649 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:39.780613 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7kq8\" (UniqueName: \"kubernetes.io/projected/5cdab163-3c8a-45b8-b7a3-70777a714705-kube-api-access-l7kq8\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:23:40.450511 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:40.450478 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd" Sep 17 23:23:40.450893 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:40.450542 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd" event={"ID":"5cdab163-3c8a-45b8-b7a3-70777a714705","Type":"ContainerDied","Data":"c3fe7ee7dc71a66f27d10c22c27a9e6d210d878f4ae1a30b0754966b1972405c"} Sep 17 23:23:40.450893 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:40.450602 2582 scope.go:117] "RemoveContainer" containerID="03d1409a57795c00dabc6a12115eb7affaa3bf810b52ccba74203cf61e92110e" Sep 17 23:23:40.475930 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:40.475907 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd"] Sep 17 23:23:40.480922 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:40.480901 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc9b429c-d55f55d5c-pmjvd"] Sep 17 23:23:41.295395 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.295363 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5949408a-6d3b-4661-aea8-effbd0d7fd89" path="/var/lib/kubelet/pods/5949408a-6d3b-4661-aea8-effbd0d7fd89/volumes" Sep 17 23:23:41.295759 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.295744 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cdab163-3c8a-45b8-b7a3-70777a714705" path="/var/lib/kubelet/pods/5cdab163-3c8a-45b8-b7a3-70777a714705/volumes" Sep 17 23:23:41.427636 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.427600 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr"] Sep 17 23:23:41.428345 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.428327 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5949408a-6d3b-4661-aea8-effbd0d7fd89" containerName="revoke-keys" Sep 17 23:23:41.428404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.428350 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5949408a-6d3b-4661-aea8-effbd0d7fd89" containerName="revoke-keys" Sep 17 23:23:41.428404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.428386 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cdab163-3c8a-45b8-b7a3-70777a714705" containerName="payload-processing" Sep 17 23:23:41.428404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.428395 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cdab163-3c8a-45b8-b7a3-70777a714705" containerName="payload-processing" Sep 17 23:23:41.428512 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.428485 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5949408a-6d3b-4661-aea8-effbd0d7fd89" containerName="revoke-keys" Sep 17 23:23:41.428512 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.428493 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cdab163-3c8a-45b8-b7a3-70777a714705" containerName="payload-processing" Sep 17 23:23:41.433445 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.433425 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.435876 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.435853 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-4f19d32d-openshift-default-dockercfg-2m2fz\"" Sep 17 23:23:41.443622 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.443593 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr"] Sep 17 23:23:41.498477 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.498447 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-workload-certs\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.498918 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.498488 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-workload-socket\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.498918 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.498583 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pczd6\" (UniqueName: \"kubernetes.io/projected/a1305951-3e1b-4738-87dd-23c29d1ad327-kube-api-access-pczd6\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.498918 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.498656 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1305951-3e1b-4738-87dd-23c29d1ad327-istiod-ca-cert\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.498918 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.498741 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-envoy\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.498918 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.498778 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-token\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.498918 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.498805 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-credential-socket\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.498918 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.498868 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-podinfo\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.498918 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.498910 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-data\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.599925 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.599844 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-envoy\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.599925 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.599882 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-token\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.599925 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.599901 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-credential-socket\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.600198 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.599933 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-podinfo\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.600198 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.600057 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-data\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.600198 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.600144 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-workload-certs\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.600198 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.600172 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-workload-socket\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.600408 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.600205 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pczd6\" (UniqueName: \"kubernetes.io/projected/a1305951-3e1b-4738-87dd-23c29d1ad327-kube-api-access-pczd6\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.600408 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.600257 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1305951-3e1b-4738-87dd-23c29d1ad327-istiod-ca-cert\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.600408 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.600289 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-credential-socket\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.600408 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.600363 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-data\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.600891 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.600482 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-workload-certs\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.600891 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.600618 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-workload-socket\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.600891 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.600857 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1305951-3e1b-4738-87dd-23c29d1ad327-istiod-ca-cert\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.602204 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.602182 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-envoy\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.602469 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.602451 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-podinfo\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.608006 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.607985 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pczd6\" (UniqueName: \"kubernetes.io/projected/a1305951-3e1b-4738-87dd-23c29d1ad327-kube-api-access-pczd6\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.608006 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.608000 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-token\") pod \"e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:41.746175 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:41.745842 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:42.028734 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:42.028703 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr"] Sep 17 23:23:42.029797 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:23:42.029761 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1305951_3e1b_4738_87dd_23c29d1ad327.slice/crio-1ccfb59a578aeb5f5430d9336b892d391ffb3913861f6b92387bddd56e72e956 WatchSource:0}: Error finding container 1ccfb59a578aeb5f5430d9336b892d391ffb3913861f6b92387bddd56e72e956: Status 404 returned error can't find the container with id 1ccfb59a578aeb5f5430d9336b892d391ffb3913861f6b92387bddd56e72e956 Sep 17 23:23:42.032322 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:42.032284 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:23:42.032403 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:42.032368 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:23:42.032440 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:42.032411 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:23:42.466479 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:42.466394 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" event={"ID":"a1305951-3e1b-4738-87dd-23c29d1ad327","Type":"ContainerStarted","Data":"d2eb8420490f56641e553ab2eca8f79b32518c0f949f691955366c07bfa190f9"} Sep 17 23:23:42.466479 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:42.466445 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" event={"ID":"a1305951-3e1b-4738-87dd-23c29d1ad327","Type":"ContainerStarted","Data":"1ccfb59a578aeb5f5430d9336b892d391ffb3913861f6b92387bddd56e72e956"} Sep 17 23:23:42.494877 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:42.494831 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" podStartSLOduration=1.494817291 podStartE2EDuration="1.494817291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:42.492762269 +0000 UTC m=+1061.426669405" watchObservedRunningTime="2026-09-17 23:23:42.494817291 +0000 UTC m=+1061.428724425" Sep 17 23:23:42.746967 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:42.746939 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:43.751977 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:43.751950 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:44.480474 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:44.480440 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:23:44.481364 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:23:44.481341 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:24:04.601859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:04.601825 2582 generic.go:356] "Generic (PLEG): container finished" podID="a1305951-3e1b-4738-87dd-23c29d1ad327" containerID="d2eb8420490f56641e553ab2eca8f79b32518c0f949f691955366c07bfa190f9" exitCode=0 Sep 17 23:24:04.602296 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:04.601893 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" event={"ID":"a1305951-3e1b-4738-87dd-23c29d1ad327","Type":"ContainerDied","Data":"d2eb8420490f56641e553ab2eca8f79b32518c0f949f691955366c07bfa190f9"} Sep 17 23:24:04.602296 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:04.602283 2582 scope.go:117] "RemoveContainer" containerID="d2eb8420490f56641e553ab2eca8f79b32518c0f949f691955366c07bfa190f9" Sep 17 23:24:04.603371 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:04.603342 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:24:04.603427 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:04.603414 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:24:04.603465 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:04.603443 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:24:05.609773 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:05.609736 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" event={"ID":"a1305951-3e1b-4738-87dd-23c29d1ad327","Type":"ContainerStarted","Data":"db148a2ace36bf920d6b7aa995a3146a1278bd5c5b793deb8e2ab4a30fe5a921"} Sep 17 23:24:05.746521 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:05.746468 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:24:05.746781 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:05.746760 2582 patch_prober.go:28] interesting pod/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 17 23:24:05.746846 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:05.746808 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" podUID="a1305951-3e1b-4738-87dd-23c29d1ad327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 17 23:24:06.747218 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:06.747183 2582 patch_prober.go:28] interesting pod/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 17 23:24:06.747610 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:06.747238 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" podUID="a1305951-3e1b-4738-87dd-23c29d1ad327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 17 23:24:07.746189 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:07.746157 2582 patch_prober.go:28] interesting pod/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 17 23:24:07.746383 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:07.746219 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" podUID="a1305951-3e1b-4738-87dd-23c29d1ad327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 17 23:24:08.746350 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:08.746312 2582 patch_prober.go:28] interesting pod/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 17 23:24:08.746762 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:08.746382 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" podUID="a1305951-3e1b-4738-87dd-23c29d1ad327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 17 23:24:09.746248 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:09.746213 2582 patch_prober.go:28] interesting pod/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 17 23:24:09.746431 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:09.746280 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" podUID="a1305951-3e1b-4738-87dd-23c29d1ad327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 17 23:24:10.750270 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:10.750242 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:24:10.750687 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:10.750530 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:24:10.751052 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:10.751038 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:24:13.801156 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:13.801118 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4f19d32d-flhb9"] Sep 17 23:24:13.805086 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:13.805065 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4f19d32d-flhb9" Sep 17 23:24:13.807575 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:13.807555 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:24:13.812573 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:13.812357 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4f19d32d-flhb9"] Sep 17 23:24:13.943487 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:13.943450 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf1d788-d152-4365-8e83-1beb261038b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4f19d32d-flhb9\" (UID: \"fbf1d788-d152-4365-8e83-1beb261038b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4f19d32d-flhb9" Sep 17 23:24:14.044935 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:14.044900 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf1d788-d152-4365-8e83-1beb261038b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4f19d32d-flhb9\" (UID: \"fbf1d788-d152-4365-8e83-1beb261038b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4f19d32d-flhb9" Sep 17 23:24:14.045616 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:14.045591 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf1d788-d152-4365-8e83-1beb261038b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4f19d32d-flhb9\" (UID: \"fbf1d788-d152-4365-8e83-1beb261038b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4f19d32d-flhb9" Sep 17 23:24:14.117881 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:14.117818 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4f19d32d-flhb9" Sep 17 23:24:14.396033 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:14.396007 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4f19d32d-flhb9"] Sep 17 23:24:14.397328 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:24:14.397301 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf1d788_d152_4365_8e83_1beb261038b5.slice/crio-18dc71104e63e1c81c17f6f5d2e18b10f7c89805bf2a848d0f08970a235b80fa WatchSource:0}: Error finding container 18dc71104e63e1c81c17f6f5d2e18b10f7c89805bf2a848d0f08970a235b80fa: Status 404 returned error can't find the container with id 18dc71104e63e1c81c17f6f5d2e18b10f7c89805bf2a848d0f08970a235b80fa Sep 17 23:24:14.663080 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:14.663000 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4f19d32d-flhb9" event={"ID":"fbf1d788-d152-4365-8e83-1beb261038b5","Type":"ContainerStarted","Data":"18dc71104e63e1c81c17f6f5d2e18b10f7c89805bf2a848d0f08970a235b80fa"} Sep 17 23:24:15.670489 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:15.670403 2582 generic.go:356] "Generic (PLEG): container finished" podID="fbf1d788-d152-4365-8e83-1beb261038b5" containerID="9eb62ae29e9cc038b179a7b250e343dda82b32a6cbc4401aa93efe3c74af67e4" exitCode=0 Sep 17 23:24:15.670905 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:15.670484 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4f19d32d-flhb9" event={"ID":"fbf1d788-d152-4365-8e83-1beb261038b5","Type":"ContainerDied","Data":"9eb62ae29e9cc038b179a7b250e343dda82b32a6cbc4401aa93efe3c74af67e4"} Sep 17 23:24:16.821042 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:16.821017 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4f19d32d-flhb9" Sep 17 23:24:16.975293 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:16.975264 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf1d788-d152-4365-8e83-1beb261038b5-maas-api-service-ca\") pod \"fbf1d788-d152-4365-8e83-1beb261038b5\" (UID: \"fbf1d788-d152-4365-8e83-1beb261038b5\") " Sep 17 23:24:16.975635 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:16.975607 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf1d788-d152-4365-8e83-1beb261038b5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fbf1d788-d152-4365-8e83-1beb261038b5" (UID: "fbf1d788-d152-4365-8e83-1beb261038b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:24:17.076326 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:17.076290 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf1d788-d152-4365-8e83-1beb261038b5-maas-api-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:24:17.683646 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:17.683575 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4f19d32d-flhb9" Sep 17 23:24:17.683646 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:17.683614 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4f19d32d-flhb9" event={"ID":"fbf1d788-d152-4365-8e83-1beb261038b5","Type":"ContainerDied","Data":"18dc71104e63e1c81c17f6f5d2e18b10f7c89805bf2a848d0f08970a235b80fa"} Sep 17 23:24:17.683819 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:17.683660 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18dc71104e63e1c81c17f6f5d2e18b10f7c89805bf2a848d0f08970a235b80fa" Sep 17 23:24:26.132177 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:26.132140 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4f19d32d-flhb9"] Sep 17 23:24:26.136305 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:26.136275 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4f19d32d-flhb9"] Sep 17 23:24:26.267724 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:26.267686 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr"] Sep 17 23:24:26.267974 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:26.267925 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" podUID="a1305951-3e1b-4738-87dd-23c29d1ad327" containerName="istio-proxy" containerID="cri-o://db148a2ace36bf920d6b7aa995a3146a1278bd5c5b793deb8e2ab4a30fe5a921" gracePeriod=30 Sep 17 23:24:27.294809 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:27.294772 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbf1d788-d152-4365-8e83-1beb261038b5" path="/var/lib/kubelet/pods/fbf1d788-d152-4365-8e83-1beb261038b5/volumes" Sep 17 23:24:31.523278 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.523250 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:24:31.619671 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.619598 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1305951-3e1b-4738-87dd-23c29d1ad327-istiod-ca-cert\") pod \"a1305951-3e1b-4738-87dd-23c29d1ad327\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " Sep 17 23:24:31.619671 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.619635 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-data\") pod \"a1305951-3e1b-4738-87dd-23c29d1ad327\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " Sep 17 23:24:31.619671 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.619663 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-credential-socket\") pod \"a1305951-3e1b-4738-87dd-23c29d1ad327\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " Sep 17 23:24:31.619912 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.619782 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-envoy\") pod \"a1305951-3e1b-4738-87dd-23c29d1ad327\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " Sep 17 23:24:31.619912 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.619809 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-podinfo\") pod \"a1305951-3e1b-4738-87dd-23c29d1ad327\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " Sep 17 23:24:31.619912 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.619847 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-workload-certs\") pod \"a1305951-3e1b-4738-87dd-23c29d1ad327\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " Sep 17 23:24:31.620012 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.619939 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a1305951-3e1b-4738-87dd-23c29d1ad327" (UID: "a1305951-3e1b-4738-87dd-23c29d1ad327"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:24:31.620012 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.619990 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-data" (OuterVolumeSpecName: "istio-data") pod "a1305951-3e1b-4738-87dd-23c29d1ad327" (UID: "a1305951-3e1b-4738-87dd-23c29d1ad327"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:24:31.620012 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.620001 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-token\") pod \"a1305951-3e1b-4738-87dd-23c29d1ad327\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " Sep 17 23:24:31.620118 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.620033 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-workload-socket\") pod \"a1305951-3e1b-4738-87dd-23c29d1ad327\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " Sep 17 23:24:31.620118 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.620043 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1305951-3e1b-4738-87dd-23c29d1ad327-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a1305951-3e1b-4738-87dd-23c29d1ad327" (UID: "a1305951-3e1b-4738-87dd-23c29d1ad327"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:24:31.620118 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.620115 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pczd6\" (UniqueName: \"kubernetes.io/projected/a1305951-3e1b-4738-87dd-23c29d1ad327-kube-api-access-pczd6\") pod \"a1305951-3e1b-4738-87dd-23c29d1ad327\" (UID: \"a1305951-3e1b-4738-87dd-23c29d1ad327\") " Sep 17 23:24:31.620276 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.620129 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a1305951-3e1b-4738-87dd-23c29d1ad327" (UID: "a1305951-3e1b-4738-87dd-23c29d1ad327"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:24:31.620335 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.620292 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a1305951-3e1b-4738-87dd-23c29d1ad327" (UID: "a1305951-3e1b-4738-87dd-23c29d1ad327"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:24:31.620472 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.620456 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1305951-3e1b-4738-87dd-23c29d1ad327-istiod-ca-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:24:31.620574 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.620475 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-data\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:24:31.620574 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.620484 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-credential-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:24:31.620574 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.620495 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-workload-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:24:31.620574 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.620523 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-workload-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:24:31.622375 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.622346 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a1305951-3e1b-4738-87dd-23c29d1ad327" (UID: "a1305951-3e1b-4738-87dd-23c29d1ad327"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:24:31.622473 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.622380 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-token" (OuterVolumeSpecName: "istio-token") pod "a1305951-3e1b-4738-87dd-23c29d1ad327" (UID: "a1305951-3e1b-4738-87dd-23c29d1ad327"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:24:31.622473 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.622441 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a1305951-3e1b-4738-87dd-23c29d1ad327" (UID: "a1305951-3e1b-4738-87dd-23c29d1ad327"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:24:31.622473 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.622449 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1305951-3e1b-4738-87dd-23c29d1ad327-kube-api-access-pczd6" (OuterVolumeSpecName: "kube-api-access-pczd6") pod "a1305951-3e1b-4738-87dd-23c29d1ad327" (UID: "a1305951-3e1b-4738-87dd-23c29d1ad327"). InnerVolumeSpecName "kube-api-access-pczd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:24:31.721829 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.721799 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pczd6\" (UniqueName: \"kubernetes.io/projected/a1305951-3e1b-4738-87dd-23c29d1ad327-kube-api-access-pczd6\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:24:31.721829 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.721826 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-envoy\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:24:31.721829 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.721835 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-podinfo\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:24:31.722038 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.721844 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1305951-3e1b-4738-87dd-23c29d1ad327-istio-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:24:31.782128 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.782091 2582 generic.go:356] "Generic (PLEG): container finished" podID="a1305951-3e1b-4738-87dd-23c29d1ad327" containerID="db148a2ace36bf920d6b7aa995a3146a1278bd5c5b793deb8e2ab4a30fe5a921" exitCode=0 Sep 17 23:24:31.782270 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.782148 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" Sep 17 23:24:31.782270 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.782176 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" event={"ID":"a1305951-3e1b-4738-87dd-23c29d1ad327","Type":"ContainerDied","Data":"db148a2ace36bf920d6b7aa995a3146a1278bd5c5b793deb8e2ab4a30fe5a921"} Sep 17 23:24:31.782270 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.782234 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr" event={"ID":"a1305951-3e1b-4738-87dd-23c29d1ad327","Type":"ContainerDied","Data":"1ccfb59a578aeb5f5430d9336b892d391ffb3913861f6b92387bddd56e72e956"} Sep 17 23:24:31.782270 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.782256 2582 scope.go:117] "RemoveContainer" containerID="db148a2ace36bf920d6b7aa995a3146a1278bd5c5b793deb8e2ab4a30fe5a921" Sep 17 23:24:31.792749 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.792730 2582 scope.go:117] "RemoveContainer" containerID="d2eb8420490f56641e553ab2eca8f79b32518c0f949f691955366c07bfa190f9" Sep 17 23:24:31.800991 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.800974 2582 scope.go:117] "RemoveContainer" containerID="db148a2ace36bf920d6b7aa995a3146a1278bd5c5b793deb8e2ab4a30fe5a921" Sep 17 23:24:31.801256 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:24:31.801233 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db148a2ace36bf920d6b7aa995a3146a1278bd5c5b793deb8e2ab4a30fe5a921\": container with ID starting with db148a2ace36bf920d6b7aa995a3146a1278bd5c5b793deb8e2ab4a30fe5a921 not found: ID does not exist" containerID="db148a2ace36bf920d6b7aa995a3146a1278bd5c5b793deb8e2ab4a30fe5a921" Sep 17 23:24:31.801350 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.801260 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db148a2ace36bf920d6b7aa995a3146a1278bd5c5b793deb8e2ab4a30fe5a921"} err="failed to get container status \"db148a2ace36bf920d6b7aa995a3146a1278bd5c5b793deb8e2ab4a30fe5a921\": rpc error: code = NotFound desc = could not find container \"db148a2ace36bf920d6b7aa995a3146a1278bd5c5b793deb8e2ab4a30fe5a921\": container with ID starting with db148a2ace36bf920d6b7aa995a3146a1278bd5c5b793deb8e2ab4a30fe5a921 not found: ID does not exist" Sep 17 23:24:31.801350 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.801276 2582 scope.go:117] "RemoveContainer" containerID="d2eb8420490f56641e553ab2eca8f79b32518c0f949f691955366c07bfa190f9" Sep 17 23:24:31.801522 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:24:31.801482 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2eb8420490f56641e553ab2eca8f79b32518c0f949f691955366c07bfa190f9\": container with ID starting with d2eb8420490f56641e553ab2eca8f79b32518c0f949f691955366c07bfa190f9 not found: ID does not exist" containerID="d2eb8420490f56641e553ab2eca8f79b32518c0f949f691955366c07bfa190f9" Sep 17 23:24:31.801585 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.801527 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2eb8420490f56641e553ab2eca8f79b32518c0f949f691955366c07bfa190f9"} err="failed to get container status \"d2eb8420490f56641e553ab2eca8f79b32518c0f949f691955366c07bfa190f9\": rpc error: code = NotFound desc = could not find container \"d2eb8420490f56641e553ab2eca8f79b32518c0f949f691955366c07bfa190f9\": container with ID starting with d2eb8420490f56641e553ab2eca8f79b32518c0f949f691955366c07bfa190f9 not found: ID does not exist" Sep 17 23:24:31.806831 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.806809 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr"] Sep 17 23:24:31.810824 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:31.810803 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-4f19d32d-openshift-default-5876f56544-tk5qr"] Sep 17 23:24:33.295288 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:33.295253 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1305951-3e1b-4738-87dd-23c29d1ad327" path="/var/lib/kubelet/pods/a1305951-3e1b-4738-87dd-23c29d1ad327/volumes" Sep 17 23:24:51.298011 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:51.297975 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63b144-ztbrb"] Sep 17 23:24:51.298663 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:51.298625 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbf1d788-d152-4365-8e83-1beb261038b5" containerName="revoke-keys" Sep 17 23:24:51.298663 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:51.298650 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbf1d788-d152-4365-8e83-1beb261038b5" containerName="revoke-keys" Sep 17 23:24:51.298871 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:51.298674 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1305951-3e1b-4738-87dd-23c29d1ad327" containerName="istio-proxy" Sep 17 23:24:51.298871 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:51.298685 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1305951-3e1b-4738-87dd-23c29d1ad327" containerName="istio-proxy" Sep 17 23:24:51.298871 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:51.298804 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1305951-3e1b-4738-87dd-23c29d1ad327" containerName="istio-proxy" Sep 17 23:24:51.298871 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:51.298815 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1305951-3e1b-4738-87dd-23c29d1ad327" containerName="istio-proxy" Sep 17 23:24:51.298871 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:51.298826 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbf1d788-d152-4365-8e83-1beb261038b5" containerName="revoke-keys" Sep 17 23:24:51.301914 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:51.301895 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63b144-ztbrb" Sep 17 23:24:51.304105 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:51.304085 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:24:51.309295 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:51.309256 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63b144-ztbrb"] Sep 17 23:24:51.413427 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:51.413395 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7cf450a-9fab-4708-bf7c-378035a2e6e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-63b144-ztbrb\" (UID: \"a7cf450a-9fab-4708-bf7c-378035a2e6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63b144-ztbrb" Sep 17 23:24:51.514014 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:51.513984 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7cf450a-9fab-4708-bf7c-378035a2e6e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-63b144-ztbrb\" (UID: \"a7cf450a-9fab-4708-bf7c-378035a2e6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63b144-ztbrb" Sep 17 23:24:51.514732 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:51.514708 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7cf450a-9fab-4708-bf7c-378035a2e6e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-63b144-ztbrb\" (UID: \"a7cf450a-9fab-4708-bf7c-378035a2e6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63b144-ztbrb" Sep 17 23:24:51.614194 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:51.614118 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63b144-ztbrb" Sep 17 23:24:51.884455 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:51.884422 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63b144-ztbrb"] Sep 17 23:24:51.886156 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:24:51.886124 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7cf450a_9fab_4708_bf7c_378035a2e6e5.slice/crio-c30fedddbd0d61143107d82f43feca2724a9554704244a5cbac388f8df60bc08 WatchSource:0}: Error finding container c30fedddbd0d61143107d82f43feca2724a9554704244a5cbac388f8df60bc08: Status 404 returned error can't find the container with id c30fedddbd0d61143107d82f43feca2724a9554704244a5cbac388f8df60bc08 Sep 17 23:24:51.905064 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:51.904971 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63b144-ztbrb" event={"ID":"a7cf450a-9fab-4708-bf7c-378035a2e6e5","Type":"ContainerStarted","Data":"c30fedddbd0d61143107d82f43feca2724a9554704244a5cbac388f8df60bc08"} Sep 17 23:24:52.720829 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:24:52.720795 2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7cf450a_9fab_4708_bf7c_378035a2e6e5.slice/crio-conmon-3910967a5e5eb6dd09a69bd7a9afbaa9e32537360f794f8f769a2173b83d5993.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7cf450a_9fab_4708_bf7c_378035a2e6e5.slice/crio-3910967a5e5eb6dd09a69bd7a9afbaa9e32537360f794f8f769a2173b83d5993.scope\": RecentStats: unable to find data in memory cache]" Sep 17 23:24:52.913785 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:52.913748 2582 generic.go:356] "Generic (PLEG): container finished" podID="a7cf450a-9fab-4708-bf7c-378035a2e6e5" containerID="3910967a5e5eb6dd09a69bd7a9afbaa9e32537360f794f8f769a2173b83d5993" exitCode=0 Sep 17 23:24:52.913931 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:52.913804 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63b144-ztbrb" event={"ID":"a7cf450a-9fab-4708-bf7c-378035a2e6e5","Type":"ContainerDied","Data":"3910967a5e5eb6dd09a69bd7a9afbaa9e32537360f794f8f769a2173b83d5993"} Sep 17 23:24:54.055801 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:54.055773 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63b144-ztbrb" Sep 17 23:24:54.142061 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:54.142037 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7cf450a-9fab-4708-bf7c-378035a2e6e5-maas-api-service-ca\") pod \"a7cf450a-9fab-4708-bf7c-378035a2e6e5\" (UID: \"a7cf450a-9fab-4708-bf7c-378035a2e6e5\") " Sep 17 23:24:54.142369 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:54.142345 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7cf450a-9fab-4708-bf7c-378035a2e6e5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a7cf450a-9fab-4708-bf7c-378035a2e6e5" (UID: "a7cf450a-9fab-4708-bf7c-378035a2e6e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:24:54.243386 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:54.243358 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7cf450a-9fab-4708-bf7c-378035a2e6e5-maas-api-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:24:54.928042 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:54.928015 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63b144-ztbrb" Sep 17 23:24:54.928215 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:54.928047 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63b144-ztbrb" event={"ID":"a7cf450a-9fab-4708-bf7c-378035a2e6e5","Type":"ContainerDied","Data":"c30fedddbd0d61143107d82f43feca2724a9554704244a5cbac388f8df60bc08"} Sep 17 23:24:54.928215 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:54.928093 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c30fedddbd0d61143107d82f43feca2724a9554704244a5cbac388f8df60bc08" Sep 17 23:24:58.679286 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:58.679253 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-983490-87kfg"] Sep 17 23:24:58.679766 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:58.679751 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7cf450a-9fab-4708-bf7c-378035a2e6e5" containerName="revoke-keys" Sep 17 23:24:58.679766 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:58.679767 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7cf450a-9fab-4708-bf7c-378035a2e6e5" containerName="revoke-keys" Sep 17 23:24:58.679865 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:58.679794 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1305951-3e1b-4738-87dd-23c29d1ad327" containerName="istio-proxy" Sep 17 23:24:58.679865 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:58.679799 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1305951-3e1b-4738-87dd-23c29d1ad327" containerName="istio-proxy" Sep 17 23:24:58.679985 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:58.679876 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7cf450a-9fab-4708-bf7c-378035a2e6e5" containerName="revoke-keys" Sep 17 23:24:58.683106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:58.683087 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-983490-87kfg" Sep 17 23:24:58.685652 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:58.685630 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:24:58.690455 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:58.690430 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-983490-87kfg"] Sep 17 23:24:58.790919 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:58.790884 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96589a36-9b8d-4974-8128-95e22ef47036-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-983490-87kfg\" (UID: \"96589a36-9b8d-4974-8128-95e22ef47036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-983490-87kfg" Sep 17 23:24:58.892532 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:58.892477 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96589a36-9b8d-4974-8128-95e22ef47036-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-983490-87kfg\" (UID: \"96589a36-9b8d-4974-8128-95e22ef47036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-983490-87kfg" Sep 17 23:24:58.893092 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:58.893067 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96589a36-9b8d-4974-8128-95e22ef47036-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-983490-87kfg\" (UID: \"96589a36-9b8d-4974-8128-95e22ef47036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-983490-87kfg" Sep 17 23:24:58.995320 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:58.995290 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-983490-87kfg" Sep 17 23:24:59.266165 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:59.266135 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-983490-87kfg"] Sep 17 23:24:59.269001 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:24:59.268971 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96589a36_9b8d_4974_8128_95e22ef47036.slice/crio-c2c1f14dcfa5458ff53931205fd490b9b5ba15cc17396fb22b1c652411cfc55f WatchSource:0}: Error finding container c2c1f14dcfa5458ff53931205fd490b9b5ba15cc17396fb22b1c652411cfc55f: Status 404 returned error can't find the container with id c2c1f14dcfa5458ff53931205fd490b9b5ba15cc17396fb22b1c652411cfc55f Sep 17 23:24:59.961983 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:59.961902 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-983490-87kfg" event={"ID":"96589a36-9b8d-4974-8128-95e22ef47036","Type":"ContainerStarted","Data":"8abbe3e2c39a66a972fbb13412389de682173593398c84f1a7a9b384a669b1a4"} Sep 17 23:24:59.961983 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:59.961965 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-983490-87kfg" event={"ID":"96589a36-9b8d-4974-8128-95e22ef47036","Type":"ContainerStarted","Data":"c2c1f14dcfa5458ff53931205fd490b9b5ba15cc17396fb22b1c652411cfc55f"} Sep 17 23:24:59.978746 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:24:59.978695 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-983490-87kfg" podStartSLOduration=1.978679735 podStartE2EDuration="1.978679735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:24:59.976788832 +0000 UTC m=+1138.910695967" watchObservedRunningTime="2026-09-17 23:24:59.978679735 +0000 UTC m=+1138.912586869" Sep 17 23:25:00.969783 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:00.969749 2582 generic.go:356] "Generic (PLEG): container finished" podID="96589a36-9b8d-4974-8128-95e22ef47036" containerID="8abbe3e2c39a66a972fbb13412389de682173593398c84f1a7a9b384a669b1a4" exitCode=0 Sep 17 23:25:00.970259 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:00.969842 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-983490-87kfg" event={"ID":"96589a36-9b8d-4974-8128-95e22ef47036","Type":"ContainerDied","Data":"8abbe3e2c39a66a972fbb13412389de682173593398c84f1a7a9b384a669b1a4"} Sep 17 23:25:02.113869 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:02.113847 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-983490-87kfg" Sep 17 23:25:02.124850 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:02.124827 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96589a36-9b8d-4974-8128-95e22ef47036-maas-api-service-ca\") pod \"96589a36-9b8d-4974-8128-95e22ef47036\" (UID: \"96589a36-9b8d-4974-8128-95e22ef47036\") " Sep 17 23:25:02.125150 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:02.125125 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96589a36-9b8d-4974-8128-95e22ef47036-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96589a36-9b8d-4974-8128-95e22ef47036" (UID: "96589a36-9b8d-4974-8128-95e22ef47036"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:25:02.125261 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:02.125245 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96589a36-9b8d-4974-8128-95e22ef47036-maas-api-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:02.986377 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:02.986346 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-983490-87kfg" event={"ID":"96589a36-9b8d-4974-8128-95e22ef47036","Type":"ContainerDied","Data":"c2c1f14dcfa5458ff53931205fd490b9b5ba15cc17396fb22b1c652411cfc55f"} Sep 17 23:25:02.986574 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:02.986418 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2c1f14dcfa5458ff53931205fd490b9b5ba15cc17396fb22b1c652411cfc55f" Sep 17 23:25:02.986574 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:02.986358 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-983490-87kfg" Sep 17 23:25:04.506323 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:04.506282 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq"] Sep 17 23:25:04.506812 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:04.506588 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq" podUID="4fc97a7d-1203-43b1-840b-1a2fd6a24a01" containerName="payload-pre-processing" containerID="cri-o://92642818bae26be2203aaf75a9f1f09e605365f2153a8aab826abde4542b08f0" gracePeriod=30 Sep 17 23:25:04.718023 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:04.717989 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63b144-ztbrb"] Sep 17 23:25:04.721940 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:04.721914 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63b144-ztbrb"] Sep 17 23:25:04.792781 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:04.792714 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq" Sep 17 23:25:04.850653 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:04.850621 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlqvd\" (UniqueName: \"kubernetes.io/projected/4fc97a7d-1203-43b1-840b-1a2fd6a24a01-kube-api-access-qlqvd\") pod \"4fc97a7d-1203-43b1-840b-1a2fd6a24a01\" (UID: \"4fc97a7d-1203-43b1-840b-1a2fd6a24a01\") " Sep 17 23:25:04.850831 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:04.850688 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc97a7d-1203-43b1-840b-1a2fd6a24a01-plugins-config-volume\") pod \"4fc97a7d-1203-43b1-840b-1a2fd6a24a01\" (UID: \"4fc97a7d-1203-43b1-840b-1a2fd6a24a01\") " Sep 17 23:25:04.851058 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:04.851035 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc97a7d-1203-43b1-840b-1a2fd6a24a01-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4fc97a7d-1203-43b1-840b-1a2fd6a24a01" (UID: "4fc97a7d-1203-43b1-840b-1a2fd6a24a01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:25:04.852748 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:04.852721 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc97a7d-1203-43b1-840b-1a2fd6a24a01-kube-api-access-qlqvd" (OuterVolumeSpecName: "kube-api-access-qlqvd") pod "4fc97a7d-1203-43b1-840b-1a2fd6a24a01" (UID: "4fc97a7d-1203-43b1-840b-1a2fd6a24a01"). InnerVolumeSpecName "kube-api-access-qlqvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:25:04.952360 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:04.952324 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc97a7d-1203-43b1-840b-1a2fd6a24a01-plugins-config-volume\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:04.952360 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:04.952356 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlqvd\" (UniqueName: \"kubernetes.io/projected/4fc97a7d-1203-43b1-840b-1a2fd6a24a01-kube-api-access-qlqvd\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:04.999302 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:04.999267 2582 generic.go:356] "Generic (PLEG): container finished" podID="4fc97a7d-1203-43b1-840b-1a2fd6a24a01" containerID="92642818bae26be2203aaf75a9f1f09e605365f2153a8aab826abde4542b08f0" exitCode=0 Sep 17 23:25:04.999459 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:04.999315 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq" Sep 17 23:25:04.999459 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:04.999347 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq" event={"ID":"4fc97a7d-1203-43b1-840b-1a2fd6a24a01","Type":"ContainerDied","Data":"92642818bae26be2203aaf75a9f1f09e605365f2153a8aab826abde4542b08f0"} Sep 17 23:25:04.999459 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:04.999401 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq" event={"ID":"4fc97a7d-1203-43b1-840b-1a2fd6a24a01","Type":"ContainerDied","Data":"b554a437adf2662984c1420f425d5dce04a7c47c6039729cc795fb7ecb2bc9b9"} Sep 17 23:25:04.999459 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:04.999425 2582 scope.go:117] "RemoveContainer" containerID="92642818bae26be2203aaf75a9f1f09e605365f2153a8aab826abde4542b08f0" Sep 17 23:25:05.008465 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:05.008449 2582 scope.go:117] "RemoveContainer" containerID="92642818bae26be2203aaf75a9f1f09e605365f2153a8aab826abde4542b08f0" Sep 17 23:25:05.008745 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:25:05.008728 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92642818bae26be2203aaf75a9f1f09e605365f2153a8aab826abde4542b08f0\": container with ID starting with 92642818bae26be2203aaf75a9f1f09e605365f2153a8aab826abde4542b08f0 not found: ID does not exist" containerID="92642818bae26be2203aaf75a9f1f09e605365f2153a8aab826abde4542b08f0" Sep 17 23:25:05.008790 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:05.008753 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92642818bae26be2203aaf75a9f1f09e605365f2153a8aab826abde4542b08f0"} err="failed to get container status \"92642818bae26be2203aaf75a9f1f09e605365f2153a8aab826abde4542b08f0\": rpc error: code = NotFound desc = could not find container \"92642818bae26be2203aaf75a9f1f09e605365f2153a8aab826abde4542b08f0\": container with ID starting with 92642818bae26be2203aaf75a9f1f09e605365f2153a8aab826abde4542b08f0 not found: ID does not exist" Sep 17 23:25:05.019977 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:05.019956 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq"] Sep 17 23:25:05.024187 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:05.024168 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-63b144-68c55f5bdb-j2jtq"] Sep 17 23:25:05.294126 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:05.294092 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fc97a7d-1203-43b1-840b-1a2fd6a24a01" path="/var/lib/kubelet/pods/4fc97a7d-1203-43b1-840b-1a2fd6a24a01/volumes" Sep 17 23:25:05.294459 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:05.294433 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7cf450a-9fab-4708-bf7c-378035a2e6e5" path="/var/lib/kubelet/pods/a7cf450a-9fab-4708-bf7c-378035a2e6e5/volumes" Sep 17 23:25:09.347204 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:09.347166 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-983490-87kfg"] Sep 17 23:25:09.350585 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:09.350549 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-983490-87kfg"] Sep 17 23:25:11.304268 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:11.304234 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96589a36-9b8d-4974-8128-95e22ef47036" path="/var/lib/kubelet/pods/96589a36-9b8d-4974-8128-95e22ef47036/volumes" Sep 17 23:25:15.813900 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:15.813822 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7"] Sep 17 23:25:15.814282 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:15.814109 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" podUID="713a3a60-185a-4ee3-8e53-803ee57e08a9" containerName="istio-proxy" containerID="cri-o://e49f1349273e4ea2c9cb72e7d0548df8dbb641755b09ed90f4ee451a34838c24" gracePeriod=30 Sep 17 23:25:20.546942 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:20.546905 2582 patch_prober.go:28] interesting pod/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 23:25:20.547324 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:20.546979 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" podUID="713a3a60-185a-4ee3-8e53-803ee57e08a9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 23:25:21.076310 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.076287 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:25:21.099153 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.099011 2582 generic.go:356] "Generic (PLEG): container finished" podID="713a3a60-185a-4ee3-8e53-803ee57e08a9" containerID="e49f1349273e4ea2c9cb72e7d0548df8dbb641755b09ed90f4ee451a34838c24" exitCode=0 Sep 17 23:25:21.099153 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.099098 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" Sep 17 23:25:21.099415 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.099194 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" event={"ID":"713a3a60-185a-4ee3-8e53-803ee57e08a9","Type":"ContainerDied","Data":"e49f1349273e4ea2c9cb72e7d0548df8dbb641755b09ed90f4ee451a34838c24"} Sep 17 23:25:21.099415 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.099233 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7" event={"ID":"713a3a60-185a-4ee3-8e53-803ee57e08a9","Type":"ContainerDied","Data":"f639d214a06795324dc323215c046459e62698053c4db8b745005dc0f0dda902"} Sep 17 23:25:21.099415 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.099259 2582 scope.go:117] "RemoveContainer" containerID="e49f1349273e4ea2c9cb72e7d0548df8dbb641755b09ed90f4ee451a34838c24" Sep 17 23:25:21.109623 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.109600 2582 scope.go:117] "RemoveContainer" containerID="c0ad59aee86c9a801468b85acbc478c2bf69d57e457decbbba4aebf86be87cfa" Sep 17 23:25:21.120148 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.120128 2582 scope.go:117] "RemoveContainer" containerID="e49f1349273e4ea2c9cb72e7d0548df8dbb641755b09ed90f4ee451a34838c24" Sep 17 23:25:21.120424 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:25:21.120406 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e49f1349273e4ea2c9cb72e7d0548df8dbb641755b09ed90f4ee451a34838c24\": container with ID starting with e49f1349273e4ea2c9cb72e7d0548df8dbb641755b09ed90f4ee451a34838c24 not found: ID does not exist" containerID="e49f1349273e4ea2c9cb72e7d0548df8dbb641755b09ed90f4ee451a34838c24" Sep 17 23:25:21.120471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.120433 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e49f1349273e4ea2c9cb72e7d0548df8dbb641755b09ed90f4ee451a34838c24"} err="failed to get container status \"e49f1349273e4ea2c9cb72e7d0548df8dbb641755b09ed90f4ee451a34838c24\": rpc error: code = NotFound desc = could not find container \"e49f1349273e4ea2c9cb72e7d0548df8dbb641755b09ed90f4ee451a34838c24\": container with ID starting with e49f1349273e4ea2c9cb72e7d0548df8dbb641755b09ed90f4ee451a34838c24 not found: ID does not exist" Sep 17 23:25:21.120471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.120453 2582 scope.go:117] "RemoveContainer" containerID="c0ad59aee86c9a801468b85acbc478c2bf69d57e457decbbba4aebf86be87cfa" Sep 17 23:25:21.120764 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:25:21.120744 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ad59aee86c9a801468b85acbc478c2bf69d57e457decbbba4aebf86be87cfa\": container with ID starting with c0ad59aee86c9a801468b85acbc478c2bf69d57e457decbbba4aebf86be87cfa not found: ID does not exist" containerID="c0ad59aee86c9a801468b85acbc478c2bf69d57e457decbbba4aebf86be87cfa" Sep 17 23:25:21.120815 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.120774 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ad59aee86c9a801468b85acbc478c2bf69d57e457decbbba4aebf86be87cfa"} err="failed to get container status \"c0ad59aee86c9a801468b85acbc478c2bf69d57e457decbbba4aebf86be87cfa\": rpc error: code = NotFound desc = could not find container \"c0ad59aee86c9a801468b85acbc478c2bf69d57e457decbbba4aebf86be87cfa\": container with ID starting with c0ad59aee86c9a801468b85acbc478c2bf69d57e457decbbba4aebf86be87cfa not found: ID does not exist" Sep 17 23:25:21.218163 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.218132 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-credential-socket\") pod \"713a3a60-185a-4ee3-8e53-803ee57e08a9\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " Sep 17 23:25:21.218311 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.218178 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-workload-socket\") pod \"713a3a60-185a-4ee3-8e53-803ee57e08a9\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " Sep 17 23:25:21.218311 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.218217 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-data\") pod \"713a3a60-185a-4ee3-8e53-803ee57e08a9\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " Sep 17 23:25:21.218311 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.218249 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-podinfo\") pod \"713a3a60-185a-4ee3-8e53-803ee57e08a9\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " Sep 17 23:25:21.218311 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.218282 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-workload-certs\") pod \"713a3a60-185a-4ee3-8e53-803ee57e08a9\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " Sep 17 23:25:21.218488 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.218345 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713a3a60-185a-4ee3-8e53-803ee57e08a9-istiod-ca-cert\") pod \"713a3a60-185a-4ee3-8e53-803ee57e08a9\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " Sep 17 23:25:21.218488 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.218372 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jq6w\" (UniqueName: \"kubernetes.io/projected/713a3a60-185a-4ee3-8e53-803ee57e08a9-kube-api-access-2jq6w\") pod \"713a3a60-185a-4ee3-8e53-803ee57e08a9\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " Sep 17 23:25:21.218488 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.218392 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "713a3a60-185a-4ee3-8e53-803ee57e08a9" (UID: "713a3a60-185a-4ee3-8e53-803ee57e08a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:25:21.218488 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.218414 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-envoy\") pod \"713a3a60-185a-4ee3-8e53-803ee57e08a9\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " Sep 17 23:25:21.218488 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.218456 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-token\") pod \"713a3a60-185a-4ee3-8e53-803ee57e08a9\" (UID: \"713a3a60-185a-4ee3-8e53-803ee57e08a9\") " Sep 17 23:25:21.218781 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.218524 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "713a3a60-185a-4ee3-8e53-803ee57e08a9" (UID: "713a3a60-185a-4ee3-8e53-803ee57e08a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:25:21.218781 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.218569 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-data" (OuterVolumeSpecName: "istio-data") pod "713a3a60-185a-4ee3-8e53-803ee57e08a9" (UID: "713a3a60-185a-4ee3-8e53-803ee57e08a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:25:21.218781 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.218616 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "713a3a60-185a-4ee3-8e53-803ee57e08a9" (UID: "713a3a60-185a-4ee3-8e53-803ee57e08a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:25:21.218781 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.218747 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/713a3a60-185a-4ee3-8e53-803ee57e08a9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "713a3a60-185a-4ee3-8e53-803ee57e08a9" (UID: "713a3a60-185a-4ee3-8e53-803ee57e08a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:25:21.218988 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.218853 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-credential-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:21.218988 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.218869 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-workload-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:21.218988 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.218881 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-data\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:21.218988 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.218893 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-workload-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:21.218988 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.218905 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/713a3a60-185a-4ee3-8e53-803ee57e08a9-istiod-ca-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:21.220620 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.220594 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-token" (OuterVolumeSpecName: "istio-token") pod "713a3a60-185a-4ee3-8e53-803ee57e08a9" (UID: "713a3a60-185a-4ee3-8e53-803ee57e08a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:25:21.220620 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.220607 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713a3a60-185a-4ee3-8e53-803ee57e08a9-kube-api-access-2jq6w" (OuterVolumeSpecName: "kube-api-access-2jq6w") pod "713a3a60-185a-4ee3-8e53-803ee57e08a9" (UID: "713a3a60-185a-4ee3-8e53-803ee57e08a9"). InnerVolumeSpecName "kube-api-access-2jq6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:25:21.220862 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.220840 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "713a3a60-185a-4ee3-8e53-803ee57e08a9" (UID: "713a3a60-185a-4ee3-8e53-803ee57e08a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:25:21.221006 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.220988 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "713a3a60-185a-4ee3-8e53-803ee57e08a9" (UID: "713a3a60-185a-4ee3-8e53-803ee57e08a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:25:21.320182 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.320152 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-envoy\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:21.320182 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.320177 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:21.320328 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.320193 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/713a3a60-185a-4ee3-8e53-803ee57e08a9-istio-podinfo\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:21.320328 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.320205 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jq6w\" (UniqueName: \"kubernetes.io/projected/713a3a60-185a-4ee3-8e53-803ee57e08a9-kube-api-access-2jq6w\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:21.416984 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.416895 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7"] Sep 17 23:25:21.418885 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:21.418867 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-983490-openshift-default-fcb48c9d-dqzq7"] Sep 17 23:25:23.294547 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:23.294480 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="713a3a60-185a-4ee3-8e53-803ee57e08a9" path="/var/lib/kubelet/pods/713a3a60-185a-4ee3-8e53-803ee57e08a9/volumes" Sep 17 23:25:26.126813 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:26.126780 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r"] Sep 17 23:25:26.127265 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:26.127044 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" podUID="2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" containerName="istio-proxy" containerID="cri-o://713f0ecede515e7327180da3b6a30c618815d72a0ea2eeff9d315e4165d00ddf" gracePeriod=30 Sep 17 23:25:31.380665 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.380630 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:25:31.517232 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.517190 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-workload-socket\") pod \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " Sep 17 23:25:31.517425 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.517279 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9vhx\" (UniqueName: \"kubernetes.io/projected/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-kube-api-access-b9vhx\") pod \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " Sep 17 23:25:31.517425 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.517309 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-podinfo\") pod \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " Sep 17 23:25:31.517425 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.517360 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-envoy\") pod \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " Sep 17 23:25:31.517597 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.517485 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" (UID: "2c70e8e6-75d0-4559-8819-36d5dc9d4a4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:25:31.517597 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.517530 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-token\") pod \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " Sep 17 23:25:31.517597 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.517569 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-credential-socket\") pod \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " Sep 17 23:25:31.517746 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.517597 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istiod-ca-cert\") pod \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " Sep 17 23:25:31.517746 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.517647 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-data\") pod \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " Sep 17 23:25:31.517746 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.517723 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-workload-certs\") pod \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\" (UID: \"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b\") " Sep 17 23:25:31.517895 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.517819 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" (UID: "2c70e8e6-75d0-4559-8819-36d5dc9d4a4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:25:31.517997 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.517979 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" (UID: "2c70e8e6-75d0-4559-8819-36d5dc9d4a4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:25:31.518062 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.518026 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-data" (OuterVolumeSpecName: "istio-data") pod "2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" (UID: "2c70e8e6-75d0-4559-8819-36d5dc9d4a4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:25:31.518175 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.518159 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-workload-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:31.518227 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.518182 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-credential-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:31.518227 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.518197 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-data\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:31.518227 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.518211 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-workload-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:31.518462 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.518445 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" (UID: "2c70e8e6-75d0-4559-8819-36d5dc9d4a4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:25:31.519710 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.519687 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" (UID: "2c70e8e6-75d0-4559-8819-36d5dc9d4a4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:25:31.519797 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.519708 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-token" (OuterVolumeSpecName: "istio-token") pod "2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" (UID: "2c70e8e6-75d0-4559-8819-36d5dc9d4a4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:25:31.519929 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.519908 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" (UID: "2c70e8e6-75d0-4559-8819-36d5dc9d4a4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:25:31.519990 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.519924 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-kube-api-access-b9vhx" (OuterVolumeSpecName: "kube-api-access-b9vhx") pod "2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" (UID: "2c70e8e6-75d0-4559-8819-36d5dc9d4a4b"). InnerVolumeSpecName "kube-api-access-b9vhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:25:31.618759 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.618726 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9vhx\" (UniqueName: \"kubernetes.io/projected/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-kube-api-access-b9vhx\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:31.618759 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.618756 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-podinfo\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:31.618966 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.618769 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-envoy\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:31.618966 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.618777 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istio-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:31.618966 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:31.618784 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b-istiod-ca-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:32.168323 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:32.168292 2582 generic.go:356] "Generic (PLEG): container finished" podID="2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" containerID="713f0ecede515e7327180da3b6a30c618815d72a0ea2eeff9d315e4165d00ddf" exitCode=0 Sep 17 23:25:32.168638 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:32.168341 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" Sep 17 23:25:32.168638 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:32.168373 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" event={"ID":"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b","Type":"ContainerDied","Data":"713f0ecede515e7327180da3b6a30c618815d72a0ea2eeff9d315e4165d00ddf"} Sep 17 23:25:32.168638 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:32.168430 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r" event={"ID":"2c70e8e6-75d0-4559-8819-36d5dc9d4a4b","Type":"ContainerDied","Data":"72a55d582a1b573ca94a1aa03c99fc7b5b613024eb92d8ac4dc09b8847ea713c"} Sep 17 23:25:32.168638 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:32.168452 2582 scope.go:117] "RemoveContainer" containerID="713f0ecede515e7327180da3b6a30c618815d72a0ea2eeff9d315e4165d00ddf" Sep 17 23:25:32.178817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:32.178795 2582 scope.go:117] "RemoveContainer" containerID="3d4e2b3f85d4f51db19daa635eed96255284a9a763459ce48066139def54650a" Sep 17 23:25:32.190317 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:32.190299 2582 scope.go:117] "RemoveContainer" containerID="713f0ecede515e7327180da3b6a30c618815d72a0ea2eeff9d315e4165d00ddf" Sep 17 23:25:32.190612 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:25:32.190594 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713f0ecede515e7327180da3b6a30c618815d72a0ea2eeff9d315e4165d00ddf\": container with ID starting with 713f0ecede515e7327180da3b6a30c618815d72a0ea2eeff9d315e4165d00ddf not found: ID does not exist" containerID="713f0ecede515e7327180da3b6a30c618815d72a0ea2eeff9d315e4165d00ddf" Sep 17 23:25:32.190665 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:32.190621 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713f0ecede515e7327180da3b6a30c618815d72a0ea2eeff9d315e4165d00ddf"} err="failed to get container status \"713f0ecede515e7327180da3b6a30c618815d72a0ea2eeff9d315e4165d00ddf\": rpc error: code = NotFound desc = could not find container \"713f0ecede515e7327180da3b6a30c618815d72a0ea2eeff9d315e4165d00ddf\": container with ID starting with 713f0ecede515e7327180da3b6a30c618815d72a0ea2eeff9d315e4165d00ddf not found: ID does not exist" Sep 17 23:25:32.190665 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:32.190638 2582 scope.go:117] "RemoveContainer" containerID="3d4e2b3f85d4f51db19daa635eed96255284a9a763459ce48066139def54650a" Sep 17 23:25:32.190885 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:25:32.190864 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d4e2b3f85d4f51db19daa635eed96255284a9a763459ce48066139def54650a\": container with ID starting with 3d4e2b3f85d4f51db19daa635eed96255284a9a763459ce48066139def54650a not found: ID does not exist" containerID="3d4e2b3f85d4f51db19daa635eed96255284a9a763459ce48066139def54650a" Sep 17 23:25:32.190942 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:32.190895 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4e2b3f85d4f51db19daa635eed96255284a9a763459ce48066139def54650a"} err="failed to get container status \"3d4e2b3f85d4f51db19daa635eed96255284a9a763459ce48066139def54650a\": rpc error: code = NotFound desc = could not find container \"3d4e2b3f85d4f51db19daa635eed96255284a9a763459ce48066139def54650a\": container with ID starting with 3d4e2b3f85d4f51db19daa635eed96255284a9a763459ce48066139def54650a not found: ID does not exist" Sep 17 23:25:32.195798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:32.195774 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r"] Sep 17 23:25:32.200200 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:32.200177 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-861522-openshift-default-75f46cb4bd-fvd4r"] Sep 17 23:25:33.296080 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:33.296051 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" path="/var/lib/kubelet/pods/2c70e8e6-75d0-4559-8819-36d5dc9d4a4b/volumes" Sep 17 23:25:44.228890 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.228850 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf"] Sep 17 23:25:44.229299 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.229117 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" podUID="869ba654-13cf-4bfe-9c6f-68bcf59df02f" containerName="istio-proxy" containerID="cri-o://2fe465fa67abb9c0a958b6eb2762b03b276d0f93726708bf788c1798c58ddf15" gracePeriod=30 Sep 17 23:25:44.975171 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.975138 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh"] Sep 17 23:25:44.975899 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.975873 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" containerName="istio-proxy" Sep 17 23:25:44.976062 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.976038 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" containerName="istio-proxy" Sep 17 23:25:44.976162 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.976096 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96589a36-9b8d-4974-8128-95e22ef47036" containerName="revoke-keys" Sep 17 23:25:44.976162 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.976110 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="96589a36-9b8d-4974-8128-95e22ef47036" containerName="revoke-keys" Sep 17 23:25:44.976162 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.976130 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc97a7d-1203-43b1-840b-1a2fd6a24a01" containerName="payload-pre-processing" Sep 17 23:25:44.976162 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.976138 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc97a7d-1203-43b1-840b-1a2fd6a24a01" containerName="payload-pre-processing" Sep 17 23:25:44.976162 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.976158 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" containerName="istio-proxy" Sep 17 23:25:44.976404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.976167 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" containerName="istio-proxy" Sep 17 23:25:44.976404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.976184 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="713a3a60-185a-4ee3-8e53-803ee57e08a9" containerName="istio-proxy" Sep 17 23:25:44.976404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.976193 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="713a3a60-185a-4ee3-8e53-803ee57e08a9" containerName="istio-proxy" Sep 17 23:25:44.976404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.976215 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="713a3a60-185a-4ee3-8e53-803ee57e08a9" containerName="istio-proxy" Sep 17 23:25:44.976404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.976222 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="713a3a60-185a-4ee3-8e53-803ee57e08a9" containerName="istio-proxy" Sep 17 23:25:44.976404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.976349 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="713a3a60-185a-4ee3-8e53-803ee57e08a9" containerName="istio-proxy" Sep 17 23:25:44.976404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.976360 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="96589a36-9b8d-4974-8128-95e22ef47036" containerName="revoke-keys" Sep 17 23:25:44.976404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.976369 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" containerName="istio-proxy" Sep 17 23:25:44.976404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.976377 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc97a7d-1203-43b1-840b-1a2fd6a24a01" containerName="payload-pre-processing" Sep 17 23:25:44.981150 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.981124 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:44.983615 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.983598 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-7f7059-openshift-default-dockercfg-cr69r\"" Sep 17 23:25:44.991783 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:44.991761 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh"] Sep 17 23:25:45.036180 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.036154 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-credential-socket\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.036312 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.036186 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istiod-ca-cert\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.036312 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.036207 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-workload-socket\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.036312 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.036291 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-token\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.036413 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.036339 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-podinfo\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.036413 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.036360 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-data\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.036413 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.036383 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-envoy\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.036413 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.036408 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-workload-certs\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.036563 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.036428 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sq2m\" (UniqueName: \"kubernetes.io/projected/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-kube-api-access-7sq2m\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.136999 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.136967 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-podinfo\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.136999 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.137006 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-data\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.137244 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.137025 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-envoy\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.137244 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.137043 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-workload-certs\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.137244 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.137066 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sq2m\" (UniqueName: \"kubernetes.io/projected/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-kube-api-access-7sq2m\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.137244 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.137104 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-credential-socket\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.137244 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.137122 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istiod-ca-cert\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.137244 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.137154 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-workload-socket\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.137244 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.137226 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-token\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.137619 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.137564 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-workload-certs\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.137687 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.137624 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-credential-socket\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.137783 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.137750 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-workload-socket\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.137852 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.137834 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-data\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.137951 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.137931 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istiod-ca-cert\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.139623 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.139596 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-envoy\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.139889 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.139870 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-podinfo\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.163582 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.163557 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sq2m\" (UniqueName: \"kubernetes.io/projected/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-kube-api-access-7sq2m\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.164267 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.164250 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-token\") pod \"e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.298395 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.298310 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:45.577280 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.577254 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh"] Sep 17 23:25:45.578963 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:25:45.578937 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00bd3695_d1d6_4ea5_a5bc_9e27ed7f35f7.slice/crio-3e8be1caa81f971cc5db06194da38856ec704bb5c72ebba8fa443961904ed47b WatchSource:0}: Error finding container 3e8be1caa81f971cc5db06194da38856ec704bb5c72ebba8fa443961904ed47b: Status 404 returned error can't find the container with id 3e8be1caa81f971cc5db06194da38856ec704bb5c72ebba8fa443961904ed47b Sep 17 23:25:45.581209 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.581174 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:25:45.581270 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.581239 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:25:45.581306 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:45.581268 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:25:46.259383 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:46.259346 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" event={"ID":"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7","Type":"ContainerStarted","Data":"e417fda6cae8769cddde15d30f2ae166f3c04ec3ca59096394f490a5a5b50b7e"} Sep 17 23:25:46.259591 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:46.259438 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" event={"ID":"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7","Type":"ContainerStarted","Data":"3e8be1caa81f971cc5db06194da38856ec704bb5c72ebba8fa443961904ed47b"} Sep 17 23:25:46.280722 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:46.280671 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" podStartSLOduration=2.280655927 podStartE2EDuration="2.280655927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:46.279870483 +0000 UTC m=+1185.213777618" watchObservedRunningTime="2026-09-17 23:25:46.280655927 +0000 UTC m=+1185.214563062" Sep 17 23:25:46.298629 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:46.298596 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:46.303949 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:46.303923 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:47.265288 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:47.265259 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:47.266204 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:47.266185 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:25:49.485812 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.485789 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:25:49.585323 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.585246 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-data\") pod \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " Sep 17 23:25:49.585323 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.585287 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-credential-socket\") pod \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " Sep 17 23:25:49.585542 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.585329 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-workload-certs\") pod \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " Sep 17 23:25:49.585542 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.585380 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-envoy\") pod \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " Sep 17 23:25:49.585542 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.585407 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-workload-socket\") pod \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " Sep 17 23:25:49.585542 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.585455 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istiod-ca-cert\") pod \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " Sep 17 23:25:49.585542 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.585487 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-token\") pod \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " Sep 17 23:25:49.585812 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.585568 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n86d\" (UniqueName: \"kubernetes.io/projected/869ba654-13cf-4bfe-9c6f-68bcf59df02f-kube-api-access-7n86d\") pod \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " Sep 17 23:25:49.585812 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.585593 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-podinfo\") pod \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\" (UID: \"869ba654-13cf-4bfe-9c6f-68bcf59df02f\") " Sep 17 23:25:49.585812 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.585588 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-data" (OuterVolumeSpecName: "istio-data") pod "869ba654-13cf-4bfe-9c6f-68bcf59df02f" (UID: "869ba654-13cf-4bfe-9c6f-68bcf59df02f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:25:49.585812 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.585600 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "869ba654-13cf-4bfe-9c6f-68bcf59df02f" (UID: "869ba654-13cf-4bfe-9c6f-68bcf59df02f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:25:49.585812 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.585749 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "869ba654-13cf-4bfe-9c6f-68bcf59df02f" (UID: "869ba654-13cf-4bfe-9c6f-68bcf59df02f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:25:49.586009 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.585824 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "869ba654-13cf-4bfe-9c6f-68bcf59df02f" (UID: "869ba654-13cf-4bfe-9c6f-68bcf59df02f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:25:49.586009 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.585981 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-workload-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:49.586009 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.585992 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-data\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:49.586009 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.586001 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-credential-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:49.586009 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.586009 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-workload-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:49.586470 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.586233 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "869ba654-13cf-4bfe-9c6f-68bcf59df02f" (UID: "869ba654-13cf-4bfe-9c6f-68bcf59df02f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:25:49.587859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.587832 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "869ba654-13cf-4bfe-9c6f-68bcf59df02f" (UID: "869ba654-13cf-4bfe-9c6f-68bcf59df02f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:25:49.587991 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.587973 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-token" (OuterVolumeSpecName: "istio-token") pod "869ba654-13cf-4bfe-9c6f-68bcf59df02f" (UID: "869ba654-13cf-4bfe-9c6f-68bcf59df02f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:25:49.588196 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.588174 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "869ba654-13cf-4bfe-9c6f-68bcf59df02f" (UID: "869ba654-13cf-4bfe-9c6f-68bcf59df02f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:25:49.588196 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.588176 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869ba654-13cf-4bfe-9c6f-68bcf59df02f-kube-api-access-7n86d" (OuterVolumeSpecName: "kube-api-access-7n86d") pod "869ba654-13cf-4bfe-9c6f-68bcf59df02f" (UID: "869ba654-13cf-4bfe-9c6f-68bcf59df02f"). InnerVolumeSpecName "kube-api-access-7n86d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:25:49.686941 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.686916 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-envoy\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:49.686941 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.686940 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istiod-ca-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:49.687095 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.686952 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:49.687095 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.686960 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n86d\" (UniqueName: \"kubernetes.io/projected/869ba654-13cf-4bfe-9c6f-68bcf59df02f-kube-api-access-7n86d\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:49.687095 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:49.686970 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/869ba654-13cf-4bfe-9c6f-68bcf59df02f-istio-podinfo\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:25:50.290815 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:50.290784 2582 generic.go:356] "Generic (PLEG): container finished" podID="869ba654-13cf-4bfe-9c6f-68bcf59df02f" containerID="2fe465fa67abb9c0a958b6eb2762b03b276d0f93726708bf788c1798c58ddf15" exitCode=0 Sep 17 23:25:50.290815 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:50.290820 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" Sep 17 23:25:50.291061 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:50.290879 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" event={"ID":"869ba654-13cf-4bfe-9c6f-68bcf59df02f","Type":"ContainerDied","Data":"2fe465fa67abb9c0a958b6eb2762b03b276d0f93726708bf788c1798c58ddf15"} Sep 17 23:25:50.291061 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:50.290933 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf" event={"ID":"869ba654-13cf-4bfe-9c6f-68bcf59df02f","Type":"ContainerDied","Data":"ea5b8a33c36d887a36911436748edc21893157a72578f0b2a5d493b04c16499a"} Sep 17 23:25:50.291061 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:50.290962 2582 scope.go:117] "RemoveContainer" containerID="2fe465fa67abb9c0a958b6eb2762b03b276d0f93726708bf788c1798c58ddf15" Sep 17 23:25:50.301492 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:50.301472 2582 scope.go:117] "RemoveContainer" containerID="ed54d655403db9afad45db8d56508624a213fa7ad4a776a9dfaa11bc51b033a0" Sep 17 23:25:50.309926 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:50.309911 2582 scope.go:117] "RemoveContainer" containerID="2fe465fa67abb9c0a958b6eb2762b03b276d0f93726708bf788c1798c58ddf15" Sep 17 23:25:50.310194 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:25:50.310178 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe465fa67abb9c0a958b6eb2762b03b276d0f93726708bf788c1798c58ddf15\": container with ID starting with 2fe465fa67abb9c0a958b6eb2762b03b276d0f93726708bf788c1798c58ddf15 not found: ID does not exist" containerID="2fe465fa67abb9c0a958b6eb2762b03b276d0f93726708bf788c1798c58ddf15" Sep 17 23:25:50.310260 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:50.310199 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe465fa67abb9c0a958b6eb2762b03b276d0f93726708bf788c1798c58ddf15"} err="failed to get container status \"2fe465fa67abb9c0a958b6eb2762b03b276d0f93726708bf788c1798c58ddf15\": rpc error: code = NotFound desc = could not find container \"2fe465fa67abb9c0a958b6eb2762b03b276d0f93726708bf788c1798c58ddf15\": container with ID starting with 2fe465fa67abb9c0a958b6eb2762b03b276d0f93726708bf788c1798c58ddf15 not found: ID does not exist" Sep 17 23:25:50.310260 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:50.310214 2582 scope.go:117] "RemoveContainer" containerID="ed54d655403db9afad45db8d56508624a213fa7ad4a776a9dfaa11bc51b033a0" Sep 17 23:25:50.310479 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:25:50.310462 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed54d655403db9afad45db8d56508624a213fa7ad4a776a9dfaa11bc51b033a0\": container with ID starting with ed54d655403db9afad45db8d56508624a213fa7ad4a776a9dfaa11bc51b033a0 not found: ID does not exist" containerID="ed54d655403db9afad45db8d56508624a213fa7ad4a776a9dfaa11bc51b033a0" Sep 17 23:25:50.310543 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:50.310482 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed54d655403db9afad45db8d56508624a213fa7ad4a776a9dfaa11bc51b033a0"} err="failed to get container status \"ed54d655403db9afad45db8d56508624a213fa7ad4a776a9dfaa11bc51b033a0\": rpc error: code = NotFound desc = could not find container \"ed54d655403db9afad45db8d56508624a213fa7ad4a776a9dfaa11bc51b033a0\": container with ID starting with ed54d655403db9afad45db8d56508624a213fa7ad4a776a9dfaa11bc51b033a0 not found: ID does not exist" Sep 17 23:25:50.313690 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:50.313669 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf"] Sep 17 23:25:50.317961 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:50.317942 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-63b144-openshift-default-54764d8c46-l9cdf"] Sep 17 23:25:51.298634 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:25:51.298593 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="869ba654-13cf-4bfe-9c6f-68bcf59df02f" path="/var/lib/kubelet/pods/869ba654-13cf-4bfe-9c6f-68bcf59df02f/volumes" Sep 17 23:26:03.606989 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.606899 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj"] Sep 17 23:26:03.607601 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.607578 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="869ba654-13cf-4bfe-9c6f-68bcf59df02f" containerName="istio-proxy" Sep 17 23:26:03.607601 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.607602 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="869ba654-13cf-4bfe-9c6f-68bcf59df02f" containerName="istio-proxy" Sep 17 23:26:03.607765 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.607614 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="869ba654-13cf-4bfe-9c6f-68bcf59df02f" containerName="istio-proxy" Sep 17 23:26:03.607765 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.607622 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="869ba654-13cf-4bfe-9c6f-68bcf59df02f" containerName="istio-proxy" Sep 17 23:26:03.607765 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.607712 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="869ba654-13cf-4bfe-9c6f-68bcf59df02f" containerName="istio-proxy" Sep 17 23:26:03.607765 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.607725 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="713a3a60-185a-4ee3-8e53-803ee57e08a9" containerName="istio-proxy" Sep 17 23:26:03.607765 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.607734 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="869ba654-13cf-4bfe-9c6f-68bcf59df02f" containerName="istio-proxy" Sep 17 23:26:03.607765 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.607745 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c70e8e6-75d0-4559-8819-36d5dc9d4a4b" containerName="istio-proxy" Sep 17 23:26:03.612640 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.612613 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj"] Sep 17 23:26:03.612775 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.612702 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.615423 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.615399 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-df969c31-openshift-default-dockercfg-rtw5f\"" Sep 17 23:26:03.726047 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.726013 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-podinfo\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.726242 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.726053 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvlfm\" (UniqueName: \"kubernetes.io/projected/9e1312b0-1b48-42b9-9102-d8e6647f3467-kube-api-access-bvlfm\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.726242 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.726119 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-data\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.726242 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.726181 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-workload-certs\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.726242 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.726210 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1312b0-1b48-42b9-9102-d8e6647f3467-istiod-ca-cert\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.726448 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.726298 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-token\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.726448 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.726362 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-workload-socket\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.726448 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.726429 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-envoy\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.726603 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.726448 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-credential-socket\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.827888 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.827851 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-envoy\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.827888 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.827886 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-credential-socket\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.828125 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.827923 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-podinfo\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.828125 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.827940 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvlfm\" (UniqueName: \"kubernetes.io/projected/9e1312b0-1b48-42b9-9102-d8e6647f3467-kube-api-access-bvlfm\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.828125 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.827956 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-data\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.828125 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.827973 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-workload-certs\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.828125 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.827990 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1312b0-1b48-42b9-9102-d8e6647f3467-istiod-ca-cert\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.828376 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.828354 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-credential-socket\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.828436 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.828419 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-workload-certs\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.828491 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.828453 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-token\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.828584 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.828454 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-data\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.828584 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.828573 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-workload-socket\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.828853 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.828828 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-workload-socket\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.828959 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.828909 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1312b0-1b48-42b9-9102-d8e6647f3467-istiod-ca-cert\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.830255 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.830239 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-envoy\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.830631 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.830617 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-podinfo\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.836957 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.836933 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-token\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.837084 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.837066 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvlfm\" (UniqueName: \"kubernetes.io/projected/9e1312b0-1b48-42b9-9102-d8e6647f3467-kube-api-access-bvlfm\") pod \"e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:03.926291 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:03.926087 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:04.289653 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:04.289599 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj"] Sep 17 23:26:04.300018 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:26:04.299984 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1312b0_1b48_42b9_9102_d8e6647f3467.slice/crio-aecfc02d21230fb8c9979b528eed0053efb0eff9e6a4be661d04b59de685982b WatchSource:0}: Error finding container aecfc02d21230fb8c9979b528eed0053efb0eff9e6a4be661d04b59de685982b: Status 404 returned error can't find the container with id aecfc02d21230fb8c9979b528eed0053efb0eff9e6a4be661d04b59de685982b Sep 17 23:26:04.302903 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:04.302871 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:26:04.303000 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:04.302938 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:26:04.303000 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:04.302969 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:26:04.407347 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:04.407306 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" event={"ID":"9e1312b0-1b48-42b9-9102-d8e6647f3467","Type":"ContainerStarted","Data":"fe440aac91c53cc6fafb15b4c11b0de6280628e4133892edb58201616f99eb6d"} Sep 17 23:26:04.407518 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:04.407373 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" event={"ID":"9e1312b0-1b48-42b9-9102-d8e6647f3467","Type":"ContainerStarted","Data":"aecfc02d21230fb8c9979b528eed0053efb0eff9e6a4be661d04b59de685982b"} Sep 17 23:26:04.430182 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:04.430124 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" podStartSLOduration=1.430089357 podStartE2EDuration="1.430089357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:26:04.429105493 +0000 UTC m=+1203.363012627" watchObservedRunningTime="2026-09-17 23:26:04.430089357 +0000 UTC m=+1203.363996492" Sep 17 23:26:04.926968 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:04.926937 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:05.931552 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:05.931523 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:06.418588 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:06.418560 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:06.419355 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:06.419338 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:11.451226 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:11.451150 2582 generic.go:356] "Generic (PLEG): container finished" podID="00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" containerID="e417fda6cae8769cddde15d30f2ae166f3c04ec3ca59096394f490a5a5b50b7e" exitCode=0 Sep 17 23:26:11.451606 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:11.451221 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" event={"ID":"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7","Type":"ContainerDied","Data":"e417fda6cae8769cddde15d30f2ae166f3c04ec3ca59096394f490a5a5b50b7e"} Sep 17 23:26:11.451658 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:11.451644 2582 scope.go:117] "RemoveContainer" containerID="e417fda6cae8769cddde15d30f2ae166f3c04ec3ca59096394f490a5a5b50b7e" Sep 17 23:26:11.452809 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:11.452779 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:26:11.452900 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:11.452853 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:26:11.452900 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:11.452882 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:26:12.460411 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:12.460377 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" event={"ID":"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7","Type":"ContainerStarted","Data":"d382000d5fdfff6635e7d06a114b22b896f65ace6d20e97ad71181365fd87698"} Sep 17 23:26:13.304681 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:13.304641 2582 patch_prober.go:28] interesting pod/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 17 23:26:13.305010 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:13.304968 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" podUID="00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 17 23:26:13.314082 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:13.314060 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:26:14.048154 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:14.048119 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n"] Sep 17 23:26:14.051894 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:14.051878 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n" Sep 17 23:26:14.054357 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:14.054333 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-df969c31-dockercfg-pfht4\"" Sep 17 23:26:14.054511 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:14.054479 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-df969c31\"" Sep 17 23:26:14.063592 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:14.063570 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n"] Sep 17 23:26:14.134338 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:14.134305 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm74k\" (UniqueName: \"kubernetes.io/projected/07f4b9ca-b73e-42fe-8f71-9d73f5415635-kube-api-access-zm74k\") pod \"payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n\" (UID: \"07f4b9ca-b73e-42fe-8f71-9d73f5415635\") " pod="openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n" Sep 17 23:26:14.134692 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:14.134649 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f4b9ca-b73e-42fe-8f71-9d73f5415635-plugins-config-volume\") pod \"payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n\" (UID: \"07f4b9ca-b73e-42fe-8f71-9d73f5415635\") " pod="openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n" Sep 17 23:26:14.235823 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:14.235779 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f4b9ca-b73e-42fe-8f71-9d73f5415635-plugins-config-volume\") pod \"payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n\" (UID: \"07f4b9ca-b73e-42fe-8f71-9d73f5415635\") " pod="openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n" Sep 17 23:26:14.236021 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:14.235915 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm74k\" (UniqueName: \"kubernetes.io/projected/07f4b9ca-b73e-42fe-8f71-9d73f5415635-kube-api-access-zm74k\") pod \"payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n\" (UID: \"07f4b9ca-b73e-42fe-8f71-9d73f5415635\") " pod="openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n" Sep 17 23:26:14.236564 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:14.236540 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f4b9ca-b73e-42fe-8f71-9d73f5415635-plugins-config-volume\") pod \"payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n\" (UID: \"07f4b9ca-b73e-42fe-8f71-9d73f5415635\") " pod="openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n" Sep 17 23:26:14.246453 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:14.246435 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm74k\" (UniqueName: \"kubernetes.io/projected/07f4b9ca-b73e-42fe-8f71-9d73f5415635-kube-api-access-zm74k\") pod \"payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n\" (UID: \"07f4b9ca-b73e-42fe-8f71-9d73f5415635\") " pod="openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n" Sep 17 23:26:14.298800 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:14.298716 2582 patch_prober.go:28] interesting pod/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 17 23:26:14.298800 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:14.298779 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" podUID="00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 17 23:26:14.362300 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:14.362275 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n" Sep 17 23:26:14.656763 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:14.656728 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n"] Sep 17 23:26:14.658125 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:26:14.658101 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f4b9ca_b73e_42fe_8f71_9d73f5415635.slice/crio-25a91b72a73a2b9f2d2224f5e251f44e5af54842b8a2beb8eb74ec9fd6c4cbc6 WatchSource:0}: Error finding container 25a91b72a73a2b9f2d2224f5e251f44e5af54842b8a2beb8eb74ec9fd6c4cbc6: Status 404 returned error can't find the container with id 25a91b72a73a2b9f2d2224f5e251f44e5af54842b8a2beb8eb74ec9fd6c4cbc6 Sep 17 23:26:15.300316 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:15.300271 2582 patch_prober.go:28] interesting pod/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 17 23:26:15.300807 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:15.300338 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" podUID="00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 17 23:26:15.512442 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:15.512403 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n" event={"ID":"07f4b9ca-b73e-42fe-8f71-9d73f5415635","Type":"ContainerStarted","Data":"b7955af35bda94134fab4a32188d444279f488964eacebbd149eca937f86ebaf"} Sep 17 23:26:15.512654 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:15.512542 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n" Sep 17 23:26:15.512654 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:15.512570 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n" event={"ID":"07f4b9ca-b73e-42fe-8f71-9d73f5415635","Type":"ContainerStarted","Data":"25a91b72a73a2b9f2d2224f5e251f44e5af54842b8a2beb8eb74ec9fd6c4cbc6"} Sep 17 23:26:15.533133 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:15.533083 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n" podStartSLOduration=1.5330694089999999 podStartE2EDuration="1.533069409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:26:15.531570409 +0000 UTC m=+1214.465477542" watchObservedRunningTime="2026-09-17 23:26:15.533069409 +0000 UTC m=+1214.466976544" Sep 17 23:26:16.299482 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:16.299449 2582 patch_prober.go:28] interesting pod/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 17 23:26:16.299698 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:16.299526 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" podUID="00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 17 23:26:17.265881 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:17.265847 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:26:17.299060 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:17.299024 2582 patch_prober.go:28] interesting pod/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Sep 17 23:26:17.299223 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:17.299082 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" podUID="00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Sep 17 23:26:18.303554 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:18.303528 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:26:18.304625 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:18.304607 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:26:26.520226 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:26.520196 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n" Sep 17 23:26:32.624773 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:32.624735 2582 generic.go:356] "Generic (PLEG): container finished" podID="9e1312b0-1b48-42b9-9102-d8e6647f3467" containerID="fe440aac91c53cc6fafb15b4c11b0de6280628e4133892edb58201616f99eb6d" exitCode=0 Sep 17 23:26:32.625209 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:32.624809 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" event={"ID":"9e1312b0-1b48-42b9-9102-d8e6647f3467","Type":"ContainerDied","Data":"fe440aac91c53cc6fafb15b4c11b0de6280628e4133892edb58201616f99eb6d"} Sep 17 23:26:32.625276 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:32.625251 2582 scope.go:117] "RemoveContainer" containerID="fe440aac91c53cc6fafb15b4c11b0de6280628e4133892edb58201616f99eb6d" Sep 17 23:26:32.626331 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:32.626298 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:26:32.626461 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:32.626377 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:26:32.626461 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:32.626424 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:26:33.632490 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:33.632454 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" event={"ID":"9e1312b0-1b48-42b9-9102-d8e6647f3467","Type":"ContainerStarted","Data":"6bdd14ec7a7706f8d96be89912ca0f619563a952974f67dfae60cad910c68b8f"} Sep 17 23:26:33.926891 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:33.926798 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:33.928598 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:33.928551 2582 patch_prober.go:28] interesting pod/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 17 23:26:33.928722 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:33.928631 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" podUID="9e1312b0-1b48-42b9-9102-d8e6647f3467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 17 23:26:34.926934 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:34.926899 2582 patch_prober.go:28] interesting pod/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 17 23:26:34.927402 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:34.926986 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" podUID="9e1312b0-1b48-42b9-9102-d8e6647f3467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 17 23:26:35.533888 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:35.533853 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7f7059-nvzxd"] Sep 17 23:26:35.536690 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:35.536672 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7f7059-nvzxd" Sep 17 23:26:35.539045 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:35.539024 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:26:35.542995 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:35.542974 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7f7059-nvzxd"] Sep 17 23:26:35.646219 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:35.646183 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944a33e9-713a-4805-9804-2992731d48fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7f7059-nvzxd\" (UID: \"944a33e9-713a-4805-9804-2992731d48fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7f7059-nvzxd" Sep 17 23:26:35.746867 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:35.746833 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944a33e9-713a-4805-9804-2992731d48fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7f7059-nvzxd\" (UID: \"944a33e9-713a-4805-9804-2992731d48fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7f7059-nvzxd" Sep 17 23:26:35.747448 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:35.747429 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944a33e9-713a-4805-9804-2992731d48fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7f7059-nvzxd\" (UID: \"944a33e9-713a-4805-9804-2992731d48fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7f7059-nvzxd" Sep 17 23:26:35.848312 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:35.848234 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7f7059-nvzxd" Sep 17 23:26:35.927337 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:35.927043 2582 patch_prober.go:28] interesting pod/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 17 23:26:35.927337 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:35.927119 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" podUID="9e1312b0-1b48-42b9-9102-d8e6647f3467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 17 23:26:36.143417 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:36.143392 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7f7059-nvzxd"] Sep 17 23:26:36.144978 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:26:36.144950 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944a33e9_713a_4805_9804_2992731d48fe.slice/crio-3afdfbcd2a2d994ce73986fca35a3f12894c9cea40ee39898e85b9a65840c791 WatchSource:0}: Error finding container 3afdfbcd2a2d994ce73986fca35a3f12894c9cea40ee39898e85b9a65840c791: Status 404 returned error can't find the container with id 3afdfbcd2a2d994ce73986fca35a3f12894c9cea40ee39898e85b9a65840c791 Sep 17 23:26:36.418695 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:36.418598 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:36.652404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:36.652367 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7f7059-nvzxd" event={"ID":"944a33e9-713a-4805-9804-2992731d48fe","Type":"ContainerStarted","Data":"7101ed48e08d0feee0a3b9cb93c9ad8835720975e1c6cbb402c7f0f1b9809f17"} Sep 17 23:26:36.652578 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:36.652423 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7f7059-nvzxd" event={"ID":"944a33e9-713a-4805-9804-2992731d48fe","Type":"ContainerStarted","Data":"3afdfbcd2a2d994ce73986fca35a3f12894c9cea40ee39898e85b9a65840c791"} Sep 17 23:26:36.926864 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:36.926834 2582 patch_prober.go:28] interesting pod/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 17 23:26:36.927027 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:36.926884 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" podUID="9e1312b0-1b48-42b9-9102-d8e6647f3467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 17 23:26:37.662080 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:37.662036 2582 generic.go:356] "Generic (PLEG): container finished" podID="944a33e9-713a-4805-9804-2992731d48fe" containerID="7101ed48e08d0feee0a3b9cb93c9ad8835720975e1c6cbb402c7f0f1b9809f17" exitCode=0 Sep 17 23:26:37.662523 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:37.662163 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7f7059-nvzxd" event={"ID":"944a33e9-713a-4805-9804-2992731d48fe","Type":"ContainerDied","Data":"7101ed48e08d0feee0a3b9cb93c9ad8835720975e1c6cbb402c7f0f1b9809f17"} Sep 17 23:26:37.926462 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:37.926357 2582 patch_prober.go:28] interesting pod/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 17 23:26:37.926462 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:37.926428 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" podUID="9e1312b0-1b48-42b9-9102-d8e6647f3467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 17 23:26:38.836424 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:38.836400 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7f7059-nvzxd" Sep 17 23:26:38.889435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:38.888840 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944a33e9-713a-4805-9804-2992731d48fe-maas-api-service-ca\") pod \"944a33e9-713a-4805-9804-2992731d48fe\" (UID: \"944a33e9-713a-4805-9804-2992731d48fe\") " Sep 17 23:26:38.889796 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:38.889771 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944a33e9-713a-4805-9804-2992731d48fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "944a33e9-713a-4805-9804-2992731d48fe" (UID: "944a33e9-713a-4805-9804-2992731d48fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:26:38.927047 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:38.926977 2582 patch_prober.go:28] interesting pod/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 17 23:26:38.927047 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:38.927023 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" podUID="9e1312b0-1b48-42b9-9102-d8e6647f3467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 17 23:26:38.990106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:38.990077 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/944a33e9-713a-4805-9804-2992731d48fe-maas-api-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:26:39.676144 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:39.676051 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7f7059-nvzxd" event={"ID":"944a33e9-713a-4805-9804-2992731d48fe","Type":"ContainerDied","Data":"3afdfbcd2a2d994ce73986fca35a3f12894c9cea40ee39898e85b9a65840c791"} Sep 17 23:26:39.676144 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:39.676076 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7f7059-nvzxd" Sep 17 23:26:39.676144 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:39.676110 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3afdfbcd2a2d994ce73986fca35a3f12894c9cea40ee39898e85b9a65840c791" Sep 17 23:26:39.930627 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:39.930550 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:39.931554 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:39.931534 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:26:46.555809 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:46.555737 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7f7059-nvzxd"] Sep 17 23:26:46.559306 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:46.559279 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7f7059-nvzxd"] Sep 17 23:26:47.296313 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:47.296282 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="944a33e9-713a-4805-9804-2992731d48fe" path="/var/lib/kubelet/pods/944a33e9-713a-4805-9804-2992731d48fe/volumes" Sep 17 23:26:53.922042 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:53.922006 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh"] Sep 17 23:26:53.922438 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:53.922315 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" podUID="00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" containerName="istio-proxy" containerID="cri-o://d382000d5fdfff6635e7d06a114b22b896f65ace6d20e97ad71181365fd87698" gracePeriod=30 Sep 17 23:26:58.141941 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:58.141904 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n"] Sep 17 23:26:58.142489 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:58.142154 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n" podUID="07f4b9ca-b73e-42fe-8f71-9d73f5415635" containerName="payload-processing" containerID="cri-o://b7955af35bda94134fab4a32188d444279f488964eacebbd149eca937f86ebaf" gracePeriod=30 Sep 17 23:26:58.397340 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:58.397278 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n" Sep 17 23:26:58.581026 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:58.580980 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f4b9ca-b73e-42fe-8f71-9d73f5415635-plugins-config-volume\") pod \"07f4b9ca-b73e-42fe-8f71-9d73f5415635\" (UID: \"07f4b9ca-b73e-42fe-8f71-9d73f5415635\") " Sep 17 23:26:58.581232 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:58.581146 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm74k\" (UniqueName: \"kubernetes.io/projected/07f4b9ca-b73e-42fe-8f71-9d73f5415635-kube-api-access-zm74k\") pod \"07f4b9ca-b73e-42fe-8f71-9d73f5415635\" (UID: \"07f4b9ca-b73e-42fe-8f71-9d73f5415635\") " Sep 17 23:26:58.581368 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:58.581344 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f4b9ca-b73e-42fe-8f71-9d73f5415635-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "07f4b9ca-b73e-42fe-8f71-9d73f5415635" (UID: "07f4b9ca-b73e-42fe-8f71-9d73f5415635"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:26:58.581488 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:58.581471 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f4b9ca-b73e-42fe-8f71-9d73f5415635-plugins-config-volume\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:26:58.583389 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:58.583363 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f4b9ca-b73e-42fe-8f71-9d73f5415635-kube-api-access-zm74k" (OuterVolumeSpecName: "kube-api-access-zm74k") pod "07f4b9ca-b73e-42fe-8f71-9d73f5415635" (UID: "07f4b9ca-b73e-42fe-8f71-9d73f5415635"). InnerVolumeSpecName "kube-api-access-zm74k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:26:58.682427 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:58.682349 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm74k\" (UniqueName: \"kubernetes.io/projected/07f4b9ca-b73e-42fe-8f71-9d73f5415635-kube-api-access-zm74k\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:26:58.800161 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:58.800125 2582 generic.go:356] "Generic (PLEG): container finished" podID="07f4b9ca-b73e-42fe-8f71-9d73f5415635" containerID="b7955af35bda94134fab4a32188d444279f488964eacebbd149eca937f86ebaf" exitCode=0 Sep 17 23:26:58.800316 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:58.800182 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n" Sep 17 23:26:58.800316 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:58.800202 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n" event={"ID":"07f4b9ca-b73e-42fe-8f71-9d73f5415635","Type":"ContainerDied","Data":"b7955af35bda94134fab4a32188d444279f488964eacebbd149eca937f86ebaf"} Sep 17 23:26:58.800316 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:58.800275 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n" event={"ID":"07f4b9ca-b73e-42fe-8f71-9d73f5415635","Type":"ContainerDied","Data":"25a91b72a73a2b9f2d2224f5e251f44e5af54842b8a2beb8eb74ec9fd6c4cbc6"} Sep 17 23:26:58.800316 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:58.800298 2582 scope.go:117] "RemoveContainer" containerID="b7955af35bda94134fab4a32188d444279f488964eacebbd149eca937f86ebaf" Sep 17 23:26:58.809522 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:58.809476 2582 scope.go:117] "RemoveContainer" containerID="b7955af35bda94134fab4a32188d444279f488964eacebbd149eca937f86ebaf" Sep 17 23:26:58.809798 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:26:58.809782 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7955af35bda94134fab4a32188d444279f488964eacebbd149eca937f86ebaf\": container with ID starting with b7955af35bda94134fab4a32188d444279f488964eacebbd149eca937f86ebaf not found: ID does not exist" containerID="b7955af35bda94134fab4a32188d444279f488964eacebbd149eca937f86ebaf" Sep 17 23:26:58.809847 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:58.809805 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7955af35bda94134fab4a32188d444279f488964eacebbd149eca937f86ebaf"} err="failed to get container status \"b7955af35bda94134fab4a32188d444279f488964eacebbd149eca937f86ebaf\": rpc error: code = NotFound desc = could not find container \"b7955af35bda94134fab4a32188d444279f488964eacebbd149eca937f86ebaf\": container with ID starting with b7955af35bda94134fab4a32188d444279f488964eacebbd149eca937f86ebaf not found: ID does not exist" Sep 17 23:26:58.824026 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:58.824007 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n"] Sep 17 23:26:58.830815 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:58.830796 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df969c31-f668fbf5c-rhk8n"] Sep 17 23:26:59.168452 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.168431 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:26:59.186446 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.186384 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-data\") pod \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " Sep 17 23:26:59.186593 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.186452 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-envoy\") pod \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " Sep 17 23:26:59.186593 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.186512 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-credential-socket\") pod \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " Sep 17 23:26:59.186593 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.186562 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-workload-socket\") pod \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " Sep 17 23:26:59.186759 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.186613 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sq2m\" (UniqueName: \"kubernetes.io/projected/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-kube-api-access-7sq2m\") pod \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " Sep 17 23:26:59.186759 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.186681 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-workload-certs\") pod \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " Sep 17 23:26:59.186759 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.186689 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-data" (OuterVolumeSpecName: "istio-data") pod "00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" (UID: "00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:26:59.186759 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.186707 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-token\") pod \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " Sep 17 23:26:59.186969 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.186857 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" (UID: "00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:26:59.186969 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.186871 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" (UID: "00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:26:59.186969 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.186929 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-podinfo\") pod \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " Sep 17 23:26:59.187116 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.186976 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" (UID: "00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:26:59.187116 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.187024 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istiod-ca-cert\") pod \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\" (UID: \"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7\") " Sep 17 23:26:59.187630 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.187602 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" (UID: "00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:26:59.187945 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.187894 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-credential-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:26:59.187945 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.187940 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-workload-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:26:59.188109 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.187955 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-workload-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:26:59.188109 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.187970 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istiod-ca-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:26:59.188109 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.188023 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-data\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:26:59.190905 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.190694 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-token" (OuterVolumeSpecName: "istio-token") pod "00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" (UID: "00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:26:59.191516 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.191474 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" (UID: "00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:26:59.191636 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.191570 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" (UID: "00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:26:59.191636 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.191569 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-kube-api-access-7sq2m" (OuterVolumeSpecName: "kube-api-access-7sq2m") pod "00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" (UID: "00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7"). InnerVolumeSpecName "kube-api-access-7sq2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:26:59.289074 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.289048 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-envoy\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:26:59.289212 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.289076 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sq2m\" (UniqueName: \"kubernetes.io/projected/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-kube-api-access-7sq2m\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:26:59.289212 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.289092 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:26:59.289212 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.289101 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7-istio-podinfo\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:26:59.294450 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.294426 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07f4b9ca-b73e-42fe-8f71-9d73f5415635" path="/var/lib/kubelet/pods/07f4b9ca-b73e-42fe-8f71-9d73f5415635/volumes" Sep 17 23:26:59.808236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.808200 2582 generic.go:356] "Generic (PLEG): container finished" podID="00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" containerID="d382000d5fdfff6635e7d06a114b22b896f65ace6d20e97ad71181365fd87698" exitCode=0 Sep 17 23:26:59.808395 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.808241 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" event={"ID":"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7","Type":"ContainerDied","Data":"d382000d5fdfff6635e7d06a114b22b896f65ace6d20e97ad71181365fd87698"} Sep 17 23:26:59.808395 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.808254 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" Sep 17 23:26:59.808395 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.808291 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh" event={"ID":"00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7","Type":"ContainerDied","Data":"3e8be1caa81f971cc5db06194da38856ec704bb5c72ebba8fa443961904ed47b"} Sep 17 23:26:59.808395 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.808321 2582 scope.go:117] "RemoveContainer" containerID="d382000d5fdfff6635e7d06a114b22b896f65ace6d20e97ad71181365fd87698" Sep 17 23:26:59.817649 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.817619 2582 scope.go:117] "RemoveContainer" containerID="e417fda6cae8769cddde15d30f2ae166f3c04ec3ca59096394f490a5a5b50b7e" Sep 17 23:26:59.825362 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.825335 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh"] Sep 17 23:26:59.826031 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.826011 2582 scope.go:117] "RemoveContainer" containerID="d382000d5fdfff6635e7d06a114b22b896f65ace6d20e97ad71181365fd87698" Sep 17 23:26:59.826514 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:26:59.826390 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d382000d5fdfff6635e7d06a114b22b896f65ace6d20e97ad71181365fd87698\": container with ID starting with d382000d5fdfff6635e7d06a114b22b896f65ace6d20e97ad71181365fd87698 not found: ID does not exist" containerID="d382000d5fdfff6635e7d06a114b22b896f65ace6d20e97ad71181365fd87698" Sep 17 23:26:59.826514 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.826431 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d382000d5fdfff6635e7d06a114b22b896f65ace6d20e97ad71181365fd87698"} err="failed to get container status \"d382000d5fdfff6635e7d06a114b22b896f65ace6d20e97ad71181365fd87698\": rpc error: code = NotFound desc = could not find container \"d382000d5fdfff6635e7d06a114b22b896f65ace6d20e97ad71181365fd87698\": container with ID starting with d382000d5fdfff6635e7d06a114b22b896f65ace6d20e97ad71181365fd87698 not found: ID does not exist" Sep 17 23:26:59.826514 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.826453 2582 scope.go:117] "RemoveContainer" containerID="e417fda6cae8769cddde15d30f2ae166f3c04ec3ca59096394f490a5a5b50b7e" Sep 17 23:26:59.826848 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:26:59.826819 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e417fda6cae8769cddde15d30f2ae166f3c04ec3ca59096394f490a5a5b50b7e\": container with ID starting with e417fda6cae8769cddde15d30f2ae166f3c04ec3ca59096394f490a5a5b50b7e not found: ID does not exist" containerID="e417fda6cae8769cddde15d30f2ae166f3c04ec3ca59096394f490a5a5b50b7e" Sep 17 23:26:59.826931 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.826857 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e417fda6cae8769cddde15d30f2ae166f3c04ec3ca59096394f490a5a5b50b7e"} err="failed to get container status \"e417fda6cae8769cddde15d30f2ae166f3c04ec3ca59096394f490a5a5b50b7e\": rpc error: code = NotFound desc = could not find container \"e417fda6cae8769cddde15d30f2ae166f3c04ec3ca59096394f490a5a5b50b7e\": container with ID starting with e417fda6cae8769cddde15d30f2ae166f3c04ec3ca59096394f490a5a5b50b7e not found: ID does not exist" Sep 17 23:26:59.828842 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:26:59.828822 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-7f7059-openshift-default-79566fc99b-5cvsh"] Sep 17 23:27:01.293889 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:01.293853 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" path="/var/lib/kubelet/pods/00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7/volumes" Sep 17 23:27:04.495079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:04.495049 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj"] Sep 17 23:27:04.495482 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:04.495249 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" podUID="9e1312b0-1b48-42b9-9102-d8e6647f3467" containerName="istio-proxy" containerID="cri-o://6bdd14ec7a7706f8d96be89912ca0f619563a952974f67dfae60cad910c68b8f" gracePeriod=30 Sep 17 23:27:07.887411 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:07.887375 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s"] Sep 17 23:27:07.887844 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:07.887654 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s" podUID="5866863a-2c2f-4436-92b7-17278e4f05fb" containerName="payload-processing" containerID="cri-o://3bb2ae575cfb3578b647a17ede0aed4bb438a809bf80eb8c9e0540bb949f8d0f" gracePeriod=30 Sep 17 23:27:08.154156 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:08.154127 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s" Sep 17 23:27:08.272779 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:08.272134 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2pzz\" (UniqueName: \"kubernetes.io/projected/5866863a-2c2f-4436-92b7-17278e4f05fb-kube-api-access-s2pzz\") pod \"5866863a-2c2f-4436-92b7-17278e4f05fb\" (UID: \"5866863a-2c2f-4436-92b7-17278e4f05fb\") " Sep 17 23:27:08.272779 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:08.272200 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5866863a-2c2f-4436-92b7-17278e4f05fb-plugins-config-volume\") pod \"5866863a-2c2f-4436-92b7-17278e4f05fb\" (UID: \"5866863a-2c2f-4436-92b7-17278e4f05fb\") " Sep 17 23:27:08.273208 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:08.273174 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5866863a-2c2f-4436-92b7-17278e4f05fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5866863a-2c2f-4436-92b7-17278e4f05fb" (UID: "5866863a-2c2f-4436-92b7-17278e4f05fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:27:08.276021 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:08.275995 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5866863a-2c2f-4436-92b7-17278e4f05fb-kube-api-access-s2pzz" (OuterVolumeSpecName: "kube-api-access-s2pzz") pod "5866863a-2c2f-4436-92b7-17278e4f05fb" (UID: "5866863a-2c2f-4436-92b7-17278e4f05fb"). InnerVolumeSpecName "kube-api-access-s2pzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:27:08.374154 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:08.374114 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5866863a-2c2f-4436-92b7-17278e4f05fb-plugins-config-volume\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:08.374154 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:08.374150 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2pzz\" (UniqueName: \"kubernetes.io/projected/5866863a-2c2f-4436-92b7-17278e4f05fb-kube-api-access-s2pzz\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:08.863212 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:08.863179 2582 generic.go:356] "Generic (PLEG): container finished" podID="5866863a-2c2f-4436-92b7-17278e4f05fb" containerID="3bb2ae575cfb3578b647a17ede0aed4bb438a809bf80eb8c9e0540bb949f8d0f" exitCode=0 Sep 17 23:27:08.863377 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:08.863225 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s" Sep 17 23:27:08.863377 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:08.863263 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s" event={"ID":"5866863a-2c2f-4436-92b7-17278e4f05fb","Type":"ContainerDied","Data":"3bb2ae575cfb3578b647a17ede0aed4bb438a809bf80eb8c9e0540bb949f8d0f"} Sep 17 23:27:08.863377 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:08.863317 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s" event={"ID":"5866863a-2c2f-4436-92b7-17278e4f05fb","Type":"ContainerDied","Data":"e99366963ff31bf78a8422e7dd4f26ac89e1a3932bcd6c1c49b6e1a557f88e68"} Sep 17 23:27:08.863377 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:08.863348 2582 scope.go:117] "RemoveContainer" containerID="3bb2ae575cfb3578b647a17ede0aed4bb438a809bf80eb8c9e0540bb949f8d0f" Sep 17 23:27:08.873078 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:08.873059 2582 scope.go:117] "RemoveContainer" containerID="3bb2ae575cfb3578b647a17ede0aed4bb438a809bf80eb8c9e0540bb949f8d0f" Sep 17 23:27:08.873427 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:27:08.873397 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb2ae575cfb3578b647a17ede0aed4bb438a809bf80eb8c9e0540bb949f8d0f\": container with ID starting with 3bb2ae575cfb3578b647a17ede0aed4bb438a809bf80eb8c9e0540bb949f8d0f not found: ID does not exist" containerID="3bb2ae575cfb3578b647a17ede0aed4bb438a809bf80eb8c9e0540bb949f8d0f" Sep 17 23:27:08.873534 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:08.873425 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb2ae575cfb3578b647a17ede0aed4bb438a809bf80eb8c9e0540bb949f8d0f"} err="failed to get container status \"3bb2ae575cfb3578b647a17ede0aed4bb438a809bf80eb8c9e0540bb949f8d0f\": rpc error: code = NotFound desc = could not find container \"3bb2ae575cfb3578b647a17ede0aed4bb438a809bf80eb8c9e0540bb949f8d0f\": container with ID starting with 3bb2ae575cfb3578b647a17ede0aed4bb438a809bf80eb8c9e0540bb949f8d0f not found: ID does not exist" Sep 17 23:27:08.889423 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:08.889400 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s"] Sep 17 23:27:08.893532 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:08.893493 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0c363f-78c58bb5bc-6kr6s"] Sep 17 23:27:09.294478 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.294439 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5866863a-2c2f-4436-92b7-17278e4f05fb" path="/var/lib/kubelet/pods/5866863a-2c2f-4436-92b7-17278e4f05fb/volumes" Sep 17 23:27:09.746444 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.746423 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:27:09.787909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.787882 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-credential-socket\") pod \"9e1312b0-1b48-42b9-9102-d8e6647f3467\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " Sep 17 23:27:09.788055 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.787919 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-podinfo\") pod \"9e1312b0-1b48-42b9-9102-d8e6647f3467\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " Sep 17 23:27:09.788055 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.787972 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvlfm\" (UniqueName: \"kubernetes.io/projected/9e1312b0-1b48-42b9-9102-d8e6647f3467-kube-api-access-bvlfm\") pod \"9e1312b0-1b48-42b9-9102-d8e6647f3467\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " Sep 17 23:27:09.788055 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.787999 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-data\") pod \"9e1312b0-1b48-42b9-9102-d8e6647f3467\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " Sep 17 23:27:09.788055 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.788014 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-workload-certs\") pod \"9e1312b0-1b48-42b9-9102-d8e6647f3467\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " Sep 17 23:27:09.788269 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.788092 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-workload-socket\") pod \"9e1312b0-1b48-42b9-9102-d8e6647f3467\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " Sep 17 23:27:09.788269 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.788128 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-envoy\") pod \"9e1312b0-1b48-42b9-9102-d8e6647f3467\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " Sep 17 23:27:09.788269 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.788129 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9e1312b0-1b48-42b9-9102-d8e6647f3467" (UID: "9e1312b0-1b48-42b9-9102-d8e6647f3467"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:27:09.788269 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.788152 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1312b0-1b48-42b9-9102-d8e6647f3467-istiod-ca-cert\") pod \"9e1312b0-1b48-42b9-9102-d8e6647f3467\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " Sep 17 23:27:09.788269 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.788176 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-token\") pod \"9e1312b0-1b48-42b9-9102-d8e6647f3467\" (UID: \"9e1312b0-1b48-42b9-9102-d8e6647f3467\") " Sep 17 23:27:09.788546 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.788278 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9e1312b0-1b48-42b9-9102-d8e6647f3467" (UID: "9e1312b0-1b48-42b9-9102-d8e6647f3467"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:27:09.788546 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.788290 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9e1312b0-1b48-42b9-9102-d8e6647f3467" (UID: "9e1312b0-1b48-42b9-9102-d8e6647f3467"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:27:09.788546 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.788303 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-data" (OuterVolumeSpecName: "istio-data") pod "9e1312b0-1b48-42b9-9102-d8e6647f3467" (UID: "9e1312b0-1b48-42b9-9102-d8e6647f3467"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:27:09.788733 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.788566 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1312b0-1b48-42b9-9102-d8e6647f3467-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9e1312b0-1b48-42b9-9102-d8e6647f3467" (UID: "9e1312b0-1b48-42b9-9102-d8e6647f3467"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:27:09.788733 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.788620 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-credential-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:09.788733 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.788640 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-data\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:09.788733 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.788653 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-workload-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:09.788733 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.788667 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-workload-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:09.790208 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.790182 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1312b0-1b48-42b9-9102-d8e6647f3467-kube-api-access-bvlfm" (OuterVolumeSpecName: "kube-api-access-bvlfm") pod "9e1312b0-1b48-42b9-9102-d8e6647f3467" (UID: "9e1312b0-1b48-42b9-9102-d8e6647f3467"). InnerVolumeSpecName "kube-api-access-bvlfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:27:09.790334 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.790311 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9e1312b0-1b48-42b9-9102-d8e6647f3467" (UID: "9e1312b0-1b48-42b9-9102-d8e6647f3467"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:27:09.790611 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.790594 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9e1312b0-1b48-42b9-9102-d8e6647f3467" (UID: "9e1312b0-1b48-42b9-9102-d8e6647f3467"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:27:09.790686 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.790604 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-token" (OuterVolumeSpecName: "istio-token") pod "9e1312b0-1b48-42b9-9102-d8e6647f3467" (UID: "9e1312b0-1b48-42b9-9102-d8e6647f3467"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:27:09.871999 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.871919 2582 generic.go:356] "Generic (PLEG): container finished" podID="9e1312b0-1b48-42b9-9102-d8e6647f3467" containerID="6bdd14ec7a7706f8d96be89912ca0f619563a952974f67dfae60cad910c68b8f" exitCode=0 Sep 17 23:27:09.871999 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.871969 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" Sep 17 23:27:09.872183 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.872003 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" event={"ID":"9e1312b0-1b48-42b9-9102-d8e6647f3467","Type":"ContainerDied","Data":"6bdd14ec7a7706f8d96be89912ca0f619563a952974f67dfae60cad910c68b8f"} Sep 17 23:27:09.872183 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.872074 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj" event={"ID":"9e1312b0-1b48-42b9-9102-d8e6647f3467","Type":"ContainerDied","Data":"aecfc02d21230fb8c9979b528eed0053efb0eff9e6a4be661d04b59de685982b"} Sep 17 23:27:09.872183 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.872105 2582 scope.go:117] "RemoveContainer" containerID="6bdd14ec7a7706f8d96be89912ca0f619563a952974f67dfae60cad910c68b8f" Sep 17 23:27:09.882456 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.882437 2582 scope.go:117] "RemoveContainer" containerID="fe440aac91c53cc6fafb15b4c11b0de6280628e4133892edb58201616f99eb6d" Sep 17 23:27:09.890291 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.890267 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-podinfo\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:09.890557 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.890298 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvlfm\" (UniqueName: \"kubernetes.io/projected/9e1312b0-1b48-42b9-9102-d8e6647f3467-kube-api-access-bvlfm\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:09.890557 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.890313 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-envoy\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:09.890557 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.890330 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1312b0-1b48-42b9-9102-d8e6647f3467-istiod-ca-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:09.890557 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.890345 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1312b0-1b48-42b9-9102-d8e6647f3467-istio-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:09.890749 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.890734 2582 scope.go:117] "RemoveContainer" containerID="6bdd14ec7a7706f8d96be89912ca0f619563a952974f67dfae60cad910c68b8f" Sep 17 23:27:09.891000 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:27:09.890983 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bdd14ec7a7706f8d96be89912ca0f619563a952974f67dfae60cad910c68b8f\": container with ID starting with 6bdd14ec7a7706f8d96be89912ca0f619563a952974f67dfae60cad910c68b8f not found: ID does not exist" containerID="6bdd14ec7a7706f8d96be89912ca0f619563a952974f67dfae60cad910c68b8f" Sep 17 23:27:09.891035 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.891007 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bdd14ec7a7706f8d96be89912ca0f619563a952974f67dfae60cad910c68b8f"} err="failed to get container status \"6bdd14ec7a7706f8d96be89912ca0f619563a952974f67dfae60cad910c68b8f\": rpc error: code = NotFound desc = could not find container \"6bdd14ec7a7706f8d96be89912ca0f619563a952974f67dfae60cad910c68b8f\": container with ID starting with 6bdd14ec7a7706f8d96be89912ca0f619563a952974f67dfae60cad910c68b8f not found: ID does not exist" Sep 17 23:27:09.891035 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.891023 2582 scope.go:117] "RemoveContainer" containerID="fe440aac91c53cc6fafb15b4c11b0de6280628e4133892edb58201616f99eb6d" Sep 17 23:27:09.891288 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:27:09.891269 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe440aac91c53cc6fafb15b4c11b0de6280628e4133892edb58201616f99eb6d\": container with ID starting with fe440aac91c53cc6fafb15b4c11b0de6280628e4133892edb58201616f99eb6d not found: ID does not exist" containerID="fe440aac91c53cc6fafb15b4c11b0de6280628e4133892edb58201616f99eb6d" Sep 17 23:27:09.891333 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.891293 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe440aac91c53cc6fafb15b4c11b0de6280628e4133892edb58201616f99eb6d"} err="failed to get container status \"fe440aac91c53cc6fafb15b4c11b0de6280628e4133892edb58201616f99eb6d\": rpc error: code = NotFound desc = could not find container \"fe440aac91c53cc6fafb15b4c11b0de6280628e4133892edb58201616f99eb6d\": container with ID starting with fe440aac91c53cc6fafb15b4c11b0de6280628e4133892edb58201616f99eb6d not found: ID does not exist" Sep 17 23:27:09.897713 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.897693 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj"] Sep 17 23:27:09.903149 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:09.903126 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-df969c31-openshift-default-6b9ccc96cb-8kgjj"] Sep 17 23:27:11.293794 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:11.293763 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e1312b0-1b48-42b9-9102-d8e6647f3467" path="/var/lib/kubelet/pods/9e1312b0-1b48-42b9-9102-d8e6647f3467/volumes" Sep 17 23:27:14.379865 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.379832 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx"] Sep 17 23:27:14.380273 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.380119 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" podUID="8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" containerName="istio-proxy" containerID="cri-o://7d55c243fd72a2c6c6fc444195cf96d813197406bbd5fec4b4a70a77cae06f86" gracePeriod=30 Sep 17 23:27:14.728627 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.728599 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-307a7b-qp4l9"] Sep 17 23:27:14.729004 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.728991 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07f4b9ca-b73e-42fe-8f71-9d73f5415635" containerName="payload-processing" Sep 17 23:27:14.729052 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.729006 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="07f4b9ca-b73e-42fe-8f71-9d73f5415635" containerName="payload-processing" Sep 17 23:27:14.729052 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.729016 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e1312b0-1b48-42b9-9102-d8e6647f3467" containerName="istio-proxy" Sep 17 23:27:14.729052 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.729022 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e1312b0-1b48-42b9-9102-d8e6647f3467" containerName="istio-proxy" Sep 17 23:27:14.729052 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.729032 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" containerName="istio-proxy" Sep 17 23:27:14.729052 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.729037 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" containerName="istio-proxy" Sep 17 23:27:14.729052 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.729044 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" containerName="istio-proxy" Sep 17 23:27:14.729052 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.729049 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" containerName="istio-proxy" Sep 17 23:27:14.729254 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.729067 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5866863a-2c2f-4436-92b7-17278e4f05fb" containerName="payload-processing" Sep 17 23:27:14.729254 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.729073 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5866863a-2c2f-4436-92b7-17278e4f05fb" containerName="payload-processing" Sep 17 23:27:14.729254 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.729080 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="944a33e9-713a-4805-9804-2992731d48fe" containerName="revoke-keys" Sep 17 23:27:14.729254 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.729085 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="944a33e9-713a-4805-9804-2992731d48fe" containerName="revoke-keys" Sep 17 23:27:14.729254 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.729152 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5866863a-2c2f-4436-92b7-17278e4f05fb" containerName="payload-processing" Sep 17 23:27:14.729254 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.729161 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e1312b0-1b48-42b9-9102-d8e6647f3467" containerName="istio-proxy" Sep 17 23:27:14.729254 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.729167 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="07f4b9ca-b73e-42fe-8f71-9d73f5415635" containerName="payload-processing" Sep 17 23:27:14.729254 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.729175 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" containerName="istio-proxy" Sep 17 23:27:14.729254 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.729180 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="00bd3695-d1d6-4ea5-a5bc-9e27ed7f35f7" containerName="istio-proxy" Sep 17 23:27:14.729254 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.729187 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="944a33e9-713a-4805-9804-2992731d48fe" containerName="revoke-keys" Sep 17 23:27:14.733539 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.733520 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-307a7b-qp4l9" Sep 17 23:27:14.735707 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.735685 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:27:14.741664 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.741644 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-307a7b-qp4l9"] Sep 17 23:27:14.840349 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.840315 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33a30f1-8691-4176-94f1-ebf02338b277-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-307a7b-qp4l9\" (UID: \"d33a30f1-8691-4176-94f1-ebf02338b277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-307a7b-qp4l9" Sep 17 23:27:14.941468 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.941437 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33a30f1-8691-4176-94f1-ebf02338b277-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-307a7b-qp4l9\" (UID: \"d33a30f1-8691-4176-94f1-ebf02338b277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-307a7b-qp4l9" Sep 17 23:27:14.942021 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:14.942001 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33a30f1-8691-4176-94f1-ebf02338b277-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-307a7b-qp4l9\" (UID: \"d33a30f1-8691-4176-94f1-ebf02338b277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-307a7b-qp4l9" Sep 17 23:27:15.044673 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:15.044577 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-307a7b-qp4l9" Sep 17 23:27:15.320223 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:15.317905 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-307a7b-qp4l9"] Sep 17 23:27:15.321110 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:27:15.321077 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33a30f1_8691_4176_94f1_ebf02338b277.slice/crio-2e4a65b2023ccb470b136a1d133cf74c2a668bf64586ba6b4a297628ca2ac44f WatchSource:0}: Error finding container 2e4a65b2023ccb470b136a1d133cf74c2a668bf64586ba6b4a297628ca2ac44f: Status 404 returned error can't find the container with id 2e4a65b2023ccb470b136a1d133cf74c2a668bf64586ba6b4a297628ca2ac44f Sep 17 23:27:15.911299 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:15.911211 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-307a7b-qp4l9" event={"ID":"d33a30f1-8691-4176-94f1-ebf02338b277","Type":"ContainerStarted","Data":"97cd8986b9c01c48f04d13219631d424f0cbf637427e3855e278e69e9568bf02"} Sep 17 23:27:15.911299 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:15.911265 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-307a7b-qp4l9" event={"ID":"d33a30f1-8691-4176-94f1-ebf02338b277","Type":"ContainerStarted","Data":"2e4a65b2023ccb470b136a1d133cf74c2a668bf64586ba6b4a297628ca2ac44f"} Sep 17 23:27:16.918954 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:16.918916 2582 generic.go:356] "Generic (PLEG): container finished" podID="d33a30f1-8691-4176-94f1-ebf02338b277" containerID="97cd8986b9c01c48f04d13219631d424f0cbf637427e3855e278e69e9568bf02" exitCode=0 Sep 17 23:27:16.919348 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:16.919037 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-307a7b-qp4l9" event={"ID":"d33a30f1-8691-4176-94f1-ebf02338b277","Type":"ContainerDied","Data":"97cd8986b9c01c48f04d13219631d424f0cbf637427e3855e278e69e9568bf02"} Sep 17 23:27:18.054315 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.054286 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-307a7b-qp4l9" Sep 17 23:27:18.171749 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.171723 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33a30f1-8691-4176-94f1-ebf02338b277-maas-api-service-ca\") pod \"d33a30f1-8691-4176-94f1-ebf02338b277\" (UID: \"d33a30f1-8691-4176-94f1-ebf02338b277\") " Sep 17 23:27:18.172133 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.172109 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d33a30f1-8691-4176-94f1-ebf02338b277-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d33a30f1-8691-4176-94f1-ebf02338b277" (UID: "d33a30f1-8691-4176-94f1-ebf02338b277"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:27:18.273284 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.273252 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33a30f1-8691-4176-94f1-ebf02338b277-maas-api-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:18.512255 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.512223 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc"] Sep 17 23:27:18.512668 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.512655 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d33a30f1-8691-4176-94f1-ebf02338b277" containerName="revoke-keys" Sep 17 23:27:18.512726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.512669 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d33a30f1-8691-4176-94f1-ebf02338b277" containerName="revoke-keys" Sep 17 23:27:18.512726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.512688 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e1312b0-1b48-42b9-9102-d8e6647f3467" containerName="istio-proxy" Sep 17 23:27:18.512726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.512694 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e1312b0-1b48-42b9-9102-d8e6647f3467" containerName="istio-proxy" Sep 17 23:27:18.512837 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.512761 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d33a30f1-8691-4176-94f1-ebf02338b277" containerName="revoke-keys" Sep 17 23:27:18.512837 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.512769 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e1312b0-1b48-42b9-9102-d8e6647f3467" containerName="istio-proxy" Sep 17 23:27:18.515857 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.515825 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.518164 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.518146 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-d69250-openshift-default-dockercfg-dqdmg\"" Sep 17 23:27:18.528041 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.527986 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc"] Sep 17 23:27:18.576230 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.576198 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-workload-socket\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.576341 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.576234 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-workload-certs\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.576341 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.576260 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9vlv\" (UniqueName: \"kubernetes.io/projected/64b1a682-e6fa-460e-a462-14827fb346a4-kube-api-access-x9vlv\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.576453 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.576401 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-istio-data\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.576453 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.576444 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64b1a682-e6fa-460e-a462-14827fb346a4-istio-token\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.576549 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.576528 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-credential-socket\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.576601 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.576588 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64b1a682-e6fa-460e-a462-14827fb346a4-istio-podinfo\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.576653 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.576628 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-istio-envoy\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.576653 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.576648 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64b1a682-e6fa-460e-a462-14827fb346a4-istiod-ca-cert\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.678089 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.678055 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vlv\" (UniqueName: \"kubernetes.io/projected/64b1a682-e6fa-460e-a462-14827fb346a4-kube-api-access-x9vlv\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.678247 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.678128 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-istio-data\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.678247 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.678156 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64b1a682-e6fa-460e-a462-14827fb346a4-istio-token\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.678247 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.678193 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-credential-socket\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.678247 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.678236 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64b1a682-e6fa-460e-a462-14827fb346a4-istio-podinfo\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.678487 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.678262 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-istio-envoy\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.678487 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.678284 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64b1a682-e6fa-460e-a462-14827fb346a4-istiod-ca-cert\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.678487 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.678348 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-workload-socket\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.678487 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.678388 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-workload-certs\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.678716 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.678640 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-istio-data\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.678850 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.678833 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-credential-socket\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.678919 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.678856 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-workload-socket\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.679054 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.679034 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-workload-certs\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.679120 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.679105 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64b1a682-e6fa-460e-a462-14827fb346a4-istiod-ca-cert\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.681049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.681026 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-istio-envoy\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.681272 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.681254 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64b1a682-e6fa-460e-a462-14827fb346a4-istio-podinfo\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.685952 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.685931 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9vlv\" (UniqueName: \"kubernetes.io/projected/64b1a682-e6fa-460e-a462-14827fb346a4-kube-api-access-x9vlv\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.686160 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.686141 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64b1a682-e6fa-460e-a462-14827fb346a4-istio-token\") pod \"e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.826795 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.826711 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:18.930629 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.930605 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-307a7b-qp4l9" Sep 17 23:27:18.930763 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.930603 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-307a7b-qp4l9" event={"ID":"d33a30f1-8691-4176-94f1-ebf02338b277","Type":"ContainerDied","Data":"2e4a65b2023ccb470b136a1d133cf74c2a668bf64586ba6b4a297628ca2ac44f"} Sep 17 23:27:18.930763 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:18.930724 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e4a65b2023ccb470b136a1d133cf74c2a668bf64586ba6b4a297628ca2ac44f" Sep 17 23:27:19.105530 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.105491 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc"] Sep 17 23:27:19.107485 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:27:19.107459 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b1a682_e6fa_460e_a462_14827fb346a4.slice/crio-37078798fbd6bdb6adfb58acb766f44c23109a89f82b4b6f6cf4f57e6e50c1e5 WatchSource:0}: Error finding container 37078798fbd6bdb6adfb58acb766f44c23109a89f82b4b6f6cf4f57e6e50c1e5: Status 404 returned error can't find the container with id 37078798fbd6bdb6adfb58acb766f44c23109a89f82b4b6f6cf4f57e6e50c1e5 Sep 17 23:27:19.109792 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.109765 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:27:19.109855 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.109827 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:27:19.109893 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.109864 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:27:19.625543 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.625476 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:27:19.687649 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.687626 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-envoy\") pod \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " Sep 17 23:27:19.687792 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.687658 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-data\") pod \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " Sep 17 23:27:19.687792 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.687701 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-credential-socket\") pod \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " Sep 17 23:27:19.687792 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.687726 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-token\") pod \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " Sep 17 23:27:19.687792 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.687766 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-workload-certs\") pod \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " Sep 17 23:27:19.687792 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.687792 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-podinfo\") pod \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " Sep 17 23:27:19.688068 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.687819 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-workload-socket\") pod \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " Sep 17 23:27:19.688068 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.687865 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istiod-ca-cert\") pod \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " Sep 17 23:27:19.688068 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.687926 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6khl\" (UniqueName: \"kubernetes.io/projected/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-kube-api-access-b6khl\") pod \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\" (UID: \"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc\") " Sep 17 23:27:19.688068 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.687924 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-data" (OuterVolumeSpecName: "istio-data") pod "8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" (UID: "8eae828a-0d27-457f-b49c-2f4cb9c4b4bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:27:19.688068 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.687957 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" (UID: "8eae828a-0d27-457f-b49c-2f4cb9c4b4bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:27:19.688316 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.688106 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" (UID: "8eae828a-0d27-457f-b49c-2f4cb9c4b4bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:27:19.688316 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.688176 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" (UID: "8eae828a-0d27-457f-b49c-2f4cb9c4b4bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:27:19.688316 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.688267 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" (UID: "8eae828a-0d27-457f-b49c-2f4cb9c4b4bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:27:19.688316 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.688287 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-workload-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:19.688316 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.688304 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-workload-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:19.688526 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.688317 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-data\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:19.688526 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.688332 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-credential-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:19.690083 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.690061 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-kube-api-access-b6khl" (OuterVolumeSpecName: "kube-api-access-b6khl") pod "8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" (UID: "8eae828a-0d27-457f-b49c-2f4cb9c4b4bc"). InnerVolumeSpecName "kube-api-access-b6khl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:27:19.690194 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.690179 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" (UID: "8eae828a-0d27-457f-b49c-2f4cb9c4b4bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:27:19.690299 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.690275 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" (UID: "8eae828a-0d27-457f-b49c-2f4cb9c4b4bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:27:19.690381 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.690367 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" (UID: "8eae828a-0d27-457f-b49c-2f4cb9c4b4bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:27:19.788868 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.788841 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-podinfo\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:19.788868 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.788865 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istiod-ca-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:19.789025 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.788879 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6khl\" (UniqueName: \"kubernetes.io/projected/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-kube-api-access-b6khl\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:19.789025 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.788891 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-envoy\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:19.789025 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.788902 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc-istio-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:19.937245 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.937165 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" event={"ID":"64b1a682-e6fa-460e-a462-14827fb346a4","Type":"ContainerStarted","Data":"8644db3c01e483f9bffe6d7d7fdae0808f3fc25c9fd6b3880ab7f439f4a4f33b"} Sep 17 23:27:19.937245 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.937223 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" event={"ID":"64b1a682-e6fa-460e-a462-14827fb346a4","Type":"ContainerStarted","Data":"37078798fbd6bdb6adfb58acb766f44c23109a89f82b4b6f6cf4f57e6e50c1e5"} Sep 17 23:27:19.938887 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.938864 2582 generic.go:356] "Generic (PLEG): container finished" podID="8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" containerID="7d55c243fd72a2c6c6fc444195cf96d813197406bbd5fec4b4a70a77cae06f86" exitCode=0 Sep 17 23:27:19.939012 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.938913 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" Sep 17 23:27:19.939012 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.938951 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" event={"ID":"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc","Type":"ContainerDied","Data":"7d55c243fd72a2c6c6fc444195cf96d813197406bbd5fec4b4a70a77cae06f86"} Sep 17 23:27:19.939012 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.939008 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx" event={"ID":"8eae828a-0d27-457f-b49c-2f4cb9c4b4bc","Type":"ContainerDied","Data":"432cc15aaa5ae5a10ef34744f42b70c60594a641ee4d231ffdb48b25fb9dcdc8"} Sep 17 23:27:19.939181 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.939038 2582 scope.go:117] "RemoveContainer" containerID="7d55c243fd72a2c6c6fc444195cf96d813197406bbd5fec4b4a70a77cae06f86" Sep 17 23:27:19.948249 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.948234 2582 scope.go:117] "RemoveContainer" containerID="3d22c2d2fc58f164326f53875036c8b989c435ee9e1cfd953adb30c252069441" Sep 17 23:27:19.955205 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.955166 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" podStartSLOduration=1.955153643 podStartE2EDuration="1.955153643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:27:19.95382977 +0000 UTC m=+1278.887736907" watchObservedRunningTime="2026-09-17 23:27:19.955153643 +0000 UTC m=+1278.889060777" Sep 17 23:27:19.956674 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.956654 2582 scope.go:117] "RemoveContainer" containerID="7d55c243fd72a2c6c6fc444195cf96d813197406bbd5fec4b4a70a77cae06f86" Sep 17 23:27:19.956955 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:27:19.956937 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d55c243fd72a2c6c6fc444195cf96d813197406bbd5fec4b4a70a77cae06f86\": container with ID starting with 7d55c243fd72a2c6c6fc444195cf96d813197406bbd5fec4b4a70a77cae06f86 not found: ID does not exist" containerID="7d55c243fd72a2c6c6fc444195cf96d813197406bbd5fec4b4a70a77cae06f86" Sep 17 23:27:19.957005 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.956963 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d55c243fd72a2c6c6fc444195cf96d813197406bbd5fec4b4a70a77cae06f86"} err="failed to get container status \"7d55c243fd72a2c6c6fc444195cf96d813197406bbd5fec4b4a70a77cae06f86\": rpc error: code = NotFound desc = could not find container \"7d55c243fd72a2c6c6fc444195cf96d813197406bbd5fec4b4a70a77cae06f86\": container with ID starting with 7d55c243fd72a2c6c6fc444195cf96d813197406bbd5fec4b4a70a77cae06f86 not found: ID does not exist" Sep 17 23:27:19.957005 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.956981 2582 scope.go:117] "RemoveContainer" containerID="3d22c2d2fc58f164326f53875036c8b989c435ee9e1cfd953adb30c252069441" Sep 17 23:27:19.957250 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:27:19.957232 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d22c2d2fc58f164326f53875036c8b989c435ee9e1cfd953adb30c252069441\": container with ID starting with 3d22c2d2fc58f164326f53875036c8b989c435ee9e1cfd953adb30c252069441 not found: ID does not exist" containerID="3d22c2d2fc58f164326f53875036c8b989c435ee9e1cfd953adb30c252069441" Sep 17 23:27:19.957297 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.957257 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d22c2d2fc58f164326f53875036c8b989c435ee9e1cfd953adb30c252069441"} err="failed to get container status \"3d22c2d2fc58f164326f53875036c8b989c435ee9e1cfd953adb30c252069441\": rpc error: code = NotFound desc = could not find container \"3d22c2d2fc58f164326f53875036c8b989c435ee9e1cfd953adb30c252069441\": container with ID starting with 3d22c2d2fc58f164326f53875036c8b989c435ee9e1cfd953adb30c252069441 not found: ID does not exist" Sep 17 23:27:19.969025 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.969003 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx"] Sep 17 23:27:19.973254 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:19.973234 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0c363f-openshift-default-678b6458b4-rqgfx"] Sep 17 23:27:20.827364 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:20.827337 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:20.832113 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:20.832092 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:20.948063 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:20.948036 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:20.948989 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:20.948971 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:21.294285 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:21.294250 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" path="/var/lib/kubelet/pods/8eae828a-0d27-457f-b49c-2f4cb9c4b4bc/volumes" Sep 17 23:27:24.873308 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:24.872820 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl"] Sep 17 23:27:24.873778 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:24.873446 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" containerName="istio-proxy" Sep 17 23:27:24.873778 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:24.873466 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" containerName="istio-proxy" Sep 17 23:27:24.873778 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:24.873483 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" containerName="istio-proxy" Sep 17 23:27:24.873778 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:24.873492 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" containerName="istio-proxy" Sep 17 23:27:24.873778 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:24.873604 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" containerName="istio-proxy" Sep 17 23:27:24.873778 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:24.873616 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eae828a-0d27-457f-b49c-2f4cb9c4b4bc" containerName="istio-proxy" Sep 17 23:27:24.877085 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:24.877064 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl" Sep 17 23:27:24.879764 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:24.879738 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-d69250\"" Sep 17 23:27:24.879853 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:24.879768 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-d69250-dockercfg-4kpt9\"" Sep 17 23:27:24.884030 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:24.884002 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl"] Sep 17 23:27:25.035574 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:25.035536 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks2fj\" (UniqueName: \"kubernetes.io/projected/825f8f34-8b44-4123-962e-4e20a859326d-kube-api-access-ks2fj\") pod \"payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl\" (UID: \"825f8f34-8b44-4123-962e-4e20a859326d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl" Sep 17 23:27:25.035730 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:25.035692 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825f8f34-8b44-4123-962e-4e20a859326d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl\" (UID: \"825f8f34-8b44-4123-962e-4e20a859326d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl" Sep 17 23:27:25.137285 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:25.137206 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825f8f34-8b44-4123-962e-4e20a859326d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl\" (UID: \"825f8f34-8b44-4123-962e-4e20a859326d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl" Sep 17 23:27:25.137285 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:25.137260 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks2fj\" (UniqueName: \"kubernetes.io/projected/825f8f34-8b44-4123-962e-4e20a859326d-kube-api-access-ks2fj\") pod \"payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl\" (UID: \"825f8f34-8b44-4123-962e-4e20a859326d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl" Sep 17 23:27:25.137933 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:25.137911 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825f8f34-8b44-4123-962e-4e20a859326d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl\" (UID: \"825f8f34-8b44-4123-962e-4e20a859326d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl" Sep 17 23:27:25.146190 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:25.146169 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks2fj\" (UniqueName: \"kubernetes.io/projected/825f8f34-8b44-4123-962e-4e20a859326d-kube-api-access-ks2fj\") pod \"payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl\" (UID: \"825f8f34-8b44-4123-962e-4e20a859326d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl" Sep 17 23:27:25.188131 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:25.188099 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl" Sep 17 23:27:25.301777 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:25.301728 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-307a7b-qp4l9"] Sep 17 23:27:25.303738 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:25.303708 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-307a7b-qp4l9"] Sep 17 23:27:25.463396 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:25.463372 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl"] Sep 17 23:27:25.465614 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:27:25.465589 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825f8f34_8b44_4123_962e_4e20a859326d.slice/crio-1a0e99d71b9769a837342e14d06f157264083ad7a8d96ee0421685280b56ac6b WatchSource:0}: Error finding container 1a0e99d71b9769a837342e14d06f157264083ad7a8d96ee0421685280b56ac6b: Status 404 returned error can't find the container with id 1a0e99d71b9769a837342e14d06f157264083ad7a8d96ee0421685280b56ac6b Sep 17 23:27:25.981901 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:25.981861 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl" event={"ID":"825f8f34-8b44-4123-962e-4e20a859326d","Type":"ContainerStarted","Data":"96fec26353d1b193685be39c38e404719cd493802892d2d52778e6a93465417b"} Sep 17 23:27:25.982331 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:25.981925 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl" event={"ID":"825f8f34-8b44-4123-962e-4e20a859326d","Type":"ContainerStarted","Data":"1a0e99d71b9769a837342e14d06f157264083ad7a8d96ee0421685280b56ac6b"} Sep 17 23:27:25.982331 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:25.981957 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl" Sep 17 23:27:26.000409 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:26.000359 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl" podStartSLOduration=2.000343395 podStartE2EDuration="2.000343395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:27:25.998454099 +0000 UTC m=+1284.932361235" watchObservedRunningTime="2026-09-17 23:27:26.000343395 +0000 UTC m=+1284.934250532" Sep 17 23:27:27.295431 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:27.295398 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d33a30f1-8691-4176-94f1-ebf02338b277" path="/var/lib/kubelet/pods/d33a30f1-8691-4176-94f1-ebf02338b277/volumes" Sep 17 23:27:31.637718 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:31.637680 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx"] Sep 17 23:27:31.638272 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:31.637934 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" podUID="4392fd08-44e2-4bb0-8833-eea74a376d48" containerName="istio-proxy" containerID="cri-o://42f05f83b9ca2ee610dccd5b904630d80b22470eca7b6a56c3bc35067035f6c6" gracePeriod=30 Sep 17 23:27:32.509547 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.509480 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls"] Sep 17 23:27:32.514188 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.514161 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls"] Sep 17 23:27:32.514317 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.514278 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.516720 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.516693 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-d90df4-openshift-default-dockercfg-bfcz8\"" Sep 17 23:27:32.710369 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.710336 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-data\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.710369 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.710368 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-workload-certs\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.710864 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.710387 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-credential-socket\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.710864 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.710415 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfr7x\" (UniqueName: \"kubernetes.io/projected/a2e7b030-1b71-4ce5-8232-1845609394a2-kube-api-access-zfr7x\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.710864 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.710487 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-envoy\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.710864 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.710575 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e7b030-1b71-4ce5-8232-1845609394a2-istiod-ca-cert\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.710864 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.710612 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-workload-socket\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.710864 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.710660 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-podinfo\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.710864 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.710681 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-token\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.811188 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.811102 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e7b030-1b71-4ce5-8232-1845609394a2-istiod-ca-cert\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.811188 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.811147 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-workload-socket\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.811188 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.811165 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-podinfo\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.811188 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.811179 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-token\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.811484 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.811220 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-data\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.811484 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.811235 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-workload-certs\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.811484 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.811252 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-credential-socket\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.811484 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.811279 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfr7x\" (UniqueName: \"kubernetes.io/projected/a2e7b030-1b71-4ce5-8232-1845609394a2-kube-api-access-zfr7x\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.811484 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.811311 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-envoy\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.811830 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.811772 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-workload-socket\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.811969 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.811827 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-data\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.811969 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.811881 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-workload-certs\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.811969 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.811934 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-credential-socket\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.812322 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.812065 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e7b030-1b71-4ce5-8232-1845609394a2-istiod-ca-cert\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.814567 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.814546 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-envoy\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.814878 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.814834 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-podinfo\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.818488 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.818470 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-token\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.818980 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.818965 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfr7x\" (UniqueName: \"kubernetes.io/projected/a2e7b030-1b71-4ce5-8232-1845609394a2-kube-api-access-zfr7x\") pod \"e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:32.824918 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:32.824901 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:33.130546 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:33.130519 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls"] Sep 17 23:27:33.132080 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:27:33.132049 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e7b030_1b71_4ce5_8232_1845609394a2.slice/crio-9d662e7a969596ff4f89acc9486daaad5e6052479422f7699f7fd3f9a9f2a450 WatchSource:0}: Error finding container 9d662e7a969596ff4f89acc9486daaad5e6052479422f7699f7fd3f9a9f2a450: Status 404 returned error can't find the container with id 9d662e7a969596ff4f89acc9486daaad5e6052479422f7699f7fd3f9a9f2a450 Sep 17 23:27:33.134537 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:33.134456 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:27:33.134629 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:33.134563 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:27:33.134629 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:33.134608 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:27:34.036483 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:34.036444 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" event={"ID":"a2e7b030-1b71-4ce5-8232-1845609394a2","Type":"ContainerStarted","Data":"5a33108fc17c20bdc3c9c9d52142f27180752a6622513a47f4ae3d79a3bb4519"} Sep 17 23:27:34.036892 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:34.036557 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" event={"ID":"a2e7b030-1b71-4ce5-8232-1845609394a2","Type":"ContainerStarted","Data":"9d662e7a969596ff4f89acc9486daaad5e6052479422f7699f7fd3f9a9f2a450"} Sep 17 23:27:34.061807 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:34.061761 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" podStartSLOduration=2.061747866 podStartE2EDuration="2.061747866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:27:34.059673123 +0000 UTC m=+1292.993580256" watchObservedRunningTime="2026-09-17 23:27:34.061747866 +0000 UTC m=+1292.995655000" Sep 17 23:27:34.825836 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:34.825808 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:34.830447 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:34.830417 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:35.042493 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:35.042452 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:35.043476 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:35.043455 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:36.901425 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.901366 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:27:36.950713 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.950677 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-podinfo\") pod \"4392fd08-44e2-4bb0-8833-eea74a376d48\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " Sep 17 23:27:36.950884 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.950731 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-token\") pod \"4392fd08-44e2-4bb0-8833-eea74a376d48\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " Sep 17 23:27:36.950884 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.950844 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-credential-socket\") pod \"4392fd08-44e2-4bb0-8833-eea74a376d48\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " Sep 17 23:27:36.950997 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.950902 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-data\") pod \"4392fd08-44e2-4bb0-8833-eea74a376d48\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " Sep 17 23:27:36.950997 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.950974 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-workload-certs\") pod \"4392fd08-44e2-4bb0-8833-eea74a376d48\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " Sep 17 23:27:36.951153 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.951013 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-workload-socket\") pod \"4392fd08-44e2-4bb0-8833-eea74a376d48\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " Sep 17 23:27:36.951153 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.951037 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-envoy\") pod \"4392fd08-44e2-4bb0-8833-eea74a376d48\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " Sep 17 23:27:36.951153 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.951060 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxkbg\" (UniqueName: \"kubernetes.io/projected/4392fd08-44e2-4bb0-8833-eea74a376d48-kube-api-access-cxkbg\") pod \"4392fd08-44e2-4bb0-8833-eea74a376d48\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " Sep 17 23:27:36.951153 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.951089 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4392fd08-44e2-4bb0-8833-eea74a376d48-istiod-ca-cert\") pod \"4392fd08-44e2-4bb0-8833-eea74a376d48\" (UID: \"4392fd08-44e2-4bb0-8833-eea74a376d48\") " Sep 17 23:27:36.951356 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.951227 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4392fd08-44e2-4bb0-8833-eea74a376d48" (UID: "4392fd08-44e2-4bb0-8833-eea74a376d48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:27:36.951536 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.951428 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-workload-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:36.951629 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.951562 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4392fd08-44e2-4bb0-8833-eea74a376d48" (UID: "4392fd08-44e2-4bb0-8833-eea74a376d48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:27:36.951952 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.951907 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-data" (OuterVolumeSpecName: "istio-data") pod "4392fd08-44e2-4bb0-8833-eea74a376d48" (UID: "4392fd08-44e2-4bb0-8833-eea74a376d48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:27:36.951952 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.951921 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4392fd08-44e2-4bb0-8833-eea74a376d48" (UID: "4392fd08-44e2-4bb0-8833-eea74a376d48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:27:36.952128 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.951953 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4392fd08-44e2-4bb0-8833-eea74a376d48-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4392fd08-44e2-4bb0-8833-eea74a376d48" (UID: "4392fd08-44e2-4bb0-8833-eea74a376d48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:27:36.953534 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.953469 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-token" (OuterVolumeSpecName: "istio-token") pod "4392fd08-44e2-4bb0-8833-eea74a376d48" (UID: "4392fd08-44e2-4bb0-8833-eea74a376d48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:27:36.954336 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.954298 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4392fd08-44e2-4bb0-8833-eea74a376d48" (UID: "4392fd08-44e2-4bb0-8833-eea74a376d48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:27:36.954860 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.954828 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4392fd08-44e2-4bb0-8833-eea74a376d48-kube-api-access-cxkbg" (OuterVolumeSpecName: "kube-api-access-cxkbg") pod "4392fd08-44e2-4bb0-8833-eea74a376d48" (UID: "4392fd08-44e2-4bb0-8833-eea74a376d48"). InnerVolumeSpecName "kube-api-access-cxkbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:27:36.955020 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.955003 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4392fd08-44e2-4bb0-8833-eea74a376d48" (UID: "4392fd08-44e2-4bb0-8833-eea74a376d48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:27:36.989660 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.989619 2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 17 23:27:36.989783 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.989691 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" podUID="64b1a682-e6fa-460e-a462-14827fb346a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 17 23:27:36.990943 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:36.990922 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl" Sep 17 23:27:37.052602 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.052573 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-workload-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:37.052602 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.052602 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-envoy\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:37.052843 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.052618 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxkbg\" (UniqueName: \"kubernetes.io/projected/4392fd08-44e2-4bb0-8833-eea74a376d48-kube-api-access-cxkbg\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:37.052843 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.052633 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4392fd08-44e2-4bb0-8833-eea74a376d48-istiod-ca-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:37.052843 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.052646 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-podinfo\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:37.052843 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.052657 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:37.052843 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.052669 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-credential-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:37.052843 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.052680 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4392fd08-44e2-4bb0-8833-eea74a376d48-istio-data\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:27:37.056893 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.056867 2582 generic.go:356] "Generic (PLEG): container finished" podID="4392fd08-44e2-4bb0-8833-eea74a376d48" containerID="42f05f83b9ca2ee610dccd5b904630d80b22470eca7b6a56c3bc35067035f6c6" exitCode=0 Sep 17 23:27:37.057015 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.056924 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" Sep 17 23:27:37.057015 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.056931 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" event={"ID":"4392fd08-44e2-4bb0-8833-eea74a376d48","Type":"ContainerDied","Data":"42f05f83b9ca2ee610dccd5b904630d80b22470eca7b6a56c3bc35067035f6c6"} Sep 17 23:27:37.057015 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.056991 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx" event={"ID":"4392fd08-44e2-4bb0-8833-eea74a376d48","Type":"ContainerDied","Data":"c15f2ec92eb015e5b379d8d5b52230f639214a00784e9fa41e364799e226cc93"} Sep 17 23:27:37.057124 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.057023 2582 scope.go:117] "RemoveContainer" containerID="42f05f83b9ca2ee610dccd5b904630d80b22470eca7b6a56c3bc35067035f6c6" Sep 17 23:27:37.069730 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.069709 2582 scope.go:117] "RemoveContainer" containerID="3748961b33628336df8aa9e5bd3d7d5e20a3b625897b3a92db411fcf9ca881f0" Sep 17 23:27:37.079433 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.079414 2582 scope.go:117] "RemoveContainer" containerID="42f05f83b9ca2ee610dccd5b904630d80b22470eca7b6a56c3bc35067035f6c6" Sep 17 23:27:37.079778 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:27:37.079760 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42f05f83b9ca2ee610dccd5b904630d80b22470eca7b6a56c3bc35067035f6c6\": container with ID starting with 42f05f83b9ca2ee610dccd5b904630d80b22470eca7b6a56c3bc35067035f6c6 not found: ID does not exist" containerID="42f05f83b9ca2ee610dccd5b904630d80b22470eca7b6a56c3bc35067035f6c6" Sep 17 23:27:37.079833 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.079786 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42f05f83b9ca2ee610dccd5b904630d80b22470eca7b6a56c3bc35067035f6c6"} err="failed to get container status \"42f05f83b9ca2ee610dccd5b904630d80b22470eca7b6a56c3bc35067035f6c6\": rpc error: code = NotFound desc = could not find container \"42f05f83b9ca2ee610dccd5b904630d80b22470eca7b6a56c3bc35067035f6c6\": container with ID starting with 42f05f83b9ca2ee610dccd5b904630d80b22470eca7b6a56c3bc35067035f6c6 not found: ID does not exist" Sep 17 23:27:37.079833 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.079803 2582 scope.go:117] "RemoveContainer" containerID="3748961b33628336df8aa9e5bd3d7d5e20a3b625897b3a92db411fcf9ca881f0" Sep 17 23:27:37.080094 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:27:37.080073 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3748961b33628336df8aa9e5bd3d7d5e20a3b625897b3a92db411fcf9ca881f0\": container with ID starting with 3748961b33628336df8aa9e5bd3d7d5e20a3b625897b3a92db411fcf9ca881f0 not found: ID does not exist" containerID="3748961b33628336df8aa9e5bd3d7d5e20a3b625897b3a92db411fcf9ca881f0" Sep 17 23:27:37.080207 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.080100 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3748961b33628336df8aa9e5bd3d7d5e20a3b625897b3a92db411fcf9ca881f0"} err="failed to get container status \"3748961b33628336df8aa9e5bd3d7d5e20a3b625897b3a92db411fcf9ca881f0\": rpc error: code = NotFound desc = could not find container \"3748961b33628336df8aa9e5bd3d7d5e20a3b625897b3a92db411fcf9ca881f0\": container with ID starting with 3748961b33628336df8aa9e5bd3d7d5e20a3b625897b3a92db411fcf9ca881f0 not found: ID does not exist" Sep 17 23:27:37.083542 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.083491 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx"] Sep 17 23:27:37.088588 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.088563 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-307a7b-openshift-default-fdd9b686d-bl9qx"] Sep 17 23:27:37.293850 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:37.293821 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4392fd08-44e2-4bb0-8833-eea74a376d48" path="/var/lib/kubelet/pods/4392fd08-44e2-4bb0-8833-eea74a376d48/volumes" Sep 17 23:27:38.065414 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:38.065388 2582 generic.go:356] "Generic (PLEG): container finished" podID="64b1a682-e6fa-460e-a462-14827fb346a4" containerID="8644db3c01e483f9bffe6d7d7fdae0808f3fc25c9fd6b3880ab7f439f4a4f33b" exitCode=0 Sep 17 23:27:38.065854 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:38.065463 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" event={"ID":"64b1a682-e6fa-460e-a462-14827fb346a4","Type":"ContainerDied","Data":"8644db3c01e483f9bffe6d7d7fdae0808f3fc25c9fd6b3880ab7f439f4a4f33b"} Sep 17 23:27:38.065917 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:38.065893 2582 scope.go:117] "RemoveContainer" containerID="8644db3c01e483f9bffe6d7d7fdae0808f3fc25c9fd6b3880ab7f439f4a4f33b" Sep 17 23:27:38.067246 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:38.067208 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:27:38.067353 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:38.067297 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:27:38.067353 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:38.067345 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:27:39.078200 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:39.078154 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" event={"ID":"64b1a682-e6fa-460e-a462-14827fb346a4","Type":"ContainerStarted","Data":"8bf6aadeb6bc1acbfd3136a4430df35e93f5d6f2e6148afd19bfc0436715c611"} Sep 17 23:27:39.827228 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:39.827190 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:39.827434 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:39.827407 2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 17 23:27:39.827518 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:39.827486 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" podUID="64b1a682-e6fa-460e-a462-14827fb346a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 17 23:27:40.827078 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:40.827044 2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 17 23:27:40.827552 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:40.827106 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" podUID="64b1a682-e6fa-460e-a462-14827fb346a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 17 23:27:41.827900 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:41.827867 2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 17 23:27:41.828268 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:41.827920 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" podUID="64b1a682-e6fa-460e-a462-14827fb346a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 17 23:27:42.827840 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:42.827806 2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 17 23:27:42.828004 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:42.827864 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" podUID="64b1a682-e6fa-460e-a462-14827fb346a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 17 23:27:43.827848 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:43.827808 2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 17 23:27:43.828021 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:43.827887 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" podUID="64b1a682-e6fa-460e-a462-14827fb346a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 17 23:27:43.917678 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:43.917642 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7"] Sep 17 23:27:43.918067 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:43.918055 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4392fd08-44e2-4bb0-8833-eea74a376d48" containerName="istio-proxy" Sep 17 23:27:43.918111 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:43.918068 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4392fd08-44e2-4bb0-8833-eea74a376d48" containerName="istio-proxy" Sep 17 23:27:43.918175 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:43.918151 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4392fd08-44e2-4bb0-8833-eea74a376d48" containerName="istio-proxy" Sep 17 23:27:43.918175 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:43.918158 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4392fd08-44e2-4bb0-8833-eea74a376d48" containerName="istio-proxy" Sep 17 23:27:43.920737 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:43.920718 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7" Sep 17 23:27:43.922977 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:43.922955 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-fd82a9\"" Sep 17 23:27:43.934209 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:43.934180 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7"] Sep 17 23:27:44.036277 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:44.036239 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v87bb\" (UniqueName: \"kubernetes.io/projected/04502971-5f33-4f73-b07e-5025585c6be3-kube-api-access-v87bb\") pod \"payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7\" (UID: \"04502971-5f33-4f73-b07e-5025585c6be3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7" Sep 17 23:27:44.036643 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:44.036613 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04502971-5f33-4f73-b07e-5025585c6be3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7\" (UID: \"04502971-5f33-4f73-b07e-5025585c6be3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7" Sep 17 23:27:44.138173 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:44.138083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v87bb\" (UniqueName: \"kubernetes.io/projected/04502971-5f33-4f73-b07e-5025585c6be3-kube-api-access-v87bb\") pod \"payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7\" (UID: \"04502971-5f33-4f73-b07e-5025585c6be3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7" Sep 17 23:27:44.138173 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:44.138135 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04502971-5f33-4f73-b07e-5025585c6be3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7\" (UID: \"04502971-5f33-4f73-b07e-5025585c6be3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7" Sep 17 23:27:44.138844 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:44.138827 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04502971-5f33-4f73-b07e-5025585c6be3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7\" (UID: \"04502971-5f33-4f73-b07e-5025585c6be3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7" Sep 17 23:27:44.145617 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:44.145594 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v87bb\" (UniqueName: \"kubernetes.io/projected/04502971-5f33-4f73-b07e-5025585c6be3-kube-api-access-v87bb\") pod \"payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7\" (UID: \"04502971-5f33-4f73-b07e-5025585c6be3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7" Sep 17 23:27:44.232151 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:44.232124 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7" Sep 17 23:27:44.504206 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:44.504177 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7"] Sep 17 23:27:44.506128 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:27:44.506099 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04502971_5f33_4f73_b07e_5025585c6be3.slice/crio-a73d47cb173cc5ccaf7191012fd41623d2d6e3cece59771f4839f275025b37c1 WatchSource:0}: Error finding container a73d47cb173cc5ccaf7191012fd41623d2d6e3cece59771f4839f275025b37c1: Status 404 returned error can't find the container with id a73d47cb173cc5ccaf7191012fd41623d2d6e3cece59771f4839f275025b37c1 Sep 17 23:27:44.830736 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:44.830656 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:44.831173 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:44.831021 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:44.831673 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:44.831655 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:27:45.122884 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:45.122799 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7" event={"ID":"04502971-5f33-4f73-b07e-5025585c6be3","Type":"ContainerStarted","Data":"e1f954028cc9acd1d4771e1501639cdb8777fb0caa787fb6ed83eb58b268505e"} Sep 17 23:27:45.123021 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:45.122930 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7" Sep 17 23:27:45.123021 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:45.122950 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7" event={"ID":"04502971-5f33-4f73-b07e-5025585c6be3","Type":"ContainerStarted","Data":"a73d47cb173cc5ccaf7191012fd41623d2d6e3cece59771f4839f275025b37c1"} Sep 17 23:27:45.140336 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:45.140287 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7" podStartSLOduration=2.140272372 podStartE2EDuration="2.140272372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:27:45.138767221 +0000 UTC m=+1304.072674356" watchObservedRunningTime="2026-09-17 23:27:45.140272372 +0000 UTC m=+1304.074179507" Sep 17 23:27:54.189097 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:54.188997 2582 generic.go:356] "Generic (PLEG): container finished" podID="a2e7b030-1b71-4ce5-8232-1845609394a2" containerID="5a33108fc17c20bdc3c9c9d52142f27180752a6622513a47f4ae3d79a3bb4519" exitCode=0 Sep 17 23:27:54.189097 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:54.189067 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" event={"ID":"a2e7b030-1b71-4ce5-8232-1845609394a2","Type":"ContainerDied","Data":"5a33108fc17c20bdc3c9c9d52142f27180752a6622513a47f4ae3d79a3bb4519"} Sep 17 23:27:54.189639 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:54.189437 2582 scope.go:117] "RemoveContainer" containerID="5a33108fc17c20bdc3c9c9d52142f27180752a6622513a47f4ae3d79a3bb4519" Sep 17 23:27:54.190491 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:54.190459 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:27:54.190620 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:54.190541 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:27:54.190620 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:54.190572 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:27:55.207724 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:55.207682 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" event={"ID":"a2e7b030-1b71-4ce5-8232-1845609394a2","Type":"ContainerStarted","Data":"b9d41583181aa3f257b87bf2bca32464d6f466062fed9a6495d89ebb4b489817"} Sep 17 23:27:55.826191 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:55.826156 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:27:55.826362 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:55.826341 2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 17 23:27:55.826415 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:55.826394 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" podUID="a2e7b030-1b71-4ce5-8232-1845609394a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 17 23:27:56.131060 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:56.130987 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7" Sep 17 23:27:56.825875 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:56.825845 2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 17 23:27:56.826238 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:56.825897 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" podUID="a2e7b030-1b71-4ce5-8232-1845609394a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 17 23:27:57.825407 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:57.825369 2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 17 23:27:57.825626 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:57.825444 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" podUID="a2e7b030-1b71-4ce5-8232-1845609394a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 17 23:27:58.825823 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:58.825789 2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 17 23:27:58.826219 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:58.825854 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" podUID="a2e7b030-1b71-4ce5-8232-1845609394a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 17 23:27:59.826190 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:59.826153 2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 17 23:27:59.826627 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:27:59.826214 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" podUID="a2e7b030-1b71-4ce5-8232-1845609394a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 17 23:28:00.829954 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:00.829927 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:28:00.830349 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:00.830208 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:28:00.830918 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:00.830902 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:28:01.562458 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.562429 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7"] Sep 17 23:28:01.562875 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.562861 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4392fd08-44e2-4bb0-8833-eea74a376d48" containerName="istio-proxy" Sep 17 23:28:01.562919 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.562876 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4392fd08-44e2-4bb0-8833-eea74a376d48" containerName="istio-proxy" Sep 17 23:28:01.565564 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.565543 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.567749 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.567725 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-d92920-openshift-default-dockercfg-pblnt\"" Sep 17 23:28:01.579473 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.579450 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7"] Sep 17 23:28:01.724903 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.724869 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-data\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.725059 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.724925 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whcmw\" (UniqueName: \"kubernetes.io/projected/dc656650-7bb4-46ee-9b96-55cc6cc7f124-kube-api-access-whcmw\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.725059 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.724957 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-envoy\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.725059 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.725000 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-credential-socket\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.725059 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.725026 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-workload-certs\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.725059 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.725057 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-podinfo\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.725236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.725072 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istiod-ca-cert\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.725236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.725102 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-token\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.725236 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.725153 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-workload-socket\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.826076 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.825997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-credential-socket\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.826076 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.826035 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-workload-certs\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.826076 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.826074 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-podinfo\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.826356 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.826096 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istiod-ca-cert\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.826356 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.826134 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-token\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.826356 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.826169 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-workload-socket\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.826356 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.826215 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-data\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.826614 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.826438 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-credential-socket\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.826614 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.826483 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-workload-certs\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.826614 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.826481 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whcmw\" (UniqueName: \"kubernetes.io/projected/dc656650-7bb4-46ee-9b96-55cc6cc7f124-kube-api-access-whcmw\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.826614 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.826589 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-envoy\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.826832 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.826612 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-workload-socket\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.826832 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.826623 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-data\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.826939 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.826916 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istiod-ca-cert\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.828790 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.828771 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-podinfo\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.828886 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.828800 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-envoy\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.833822 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.833801 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whcmw\" (UniqueName: \"kubernetes.io/projected/dc656650-7bb4-46ee-9b96-55cc6cc7f124-kube-api-access-whcmw\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.834113 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.833887 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-token\") pod \"e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:01.880578 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:01.880557 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:02.158484 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:02.158461 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7"] Sep 17 23:28:02.160815 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:28:02.160782 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc656650_7bb4_46ee_9b96_55cc6cc7f124.slice/crio-7ff0e53b1968b9c5ff6002912ef015c6dfc13ed59c60ced154ebf536c43b75a9 WatchSource:0}: Error finding container 7ff0e53b1968b9c5ff6002912ef015c6dfc13ed59c60ced154ebf536c43b75a9: Status 404 returned error can't find the container with id 7ff0e53b1968b9c5ff6002912ef015c6dfc13ed59c60ced154ebf536c43b75a9 Sep 17 23:28:02.163053 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:02.163018 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:28:02.163145 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:02.163081 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:28:02.163145 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:02.163109 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:28:02.262361 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:02.262311 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" event={"ID":"dc656650-7bb4-46ee-9b96-55cc6cc7f124","Type":"ContainerStarted","Data":"20dff8cb474387c803aacce3199ac60c83df7f55994d47755d6d778e61bf3e39"} Sep 17 23:28:02.262542 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:02.262464 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" event={"ID":"dc656650-7bb4-46ee-9b96-55cc6cc7f124","Type":"ContainerStarted","Data":"7ff0e53b1968b9c5ff6002912ef015c6dfc13ed59c60ced154ebf536c43b75a9"} Sep 17 23:28:02.281648 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:02.281465 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" podStartSLOduration=1.28145136 podStartE2EDuration="1.28145136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:28:02.278927782 +0000 UTC m=+1321.212834917" watchObservedRunningTime="2026-09-17 23:28:02.28145136 +0000 UTC m=+1321.215358544" Sep 17 23:28:02.881451 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:02.881419 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:03.886630 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:03.886600 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:04.276579 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:04.276548 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:04.277601 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:04.277580 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:10.637692 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:10.637658 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7849-874343"] Sep 17 23:28:10.641590 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:10.641569 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-874343" Sep 17 23:28:10.644283 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:10.644260 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m6p2p\"" Sep 17 23:28:10.646945 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:10.646926 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7849-874343"] Sep 17 23:28:10.712346 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:10.712319 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7xqj\" (UniqueName: \"kubernetes.io/projected/7b46f501-ec92-4c1a-b395-735598bcdcb5-kube-api-access-v7xqj\") pod \"test-curl-7849-874343\" (UID: \"7b46f501-ec92-4c1a-b395-735598bcdcb5\") " pod="opendatahub/test-curl-7849-874343" Sep 17 23:28:10.813171 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:10.813136 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7xqj\" (UniqueName: \"kubernetes.io/projected/7b46f501-ec92-4c1a-b395-735598bcdcb5-kube-api-access-v7xqj\") pod \"test-curl-7849-874343\" (UID: \"7b46f501-ec92-4c1a-b395-735598bcdcb5\") " pod="opendatahub/test-curl-7849-874343" Sep 17 23:28:10.820916 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:10.820892 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7xqj\" (UniqueName: \"kubernetes.io/projected/7b46f501-ec92-4c1a-b395-735598bcdcb5-kube-api-access-v7xqj\") pod \"test-curl-7849-874343\" (UID: \"7b46f501-ec92-4c1a-b395-735598bcdcb5\") " pod="opendatahub/test-curl-7849-874343" Sep 17 23:28:10.952793 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:10.952717 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-874343" Sep 17 23:28:11.221967 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:11.221935 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7849-874343"] Sep 17 23:28:11.225858 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:28:11.225830 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b46f501_ec92_4c1a_b395_735598bcdcb5.slice/crio-10cfd072a16d426b7a0d1ffb0d5e91929d1439bd7871913480291a95c5aad418 WatchSource:0}: Error finding container 10cfd072a16d426b7a0d1ffb0d5e91929d1439bd7871913480291a95c5aad418: Status 404 returned error can't find the container with id 10cfd072a16d426b7a0d1ffb0d5e91929d1439bd7871913480291a95c5aad418 Sep 17 23:28:11.318090 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:11.318058 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7849-874343" event={"ID":"7b46f501-ec92-4c1a-b395-735598bcdcb5","Type":"ContainerStarted","Data":"a1661aa44f2a1077835508a14e1d0c572d2fe0fc7fbf0f05f7c2c1d93673e085"} Sep 17 23:28:11.318241 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:11.318105 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7849-874343" event={"ID":"7b46f501-ec92-4c1a-b395-735598bcdcb5","Type":"ContainerStarted","Data":"10cfd072a16d426b7a0d1ffb0d5e91929d1439bd7871913480291a95c5aad418"} Sep 17 23:28:11.331481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:11.331443 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7849-874343" podStartSLOduration=1.331428616 podStartE2EDuration="1.331428616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:28:11.330164465 +0000 UTC m=+1330.264071602" watchObservedRunningTime="2026-09-17 23:28:11.331428616 +0000 UTC m=+1330.265335764" Sep 17 23:28:11.917864 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:11.917767 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7849-874343"] Sep 17 23:28:11.920537 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:11.920511 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7849-874343"] Sep 17 23:28:12.042265 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:12.042232 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7849-bd5129"] Sep 17 23:28:12.042692 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:12.042671 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b46f501-ec92-4c1a-b395-735598bcdcb5" containerName="test-curl-7849-874343" Sep 17 23:28:12.042692 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:12.042689 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b46f501-ec92-4c1a-b395-735598bcdcb5" containerName="test-curl-7849-874343" Sep 17 23:28:12.042878 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:12.042865 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b46f501-ec92-4c1a-b395-735598bcdcb5" containerName="test-curl-7849-874343" Sep 17 23:28:12.045302 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:12.045284 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-bd5129" Sep 17 23:28:12.050617 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:12.050595 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7849-bd5129"] Sep 17 23:28:12.125962 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:12.125927 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgzh2\" (UniqueName: \"kubernetes.io/projected/ba7187e1-9893-4402-88eb-8c6d47a70bfb-kube-api-access-bgzh2\") pod \"test-curl-7849-bd5129\" (UID: \"ba7187e1-9893-4402-88eb-8c6d47a70bfb\") " pod="opendatahub/test-curl-7849-bd5129" Sep 17 23:28:12.227483 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:12.227454 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgzh2\" (UniqueName: \"kubernetes.io/projected/ba7187e1-9893-4402-88eb-8c6d47a70bfb-kube-api-access-bgzh2\") pod \"test-curl-7849-bd5129\" (UID: \"ba7187e1-9893-4402-88eb-8c6d47a70bfb\") " pod="opendatahub/test-curl-7849-bd5129" Sep 17 23:28:12.234932 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:12.234911 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgzh2\" (UniqueName: \"kubernetes.io/projected/ba7187e1-9893-4402-88eb-8c6d47a70bfb-kube-api-access-bgzh2\") pod \"test-curl-7849-bd5129\" (UID: \"ba7187e1-9893-4402-88eb-8c6d47a70bfb\") " pod="opendatahub/test-curl-7849-bd5129" Sep 17 23:28:12.328528 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:28:12.328463 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7849-874343" container="test-curl-7849-874343" Sep 17 23:28:12.357775 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:12.357742 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-bd5129" Sep 17 23:28:12.634495 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:12.634422 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7849-bd5129"] Sep 17 23:28:12.638054 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:28:12.638030 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba7187e1_9893_4402_88eb_8c6d47a70bfb.slice/crio-25195e48a28145604ea0bd6728bd6d5cb9cba94e3b4470c15ad376a58da75f54 WatchSource:0}: Error finding container 25195e48a28145604ea0bd6728bd6d5cb9cba94e3b4470c15ad376a58da75f54: Status 404 returned error can't find the container with id 25195e48a28145604ea0bd6728bd6d5cb9cba94e3b4470c15ad376a58da75f54 Sep 17 23:28:13.335292 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:13.335262 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7849-bd5129" event={"ID":"ba7187e1-9893-4402-88eb-8c6d47a70bfb","Type":"ContainerStarted","Data":"b76b1e24e3fe042d12ed63b5072ceb39a78f7245200f5671be1a3741446eb555"} Sep 17 23:28:13.335751 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:13.335303 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7849-bd5129" event={"ID":"ba7187e1-9893-4402-88eb-8c6d47a70bfb","Type":"ContainerStarted","Data":"25195e48a28145604ea0bd6728bd6d5cb9cba94e3b4470c15ad376a58da75f54"} Sep 17 23:28:13.335751 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:13.335334 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7849-874343" podUID="7b46f501-ec92-4c1a-b395-735598bcdcb5" containerName="test-curl-7849-874343" containerID="cri-o://a1661aa44f2a1077835508a14e1d0c572d2fe0fc7fbf0f05f7c2c1d93673e085" gracePeriod=2 Sep 17 23:28:13.349004 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:13.348963 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7849-bd5129" podStartSLOduration=1.348949872 podStartE2EDuration="1.348949872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:28:13.347593872 +0000 UTC m=+1332.281501007" watchObservedRunningTime="2026-09-17 23:28:13.348949872 +0000 UTC m=+1332.282857007" Sep 17 23:28:13.954716 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:13.954683 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7849-bd5129"] Sep 17 23:28:13.958533 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:13.958505 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7849-bd5129"] Sep 17 23:28:14.341024 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:28:14.340994 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7849-bd5129" container="test-curl-7849-bd5129" Sep 17 23:28:14.834038 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:14.834002 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd"] Sep 17 23:28:14.834744 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:14.834720 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba7187e1-9893-4402-88eb-8c6d47a70bfb" containerName="test-curl-7849-bd5129" Sep 17 23:28:14.834900 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:14.834747 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba7187e1-9893-4402-88eb-8c6d47a70bfb" containerName="test-curl-7849-bd5129" Sep 17 23:28:14.834900 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:14.834860 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba7187e1-9893-4402-88eb-8c6d47a70bfb" containerName="test-curl-7849-bd5129" Sep 17 23:28:14.837769 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:14.837750 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:14.840002 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:14.839983 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1b9e74c5-openshift-default-dockercfg-j4x7z\"" Sep 17 23:28:14.848058 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:14.848027 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd"] Sep 17 23:28:14.958894 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:14.958825 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffgcj\" (UniqueName: \"kubernetes.io/projected/b4e280b6-ecba-441b-bf98-5677fdd4407b-kube-api-access-ffgcj\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:14.958894 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:14.958865 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-data\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:14.958894 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:14.958885 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-token\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:14.959112 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:14.958964 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-envoy\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:14.959112 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:14.959004 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-workload-certs\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:14.959112 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:14.959032 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e280b6-ecba-441b-bf98-5677fdd4407b-istiod-ca-cert\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:14.959112 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:14.959101 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-credential-socket\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:14.959241 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:14.959175 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-workload-socket\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:14.959241 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:14.959193 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-podinfo\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.060467 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.060434 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-envoy\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.060467 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.060472 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-workload-certs\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.060674 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.060494 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e280b6-ecba-441b-bf98-5677fdd4407b-istiod-ca-cert\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.060674 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.060564 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-credential-socket\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.060674 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.060634 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-workload-socket\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.060674 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.060659 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-podinfo\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.060873 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.060702 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffgcj\" (UniqueName: \"kubernetes.io/projected/b4e280b6-ecba-441b-bf98-5677fdd4407b-kube-api-access-ffgcj\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.060873 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.060737 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-data\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.060873 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.060761 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-token\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.060873 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.060827 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-workload-certs\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.061078 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.061008 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-workload-socket\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.061195 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.061178 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-credential-socket\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.061258 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.061235 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-data\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.061317 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.061267 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e280b6-ecba-441b-bf98-5677fdd4407b-istiod-ca-cert\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.062791 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.062771 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-envoy\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.063125 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.063104 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-podinfo\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.073251 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.073204 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffgcj\" (UniqueName: \"kubernetes.io/projected/b4e280b6-ecba-441b-bf98-5677fdd4407b-kube-api-access-ffgcj\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.073831 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.073812 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-token\") pod \"e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.149678 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.149594 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:15.346427 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.346394 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7849-bd5129" podUID="ba7187e1-9893-4402-88eb-8c6d47a70bfb" containerName="test-curl-7849-bd5129" containerID="cri-o://b76b1e24e3fe042d12ed63b5072ceb39a78f7245200f5671be1a3741446eb555" gracePeriod=2 Sep 17 23:28:15.452197 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.452169 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd"] Sep 17 23:28:15.453354 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:28:15.453310 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e280b6_ecba_441b_bf98_5677fdd4407b.slice/crio-b2f835cccfce4abd95ac0ae512401f14531190765dadb6bb480c026008c429dc WatchSource:0}: Error finding container b2f835cccfce4abd95ac0ae512401f14531190765dadb6bb480c026008c429dc: Status 404 returned error can't find the container with id b2f835cccfce4abd95ac0ae512401f14531190765dadb6bb480c026008c429dc Sep 17 23:28:15.455924 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.455887 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:28:15.455997 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.455985 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:28:15.456048 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.456032 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:28:15.539483 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.539461 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-874343" Sep 17 23:28:15.667568 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.667468 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7xqj\" (UniqueName: \"kubernetes.io/projected/7b46f501-ec92-4c1a-b395-735598bcdcb5-kube-api-access-v7xqj\") pod \"7b46f501-ec92-4c1a-b395-735598bcdcb5\" (UID: \"7b46f501-ec92-4c1a-b395-735598bcdcb5\") " Sep 17 23:28:15.669954 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.669922 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b46f501-ec92-4c1a-b395-735598bcdcb5-kube-api-access-v7xqj" (OuterVolumeSpecName: "kube-api-access-v7xqj") pod "7b46f501-ec92-4c1a-b395-735598bcdcb5" (UID: "7b46f501-ec92-4c1a-b395-735598bcdcb5"). InnerVolumeSpecName "kube-api-access-v7xqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:28:15.768619 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:15.768580 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7xqj\" (UniqueName: \"kubernetes.io/projected/7b46f501-ec92-4c1a-b395-735598bcdcb5-kube-api-access-v7xqj\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:28:16.353057 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:16.352977 2582 generic.go:356] "Generic (PLEG): container finished" podID="7b46f501-ec92-4c1a-b395-735598bcdcb5" containerID="a1661aa44f2a1077835508a14e1d0c572d2fe0fc7fbf0f05f7c2c1d93673e085" exitCode=137 Sep 17 23:28:16.353057 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:16.353015 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-874343" Sep 17 23:28:16.353532 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:16.353073 2582 scope.go:117] "RemoveContainer" containerID="a1661aa44f2a1077835508a14e1d0c572d2fe0fc7fbf0f05f7c2c1d93673e085" Sep 17 23:28:16.354871 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:16.354847 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" event={"ID":"b4e280b6-ecba-441b-bf98-5677fdd4407b","Type":"ContainerStarted","Data":"b194c08771dad43d02e070a00ef4f621b1bf7cc49b74ace13310d0d3036def13"} Sep 17 23:28:16.355097 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:16.354898 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" event={"ID":"b4e280b6-ecba-441b-bf98-5677fdd4407b","Type":"ContainerStarted","Data":"b2f835cccfce4abd95ac0ae512401f14531190765dadb6bb480c026008c429dc"} Sep 17 23:28:16.362190 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:16.362175 2582 scope.go:117] "RemoveContainer" containerID="a1661aa44f2a1077835508a14e1d0c572d2fe0fc7fbf0f05f7c2c1d93673e085" Sep 17 23:28:16.362450 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:28:16.362433 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1661aa44f2a1077835508a14e1d0c572d2fe0fc7fbf0f05f7c2c1d93673e085\": container with ID starting with a1661aa44f2a1077835508a14e1d0c572d2fe0fc7fbf0f05f7c2c1d93673e085 not found: ID does not exist" containerID="a1661aa44f2a1077835508a14e1d0c572d2fe0fc7fbf0f05f7c2c1d93673e085" Sep 17 23:28:16.362517 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:16.362457 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1661aa44f2a1077835508a14e1d0c572d2fe0fc7fbf0f05f7c2c1d93673e085"} err="failed to get container status \"a1661aa44f2a1077835508a14e1d0c572d2fe0fc7fbf0f05f7c2c1d93673e085\": rpc error: code = NotFound desc = could not find container \"a1661aa44f2a1077835508a14e1d0c572d2fe0fc7fbf0f05f7c2c1d93673e085\": container with ID starting with a1661aa44f2a1077835508a14e1d0c572d2fe0fc7fbf0f05f7c2c1d93673e085 not found: ID does not exist" Sep 17 23:28:16.375883 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:16.375828 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" podStartSLOduration=2.375812736 podStartE2EDuration="2.375812736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:28:16.372038226 +0000 UTC m=+1335.305945364" watchObservedRunningTime="2026-09-17 23:28:16.375812736 +0000 UTC m=+1335.309719872" Sep 17 23:28:16.376252 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:28:16.376209 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7849-874343\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="7b46f501-ec92-4c1a-b395-735598bcdcb5" pod="opendatahub/test-curl-7849-874343" Sep 17 23:28:17.149942 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:17.149912 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:17.155371 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:17.155349 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:17.157461 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:28:17.157436 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7849-874343\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="7b46f501-ec92-4c1a-b395-735598bcdcb5" pod="opendatahub/test-curl-7849-874343" Sep 17 23:28:17.298177 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:17.298135 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b46f501-ec92-4c1a-b395-735598bcdcb5" path="/var/lib/kubelet/pods/7b46f501-ec92-4c1a-b395-735598bcdcb5/volumes" Sep 17 23:28:17.363847 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:17.363822 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:17.364722 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:17.364703 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:17.400729 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:28:17.400663 2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba7187e1_9893_4402_88eb_8c6d47a70bfb.slice/crio-b76b1e24e3fe042d12ed63b5072ceb39a78f7245200f5671be1a3741446eb555.scope\": RecentStats: unable to find data in memory cache]" Sep 17 23:28:17.603750 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:17.603706 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-bd5129" Sep 17 23:28:17.698069 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:17.697968 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgzh2\" (UniqueName: \"kubernetes.io/projected/ba7187e1-9893-4402-88eb-8c6d47a70bfb-kube-api-access-bgzh2\") pod \"ba7187e1-9893-4402-88eb-8c6d47a70bfb\" (UID: \"ba7187e1-9893-4402-88eb-8c6d47a70bfb\") " Sep 17 23:28:17.700435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:17.700407 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7187e1-9893-4402-88eb-8c6d47a70bfb-kube-api-access-bgzh2" (OuterVolumeSpecName: "kube-api-access-bgzh2") pod "ba7187e1-9893-4402-88eb-8c6d47a70bfb" (UID: "ba7187e1-9893-4402-88eb-8c6d47a70bfb"). InnerVolumeSpecName "kube-api-access-bgzh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:28:17.799688 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:17.799658 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgzh2\" (UniqueName: \"kubernetes.io/projected/ba7187e1-9893-4402-88eb-8c6d47a70bfb-kube-api-access-bgzh2\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:28:18.375519 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:18.372102 2582 generic.go:356] "Generic (PLEG): container finished" podID="ba7187e1-9893-4402-88eb-8c6d47a70bfb" containerID="b76b1e24e3fe042d12ed63b5072ceb39a78f7245200f5671be1a3741446eb555" exitCode=137 Sep 17 23:28:18.375519 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:18.373006 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7849-bd5129" Sep 17 23:28:18.388511 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:18.388334 2582 scope.go:117] "RemoveContainer" containerID="b76b1e24e3fe042d12ed63b5072ceb39a78f7245200f5671be1a3741446eb555" Sep 17 23:28:18.391940 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:28:18.391907 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7849-bd5129\" is forbidden: User \"system:node:ip-10-0-143-82.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-82.ec2.internal' and this object" podUID="ba7187e1-9893-4402-88eb-8c6d47a70bfb" pod="opendatahub/test-curl-7849-bd5129" Sep 17 23:28:18.407109 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:18.407086 2582 scope.go:117] "RemoveContainer" containerID="b76b1e24e3fe042d12ed63b5072ceb39a78f7245200f5671be1a3741446eb555" Sep 17 23:28:18.407521 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:28:18.407488 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b76b1e24e3fe042d12ed63b5072ceb39a78f7245200f5671be1a3741446eb555\": container with ID starting with b76b1e24e3fe042d12ed63b5072ceb39a78f7245200f5671be1a3741446eb555 not found: ID does not exist" containerID="b76b1e24e3fe042d12ed63b5072ceb39a78f7245200f5671be1a3741446eb555" Sep 17 23:28:18.407643 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:18.407529 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b76b1e24e3fe042d12ed63b5072ceb39a78f7245200f5671be1a3741446eb555"} err="failed to get container status \"b76b1e24e3fe042d12ed63b5072ceb39a78f7245200f5671be1a3741446eb555\": rpc error: code = NotFound desc = could not find container \"b76b1e24e3fe042d12ed63b5072ceb39a78f7245200f5671be1a3741446eb555\": container with ID starting with b76b1e24e3fe042d12ed63b5072ceb39a78f7245200f5671be1a3741446eb555 not found: ID does not exist" Sep 17 23:28:19.300120 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:19.299933 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba7187e1-9893-4402-88eb-8c6d47a70bfb" path="/var/lib/kubelet/pods/ba7187e1-9893-4402-88eb-8c6d47a70bfb/volumes" Sep 17 23:28:23.423964 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:23.423933 2582 generic.go:356] "Generic (PLEG): container finished" podID="dc656650-7bb4-46ee-9b96-55cc6cc7f124" containerID="20dff8cb474387c803aacce3199ac60c83df7f55994d47755d6d778e61bf3e39" exitCode=0 Sep 17 23:28:23.424353 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:23.424013 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" event={"ID":"dc656650-7bb4-46ee-9b96-55cc6cc7f124","Type":"ContainerDied","Data":"20dff8cb474387c803aacce3199ac60c83df7f55994d47755d6d778e61bf3e39"} Sep 17 23:28:23.424494 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:23.424479 2582 scope.go:117] "RemoveContainer" containerID="20dff8cb474387c803aacce3199ac60c83df7f55994d47755d6d778e61bf3e39" Sep 17 23:28:23.425647 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:23.425620 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:28:23.425749 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:23.425684 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:28:23.425749 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:23.425716 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:28:24.432001 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:24.431966 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" event={"ID":"dc656650-7bb4-46ee-9b96-55cc6cc7f124","Type":"ContainerStarted","Data":"3565dbe8e458195802c7950ad8e7b266f720da80760f50e56ac9e4d5301b7466"} Sep 17 23:28:24.881388 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:24.881348 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:24.881608 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:24.881468 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Sep 17 23:28:24.881608 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:24.881540 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" podUID="dc656650-7bb4-46ee-9b96-55cc6cc7f124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Sep 17 23:28:25.881309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:25.881259 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Sep 17 23:28:25.881808 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:25.881333 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" podUID="dc656650-7bb4-46ee-9b96-55cc6cc7f124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Sep 17 23:28:26.881906 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:26.881865 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Sep 17 23:28:26.882459 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:26.882424 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" podUID="dc656650-7bb4-46ee-9b96-55cc6cc7f124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Sep 17 23:28:27.881282 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:27.881244 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Sep 17 23:28:27.881612 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:27.881578 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" podUID="dc656650-7bb4-46ee-9b96-55cc6cc7f124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Sep 17 23:28:28.881277 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:28.881228 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Sep 17 23:28:28.881792 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:28.881308 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" podUID="dc656650-7bb4-46ee-9b96-55cc6cc7f124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Sep 17 23:28:29.893384 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:29.891977 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:29.893384 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:29.893242 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:29.896860 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:29.896831 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:28:36.503807 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:36.503775 2582 generic.go:356] "Generic (PLEG): container finished" podID="b4e280b6-ecba-441b-bf98-5677fdd4407b" containerID="b194c08771dad43d02e070a00ef4f621b1bf7cc49b74ace13310d0d3036def13" exitCode=0 Sep 17 23:28:36.504199 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:36.503848 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" event={"ID":"b4e280b6-ecba-441b-bf98-5677fdd4407b","Type":"ContainerDied","Data":"b194c08771dad43d02e070a00ef4f621b1bf7cc49b74ace13310d0d3036def13"} Sep 17 23:28:36.504246 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:36.504210 2582 scope.go:117] "RemoveContainer" containerID="b194c08771dad43d02e070a00ef4f621b1bf7cc49b74ace13310d0d3036def13" Sep 17 23:28:36.505268 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:36.505243 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:28:36.505320 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:36.505305 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:28:36.505358 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:36.505334 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:28:37.511886 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:37.511856 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" event={"ID":"b4e280b6-ecba-441b-bf98-5677fdd4407b","Type":"ContainerStarted","Data":"691d207d5186816a2c4e0e04bc4b0666987314c86984128c6780efc6c71677cf"} Sep 17 23:28:38.150388 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:38.150356 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:38.150641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:38.150613 2582 patch_prober.go:28] interesting pod/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Sep 17 23:28:38.150743 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:38.150686 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" podUID="b4e280b6-ecba-441b-bf98-5677fdd4407b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Sep 17 23:28:39.150461 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:39.150430 2582 patch_prober.go:28] interesting pod/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Sep 17 23:28:39.150863 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:39.150486 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" podUID="b4e280b6-ecba-441b-bf98-5677fdd4407b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Sep 17 23:28:40.150274 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:40.150231 2582 patch_prober.go:28] interesting pod/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Sep 17 23:28:40.150446 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:40.150316 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" podUID="b4e280b6-ecba-441b-bf98-5677fdd4407b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Sep 17 23:28:41.150417 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:41.150379 2582 patch_prober.go:28] interesting pod/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Sep 17 23:28:41.150867 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:41.150445 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" podUID="b4e280b6-ecba-441b-bf98-5677fdd4407b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Sep 17 23:28:42.150087 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:42.150050 2582 patch_prober.go:28] interesting pod/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Sep 17 23:28:42.150275 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:42.150113 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" podUID="b4e280b6-ecba-441b-bf98-5677fdd4407b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Sep 17 23:28:43.154396 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:43.154369 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:43.154803 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:43.154784 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:28:43.155347 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:28:43.155324 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:29:03.418421 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:03.418391 2582 scope.go:117] "RemoveContainer" containerID="d947b532307bd80c293b763db48326042fa286b5018a764d12bf8ccf52c29409" Sep 17 23:29:03.427018 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:03.427002 2582 scope.go:117] "RemoveContainer" containerID="6fa12bb9cbbaf2ab616610c02bec0ef6922dec73a0a5d93da622e6584b9bb231" Sep 17 23:29:53.791325 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:53.791288 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb"] Sep 17 23:29:53.793851 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:53.793835 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb" Sep 17 23:29:53.795996 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:53.795976 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:29:53.801713 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:53.801236 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb"] Sep 17 23:29:53.917279 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:53.917248 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f6771ac-7373-4513-bc15-70411b44ae50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb\" (UID: \"6f6771ac-7373-4513-bc15-70411b44ae50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb" Sep 17 23:29:54.018335 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:54.018300 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f6771ac-7373-4513-bc15-70411b44ae50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb\" (UID: \"6f6771ac-7373-4513-bc15-70411b44ae50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb" Sep 17 23:29:54.018919 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:54.018900 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f6771ac-7373-4513-bc15-70411b44ae50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb\" (UID: \"6f6771ac-7373-4513-bc15-70411b44ae50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb" Sep 17 23:29:54.106024 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:54.105957 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb" Sep 17 23:29:54.375014 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:54.374985 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb"] Sep 17 23:29:54.377870 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:29:54.377827 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6771ac_7373_4513_bc15_70411b44ae50.slice/crio-890631f9501b2409f05810ea0119a13fd50f69fac69d5771ef606fd876c881eb WatchSource:0}: Error finding container 890631f9501b2409f05810ea0119a13fd50f69fac69d5771ef606fd876c881eb: Status 404 returned error can't find the container with id 890631f9501b2409f05810ea0119a13fd50f69fac69d5771ef606fd876c881eb Sep 17 23:29:54.947080 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:54.947039 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb" event={"ID":"6f6771ac-7373-4513-bc15-70411b44ae50","Type":"ContainerStarted","Data":"62e7a433e0097cc8b60911cea4828737a1ec8912532b4438cfd4750b75a3b2fe"} Sep 17 23:29:54.947442 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:54.947105 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb" event={"ID":"6f6771ac-7373-4513-bc15-70411b44ae50","Type":"ContainerStarted","Data":"890631f9501b2409f05810ea0119a13fd50f69fac69d5771ef606fd876c881eb"} Sep 17 23:29:54.963105 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:54.963062 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb" podStartSLOduration=1.963047687 podStartE2EDuration="1.963047687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:29:54.961703456 +0000 UTC m=+1433.895610591" watchObservedRunningTime="2026-09-17 23:29:54.963047687 +0000 UTC m=+1433.896954820" Sep 17 23:29:55.955839 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:55.955800 2582 generic.go:356] "Generic (PLEG): container finished" podID="6f6771ac-7373-4513-bc15-70411b44ae50" containerID="62e7a433e0097cc8b60911cea4828737a1ec8912532b4438cfd4750b75a3b2fe" exitCode=0 Sep 17 23:29:55.956204 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:55.955853 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb" event={"ID":"6f6771ac-7373-4513-bc15-70411b44ae50","Type":"ContainerDied","Data":"62e7a433e0097cc8b60911cea4828737a1ec8912532b4438cfd4750b75a3b2fe"} Sep 17 23:29:57.118164 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:57.117975 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb" Sep 17 23:29:57.252605 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:57.252556 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f6771ac-7373-4513-bc15-70411b44ae50-maas-api-service-ca\") pod \"6f6771ac-7373-4513-bc15-70411b44ae50\" (UID: \"6f6771ac-7373-4513-bc15-70411b44ae50\") " Sep 17 23:29:57.252878 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:57.252852 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f6771ac-7373-4513-bc15-70411b44ae50-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6f6771ac-7373-4513-bc15-70411b44ae50" (UID: "6f6771ac-7373-4513-bc15-70411b44ae50"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:29:57.353423 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:57.353394 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f6771ac-7373-4513-bc15-70411b44ae50-maas-api-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:29:57.970411 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:57.970380 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb" Sep 17 23:29:57.970641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:57.970417 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb" event={"ID":"6f6771ac-7373-4513-bc15-70411b44ae50","Type":"ContainerDied","Data":"890631f9501b2409f05810ea0119a13fd50f69fac69d5771ef606fd876c881eb"} Sep 17 23:29:57.970641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:29:57.970467 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="890631f9501b2409f05810ea0119a13fd50f69fac69d5771ef606fd876c881eb" Sep 17 23:30:00.135490 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.135459 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1b9e74c5-29828130-8pln7"] Sep 17 23:30:00.135914 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.135901 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f6771ac-7373-4513-bc15-70411b44ae50" containerName="revoke-keys" Sep 17 23:30:00.135959 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.135915 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f6771ac-7373-4513-bc15-70411b44ae50" containerName="revoke-keys" Sep 17 23:30:00.136005 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.135996 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f6771ac-7373-4513-bc15-70411b44ae50" containerName="revoke-keys" Sep 17 23:30:00.138474 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.138450 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1b9e74c5-29828130-8pln7" Sep 17 23:30:00.139896 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.139866 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d90df4-29828130-v78x7"] Sep 17 23:30:00.140590 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.140566 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:30:00.142427 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.142406 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828130-k86l8"] Sep 17 23:30:00.142565 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.142521 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d90df4-29828130-v78x7" Sep 17 23:30:00.144953 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.144929 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828130-k86l8" Sep 17 23:30:00.146156 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.146133 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1b9e74c5-29828130-8pln7"] Sep 17 23:30:00.152656 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.152626 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1b9e74c5-29828130-8pln7" Sep 17 23:30:00.153131 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.153100 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d90df4-29828130-v78x7"] Sep 17 23:30:00.157062 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.157039 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828130-k86l8"] Sep 17 23:30:00.168683 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.168654 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d90df4-29828130-v78x7" Sep 17 23:30:00.212794 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.212769 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828130-k86l8" Sep 17 23:30:00.446737 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.446707 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1b9e74c5-29828130-8pln7"] Sep 17 23:30:00.447955 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:30:00.447931 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704a0d11_cc5f_4f77_b918_597efd54a022.slice/crio-094ad5f58e169b6e4d27303074d92d968ea54eef080aef3a539a833d65fc0991 WatchSource:0}: Error finding container 094ad5f58e169b6e4d27303074d92d968ea54eef080aef3a539a833d65fc0991: Status 404 returned error can't find the container with id 094ad5f58e169b6e4d27303074d92d968ea54eef080aef3a539a833d65fc0991 Sep 17 23:30:00.458230 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.458211 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d90df4-29828130-v78x7"] Sep 17 23:30:00.461128 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:30:00.461097 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9318ac_fd7c_41cc_a573_a91a09360db2.slice/crio-4b6389adf0c0c32fc819b10af8a1c754351cabaceeeaeede3c8b82c406d93578 WatchSource:0}: Error finding container 4b6389adf0c0c32fc819b10af8a1c754351cabaceeeaeede3c8b82c406d93578: Status 404 returned error can't find the container with id 4b6389adf0c0c32fc819b10af8a1c754351cabaceeeaeede3c8b82c406d93578 Sep 17 23:30:00.514320 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.514299 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828130-k86l8"] Sep 17 23:30:00.515694 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:30:00.515672 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59306d16_3b7d_44be_8a9a_360276bdedb0.slice/crio-22fedaab656838c52a9add3fa3a5e3b0a9c248547f120731a1b58fc7d1fb29b2 WatchSource:0}: Error finding container 22fedaab656838c52a9add3fa3a5e3b0a9c248547f120731a1b58fc7d1fb29b2: Status 404 returned error can't find the container with id 22fedaab656838c52a9add3fa3a5e3b0a9c248547f120731a1b58fc7d1fb29b2 Sep 17 23:30:00.991177 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.991147 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1b9e74c5-29828130-8pln7" event={"ID":"704a0d11-cc5f-4f77-b918-597efd54a022","Type":"ContainerStarted","Data":"11e09570053bf1a0cad613d5f30c70184afbad137d779976f8660cd673c48772"} Sep 17 23:30:00.991357 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.991207 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1b9e74c5-29828130-8pln7" event={"ID":"704a0d11-cc5f-4f77-b918-597efd54a022","Type":"ContainerStarted","Data":"094ad5f58e169b6e4d27303074d92d968ea54eef080aef3a539a833d65fc0991"} Sep 17 23:30:00.992956 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.992927 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d90df4-29828130-v78x7" event={"ID":"cc9318ac-fd7c-41cc-a573-a91a09360db2","Type":"ContainerStarted","Data":"4e0510d469b57870ceae88b5c8efca3d537cbe299a4ac24c824f5facc3f3f6cd"} Sep 17 23:30:00.993096 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.992972 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d90df4-29828130-v78x7" event={"ID":"cc9318ac-fd7c-41cc-a573-a91a09360db2","Type":"ContainerStarted","Data":"4b6389adf0c0c32fc819b10af8a1c754351cabaceeeaeede3c8b82c406d93578"} Sep 17 23:30:00.994441 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.994419 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828130-k86l8" event={"ID":"59306d16-3b7d-44be-8a9a-360276bdedb0","Type":"ContainerStarted","Data":"4751eeae06c33cecc1525eb66029d305132da42a14ffc6ac3649b624eac43792"} Sep 17 23:30:00.994575 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:00.994454 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828130-k86l8" event={"ID":"59306d16-3b7d-44be-8a9a-360276bdedb0","Type":"ContainerStarted","Data":"22fedaab656838c52a9add3fa3a5e3b0a9c248547f120731a1b58fc7d1fb29b2"} Sep 17 23:30:01.004999 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:01.004958 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1b9e74c5-29828130-8pln7" podStartSLOduration=1.004945173 podStartE2EDuration="1.004945173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:01.004451012 +0000 UTC m=+1439.938358149" watchObservedRunningTime="2026-09-17 23:30:01.004945173 +0000 UTC m=+1439.938852371" Sep 17 23:30:01.019635 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:01.019592 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828130-k86l8" podStartSLOduration=1.019576807 podStartE2EDuration="1.019576807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:01.017064208 +0000 UTC m=+1439.950971342" watchObservedRunningTime="2026-09-17 23:30:01.019576807 +0000 UTC m=+1439.953483943" Sep 17 23:30:01.031202 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:01.031134 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d90df4-29828130-v78x7" podStartSLOduration=1.031121697 podStartE2EDuration="1.031121697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:01.029383929 +0000 UTC m=+1439.963291063" watchObservedRunningTime="2026-09-17 23:30:01.031121697 +0000 UTC m=+1439.965028831" Sep 17 23:30:02.001627 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:02.001596 2582 generic.go:356] "Generic (PLEG): container finished" podID="59306d16-3b7d-44be-8a9a-360276bdedb0" containerID="4751eeae06c33cecc1525eb66029d305132da42a14ffc6ac3649b624eac43792" exitCode=0 Sep 17 23:30:02.002054 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:02.001685 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828130-k86l8" event={"ID":"59306d16-3b7d-44be-8a9a-360276bdedb0","Type":"ContainerDied","Data":"4751eeae06c33cecc1525eb66029d305132da42a14ffc6ac3649b624eac43792"} Sep 17 23:30:02.003273 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:02.003248 2582 generic.go:356] "Generic (PLEG): container finished" podID="704a0d11-cc5f-4f77-b918-597efd54a022" containerID="11e09570053bf1a0cad613d5f30c70184afbad137d779976f8660cd673c48772" exitCode=0 Sep 17 23:30:02.003394 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:02.003334 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1b9e74c5-29828130-8pln7" event={"ID":"704a0d11-cc5f-4f77-b918-597efd54a022","Type":"ContainerDied","Data":"11e09570053bf1a0cad613d5f30c70184afbad137d779976f8660cd673c48772"} Sep 17 23:30:02.004859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:02.004840 2582 generic.go:356] "Generic (PLEG): container finished" podID="cc9318ac-fd7c-41cc-a573-a91a09360db2" containerID="4e0510d469b57870ceae88b5c8efca3d537cbe299a4ac24c824f5facc3f3f6cd" exitCode=0 Sep 17 23:30:02.004955 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:02.004933 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d90df4-29828130-v78x7" event={"ID":"cc9318ac-fd7c-41cc-a573-a91a09360db2","Type":"ContainerDied","Data":"4e0510d469b57870ceae88b5c8efca3d537cbe299a4ac24c824f5facc3f3f6cd"} Sep 17 23:30:03.167419 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:03.167387 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1b9e74c5-29828130-8pln7" Sep 17 23:30:03.202290 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:03.202266 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828130-k86l8" Sep 17 23:30:03.205845 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:03.205826 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d90df4-29828130-v78x7" Sep 17 23:30:03.457399 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:03.457371 2582 scope.go:117] "RemoveContainer" containerID="ef0ea5ee4e45f7cc22ae125f2fb22796b640014ea5f7aca69f5edfc84cb68f55" Sep 17 23:30:04.017772 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:04.017743 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1b9e74c5-29828130-8pln7" Sep 17 23:30:04.017954 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:04.017775 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1b9e74c5-29828130-8pln7" event={"ID":"704a0d11-cc5f-4f77-b918-597efd54a022","Type":"ContainerDied","Data":"094ad5f58e169b6e4d27303074d92d968ea54eef080aef3a539a833d65fc0991"} Sep 17 23:30:04.017954 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:04.017871 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="094ad5f58e169b6e4d27303074d92d968ea54eef080aef3a539a833d65fc0991" Sep 17 23:30:04.019209 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:04.019187 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d90df4-29828130-v78x7" Sep 17 23:30:04.019340 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:04.019261 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d90df4-29828130-v78x7" event={"ID":"cc9318ac-fd7c-41cc-a573-a91a09360db2","Type":"ContainerDied","Data":"4b6389adf0c0c32fc819b10af8a1c754351cabaceeeaeede3c8b82c406d93578"} Sep 17 23:30:04.019340 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:04.019328 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b6389adf0c0c32fc819b10af8a1c754351cabaceeeaeede3c8b82c406d93578" Sep 17 23:30:04.020750 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:04.020729 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828130-k86l8" event={"ID":"59306d16-3b7d-44be-8a9a-360276bdedb0","Type":"ContainerDied","Data":"22fedaab656838c52a9add3fa3a5e3b0a9c248547f120731a1b58fc7d1fb29b2"} Sep 17 23:30:04.020750 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:04.020744 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828130-k86l8" Sep 17 23:30:04.020901 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:04.020769 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22fedaab656838c52a9add3fa3a5e3b0a9c248547f120731a1b58fc7d1fb29b2" Sep 17 23:30:09.327767 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:09.327736 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1b9e74c5-29828130-8pln7"] Sep 17 23:30:09.333075 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:09.333048 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1b9e74c5-29828130-8pln7"] Sep 17 23:30:09.529330 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:09.529292 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb"] Sep 17 23:30:09.531309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:09.531289 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b9e74c5-fb9gb"] Sep 17 23:30:11.298453 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:11.298420 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f6771ac-7373-4513-bc15-70411b44ae50" path="/var/lib/kubelet/pods/6f6771ac-7373-4513-bc15-70411b44ae50/volumes" Sep 17 23:30:11.298881 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:11.298859 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="704a0d11-cc5f-4f77-b918-597efd54a022" path="/var/lib/kubelet/pods/704a0d11-cc5f-4f77-b918-597efd54a022/volumes" Sep 17 23:30:49.598093 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:49.598055 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd"] Sep 17 23:30:49.598718 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:49.598342 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" podUID="b4e280b6-ecba-441b-bf98-5677fdd4407b" containerName="istio-proxy" containerID="cri-o://691d207d5186816a2c4e0e04bc4b0666987314c86984128c6780efc6c71677cf" gracePeriod=30 Sep 17 23:30:54.856860 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.856836 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:30:54.890789 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.890728 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-credential-socket\") pod \"b4e280b6-ecba-441b-bf98-5677fdd4407b\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " Sep 17 23:30:54.890789 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.890759 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-data\") pod \"b4e280b6-ecba-441b-bf98-5677fdd4407b\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " Sep 17 23:30:54.890947 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.890808 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-workload-certs\") pod \"b4e280b6-ecba-441b-bf98-5677fdd4407b\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " Sep 17 23:30:54.890947 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.890843 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e280b6-ecba-441b-bf98-5677fdd4407b-istiod-ca-cert\") pod \"b4e280b6-ecba-441b-bf98-5677fdd4407b\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " Sep 17 23:30:54.890947 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.890918 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-token\") pod \"b4e280b6-ecba-441b-bf98-5677fdd4407b\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " Sep 17 23:30:54.890947 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.890944 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-workload-socket\") pod \"b4e280b6-ecba-441b-bf98-5677fdd4407b\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " Sep 17 23:30:54.891149 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.890976 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-envoy\") pod \"b4e280b6-ecba-441b-bf98-5677fdd4407b\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " Sep 17 23:30:54.891149 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.891012 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b4e280b6-ecba-441b-bf98-5677fdd4407b" (UID: "b4e280b6-ecba-441b-bf98-5677fdd4407b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:30:54.891149 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.891057 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-data" (OuterVolumeSpecName: "istio-data") pod "b4e280b6-ecba-441b-bf98-5677fdd4407b" (UID: "b4e280b6-ecba-441b-bf98-5677fdd4407b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:30:54.891149 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.891105 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b4e280b6-ecba-441b-bf98-5677fdd4407b" (UID: "b4e280b6-ecba-441b-bf98-5677fdd4407b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:30:54.891149 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.891107 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffgcj\" (UniqueName: \"kubernetes.io/projected/b4e280b6-ecba-441b-bf98-5677fdd4407b-kube-api-access-ffgcj\") pod \"b4e280b6-ecba-441b-bf98-5677fdd4407b\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " Sep 17 23:30:54.891413 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.891151 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-podinfo\") pod \"b4e280b6-ecba-441b-bf98-5677fdd4407b\" (UID: \"b4e280b6-ecba-441b-bf98-5677fdd4407b\") " Sep 17 23:30:54.891413 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.891237 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b4e280b6-ecba-441b-bf98-5677fdd4407b" (UID: "b4e280b6-ecba-441b-bf98-5677fdd4407b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:30:54.891413 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.891342 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e280b6-ecba-441b-bf98-5677fdd4407b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b4e280b6-ecba-441b-bf98-5677fdd4407b" (UID: "b4e280b6-ecba-441b-bf98-5677fdd4407b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:30:54.891613 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.891596 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-workload-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:30:54.891675 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.891620 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e280b6-ecba-441b-bf98-5677fdd4407b-istiod-ca-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:30:54.891675 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.891633 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-workload-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:30:54.891675 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.891645 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-credential-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:30:54.891675 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.891660 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-data\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:30:54.893147 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.893118 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e280b6-ecba-441b-bf98-5677fdd4407b-kube-api-access-ffgcj" (OuterVolumeSpecName: "kube-api-access-ffgcj") pod "b4e280b6-ecba-441b-bf98-5677fdd4407b" (UID: "b4e280b6-ecba-441b-bf98-5677fdd4407b"). InnerVolumeSpecName "kube-api-access-ffgcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:30:54.893540 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.893494 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b4e280b6-ecba-441b-bf98-5677fdd4407b" (UID: "b4e280b6-ecba-441b-bf98-5677fdd4407b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:30:54.893540 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.893490 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-token" (OuterVolumeSpecName: "istio-token") pod "b4e280b6-ecba-441b-bf98-5677fdd4407b" (UID: "b4e280b6-ecba-441b-bf98-5677fdd4407b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:30:54.893663 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.893533 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b4e280b6-ecba-441b-bf98-5677fdd4407b" (UID: "b4e280b6-ecba-441b-bf98-5677fdd4407b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:30:54.992306 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.992280 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:30:54.992306 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.992301 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-envoy\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:30:54.992306 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.992310 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffgcj\" (UniqueName: \"kubernetes.io/projected/b4e280b6-ecba-441b-bf98-5677fdd4407b-kube-api-access-ffgcj\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:30:54.992487 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:54.992319 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e280b6-ecba-441b-bf98-5677fdd4407b-istio-podinfo\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:30:55.317550 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:55.317514 2582 generic.go:356] "Generic (PLEG): container finished" podID="b4e280b6-ecba-441b-bf98-5677fdd4407b" containerID="691d207d5186816a2c4e0e04bc4b0666987314c86984128c6780efc6c71677cf" exitCode=0 Sep 17 23:30:55.317678 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:55.317598 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" Sep 17 23:30:55.317721 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:55.317592 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" event={"ID":"b4e280b6-ecba-441b-bf98-5677fdd4407b","Type":"ContainerDied","Data":"691d207d5186816a2c4e0e04bc4b0666987314c86984128c6780efc6c71677cf"} Sep 17 23:30:55.317805 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:55.317719 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd" event={"ID":"b4e280b6-ecba-441b-bf98-5677fdd4407b","Type":"ContainerDied","Data":"b2f835cccfce4abd95ac0ae512401f14531190765dadb6bb480c026008c429dc"} Sep 17 23:30:55.317805 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:55.317747 2582 scope.go:117] "RemoveContainer" containerID="691d207d5186816a2c4e0e04bc4b0666987314c86984128c6780efc6c71677cf" Sep 17 23:30:55.327364 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:55.327346 2582 scope.go:117] "RemoveContainer" containerID="b194c08771dad43d02e070a00ef4f621b1bf7cc49b74ace13310d0d3036def13" Sep 17 23:30:55.336381 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:55.336361 2582 scope.go:117] "RemoveContainer" containerID="691d207d5186816a2c4e0e04bc4b0666987314c86984128c6780efc6c71677cf" Sep 17 23:30:55.336687 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:30:55.336669 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"691d207d5186816a2c4e0e04bc4b0666987314c86984128c6780efc6c71677cf\": container with ID starting with 691d207d5186816a2c4e0e04bc4b0666987314c86984128c6780efc6c71677cf not found: ID does not exist" containerID="691d207d5186816a2c4e0e04bc4b0666987314c86984128c6780efc6c71677cf" Sep 17 23:30:55.336753 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:55.336694 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"691d207d5186816a2c4e0e04bc4b0666987314c86984128c6780efc6c71677cf"} err="failed to get container status \"691d207d5186816a2c4e0e04bc4b0666987314c86984128c6780efc6c71677cf\": rpc error: code = NotFound desc = could not find container \"691d207d5186816a2c4e0e04bc4b0666987314c86984128c6780efc6c71677cf\": container with ID starting with 691d207d5186816a2c4e0e04bc4b0666987314c86984128c6780efc6c71677cf not found: ID does not exist" Sep 17 23:30:55.336753 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:55.336712 2582 scope.go:117] "RemoveContainer" containerID="b194c08771dad43d02e070a00ef4f621b1bf7cc49b74ace13310d0d3036def13" Sep 17 23:30:55.337003 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:30:55.336985 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b194c08771dad43d02e070a00ef4f621b1bf7cc49b74ace13310d0d3036def13\": container with ID starting with b194c08771dad43d02e070a00ef4f621b1bf7cc49b74ace13310d0d3036def13 not found: ID does not exist" containerID="b194c08771dad43d02e070a00ef4f621b1bf7cc49b74ace13310d0d3036def13" Sep 17 23:30:55.337046 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:55.337009 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b194c08771dad43d02e070a00ef4f621b1bf7cc49b74ace13310d0d3036def13"} err="failed to get container status \"b194c08771dad43d02e070a00ef4f621b1bf7cc49b74ace13310d0d3036def13\": rpc error: code = NotFound desc = could not find container \"b194c08771dad43d02e070a00ef4f621b1bf7cc49b74ace13310d0d3036def13\": container with ID starting with b194c08771dad43d02e070a00ef4f621b1bf7cc49b74ace13310d0d3036def13 not found: ID does not exist" Sep 17 23:30:55.337222 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:55.337206 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd"] Sep 17 23:30:55.340161 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:55.340142 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1b9e74c5-openshift-default-9c7fd5d84-vkccd"] Sep 17 23:30:57.294587 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:30:57.294558 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4e280b6-ecba-441b-bf98-5677fdd4407b" path="/var/lib/kubelet/pods/b4e280b6-ecba-441b-bf98-5677fdd4407b/volumes" Sep 17 23:31:03.483429 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:03.483397 2582 scope.go:117] "RemoveContainer" containerID="9eb62ae29e9cc038b179a7b250e343dda82b32a6cbc4401aa93efe3c74af67e4" Sep 17 23:31:03.493782 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:03.493761 2582 scope.go:117] "RemoveContainer" containerID="8abbe3e2c39a66a972fbb13412389de682173593398c84f1a7a9b384a669b1a4" Sep 17 23:31:03.501596 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:03.501579 2582 scope.go:117] "RemoveContainer" containerID="3910967a5e5eb6dd09a69bd7a9afbaa9e32537360f794f8f769a2173b83d5993" Sep 17 23:31:37.062798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.062767 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7852-f8c1e6"] Sep 17 23:31:37.063263 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.063194 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59306d16-3b7d-44be-8a9a-360276bdedb0" containerName="cleanup" Sep 17 23:31:37.063263 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.063205 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="59306d16-3b7d-44be-8a9a-360276bdedb0" containerName="cleanup" Sep 17 23:31:37.063263 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.063218 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4e280b6-ecba-441b-bf98-5677fdd4407b" containerName="istio-proxy" Sep 17 23:31:37.063263 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.063223 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4e280b6-ecba-441b-bf98-5677fdd4407b" containerName="istio-proxy" Sep 17 23:31:37.063263 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.063231 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4e280b6-ecba-441b-bf98-5677fdd4407b" containerName="istio-proxy" Sep 17 23:31:37.063263 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.063236 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4e280b6-ecba-441b-bf98-5677fdd4407b" containerName="istio-proxy" Sep 17 23:31:37.063263 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.063243 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="704a0d11-cc5f-4f77-b918-597efd54a022" containerName="cleanup" Sep 17 23:31:37.063263 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.063248 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="704a0d11-cc5f-4f77-b918-597efd54a022" containerName="cleanup" Sep 17 23:31:37.063263 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.063259 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9318ac-fd7c-41cc-a573-a91a09360db2" containerName="cleanup" Sep 17 23:31:37.063263 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.063264 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9318ac-fd7c-41cc-a573-a91a09360db2" containerName="cleanup" Sep 17 23:31:37.063639 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.063323 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="59306d16-3b7d-44be-8a9a-360276bdedb0" containerName="cleanup" Sep 17 23:31:37.063639 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.063331 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9318ac-fd7c-41cc-a573-a91a09360db2" containerName="cleanup" Sep 17 23:31:37.063639 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.063339 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4e280b6-ecba-441b-bf98-5677fdd4407b" containerName="istio-proxy" Sep 17 23:31:37.063639 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.063346 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="704a0d11-cc5f-4f77-b918-597efd54a022" containerName="cleanup" Sep 17 23:31:37.066443 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.066426 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-f8c1e6" Sep 17 23:31:37.068852 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.068818 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m6p2p\"" Sep 17 23:31:37.071025 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.071000 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-f8c1e6"] Sep 17 23:31:37.183816 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.183771 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvx9w\" (UniqueName: \"kubernetes.io/projected/c8f88265-41af-434f-be5b-f7f0b8f6f6d3-kube-api-access-qvx9w\") pod \"test-curl-7852-f8c1e6\" (UID: \"c8f88265-41af-434f-be5b-f7f0b8f6f6d3\") " pod="opendatahub/test-curl-7852-f8c1e6" Sep 17 23:31:37.284851 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.284813 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvx9w\" (UniqueName: \"kubernetes.io/projected/c8f88265-41af-434f-be5b-f7f0b8f6f6d3-kube-api-access-qvx9w\") pod \"test-curl-7852-f8c1e6\" (UID: \"c8f88265-41af-434f-be5b-f7f0b8f6f6d3\") " pod="opendatahub/test-curl-7852-f8c1e6" Sep 17 23:31:37.292010 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.291988 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvx9w\" (UniqueName: \"kubernetes.io/projected/c8f88265-41af-434f-be5b-f7f0b8f6f6d3-kube-api-access-qvx9w\") pod \"test-curl-7852-f8c1e6\" (UID: \"c8f88265-41af-434f-be5b-f7f0b8f6f6d3\") " pod="opendatahub/test-curl-7852-f8c1e6" Sep 17 23:31:37.378441 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.378360 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-f8c1e6" Sep 17 23:31:37.647784 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:37.647755 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-f8c1e6"] Sep 17 23:31:37.649840 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:31:37.649809 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8f88265_41af_434f_be5b_f7f0b8f6f6d3.slice/crio-bb59e7fba9babf967ddbe6b3ec05d8bbd0a30cb8a3e0856e7bfa9f142de1b88a WatchSource:0}: Error finding container bb59e7fba9babf967ddbe6b3ec05d8bbd0a30cb8a3e0856e7bfa9f142de1b88a: Status 404 returned error can't find the container with id bb59e7fba9babf967ddbe6b3ec05d8bbd0a30cb8a3e0856e7bfa9f142de1b88a Sep 17 23:31:38.565037 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:38.565001 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-f8c1e6" event={"ID":"c8f88265-41af-434f-be5b-f7f0b8f6f6d3","Type":"ContainerStarted","Data":"cf853418b0205465e5ea9fec715ecf73239e2fcabc44fe12f9f281d967cb66d7"} Sep 17 23:31:38.565037 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:38.565039 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-f8c1e6" event={"ID":"c8f88265-41af-434f-be5b-f7f0b8f6f6d3","Type":"ContainerStarted","Data":"bb59e7fba9babf967ddbe6b3ec05d8bbd0a30cb8a3e0856e7bfa9f142de1b88a"} Sep 17 23:31:38.578797 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:38.578751 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7852-f8c1e6" podStartSLOduration=1.578734554 podStartE2EDuration="1.578734554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:31:38.576820999 +0000 UTC m=+1537.510728126" watchObservedRunningTime="2026-09-17 23:31:38.578734554 +0000 UTC m=+1537.512641693" Sep 17 23:31:39.231555 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:39.231522 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7852-f8c1e6"] Sep 17 23:31:39.238751 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:39.238680 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7852-f8c1e6"] Sep 17 23:31:39.572040 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:31:39.571961 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7852-f8c1e6" container="test-curl-7852-f8c1e6" Sep 17 23:31:40.577736 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:40.577690 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7852-f8c1e6" podUID="c8f88265-41af-434f-be5b-f7f0b8f6f6d3" containerName="test-curl-7852-f8c1e6" containerID="cri-o://cf853418b0205465e5ea9fec715ecf73239e2fcabc44fe12f9f281d967cb66d7" gracePeriod=2 Sep 17 23:31:42.755105 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:42.755077 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-f8c1e6" Sep 17 23:31:42.837960 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:42.837876 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvx9w\" (UniqueName: \"kubernetes.io/projected/c8f88265-41af-434f-be5b-f7f0b8f6f6d3-kube-api-access-qvx9w\") pod \"c8f88265-41af-434f-be5b-f7f0b8f6f6d3\" (UID: \"c8f88265-41af-434f-be5b-f7f0b8f6f6d3\") " Sep 17 23:31:42.839937 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:42.839911 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f88265-41af-434f-be5b-f7f0b8f6f6d3-kube-api-access-qvx9w" (OuterVolumeSpecName: "kube-api-access-qvx9w") pod "c8f88265-41af-434f-be5b-f7f0b8f6f6d3" (UID: "c8f88265-41af-434f-be5b-f7f0b8f6f6d3"). InnerVolumeSpecName "kube-api-access-qvx9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:31:42.938817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:42.938778 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvx9w\" (UniqueName: \"kubernetes.io/projected/c8f88265-41af-434f-be5b-f7f0b8f6f6d3-kube-api-access-qvx9w\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:31:43.295112 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:43.295081 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8f88265-41af-434f-be5b-f7f0b8f6f6d3" path="/var/lib/kubelet/pods/c8f88265-41af-434f-be5b-f7f0b8f6f6d3/volumes" Sep 17 23:31:43.596610 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:43.596520 2582 generic.go:356] "Generic (PLEG): container finished" podID="c8f88265-41af-434f-be5b-f7f0b8f6f6d3" containerID="cf853418b0205465e5ea9fec715ecf73239e2fcabc44fe12f9f281d967cb66d7" exitCode=137 Sep 17 23:31:43.596610 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:43.596555 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-f8c1e6" Sep 17 23:31:43.596610 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:43.596580 2582 scope.go:117] "RemoveContainer" containerID="cf853418b0205465e5ea9fec715ecf73239e2fcabc44fe12f9f281d967cb66d7" Sep 17 23:31:43.605229 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:43.605203 2582 scope.go:117] "RemoveContainer" containerID="cf853418b0205465e5ea9fec715ecf73239e2fcabc44fe12f9f281d967cb66d7" Sep 17 23:31:43.605491 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:31:43.605465 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf853418b0205465e5ea9fec715ecf73239e2fcabc44fe12f9f281d967cb66d7\": container with ID starting with cf853418b0205465e5ea9fec715ecf73239e2fcabc44fe12f9f281d967cb66d7 not found: ID does not exist" containerID="cf853418b0205465e5ea9fec715ecf73239e2fcabc44fe12f9f281d967cb66d7" Sep 17 23:31:43.605566 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:43.605514 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf853418b0205465e5ea9fec715ecf73239e2fcabc44fe12f9f281d967cb66d7"} err="failed to get container status \"cf853418b0205465e5ea9fec715ecf73239e2fcabc44fe12f9f281d967cb66d7\": rpc error: code = NotFound desc = could not find container \"cf853418b0205465e5ea9fec715ecf73239e2fcabc44fe12f9f281d967cb66d7\": container with ID starting with cf853418b0205465e5ea9fec715ecf73239e2fcabc44fe12f9f281d967cb66d7 not found: ID does not exist" Sep 17 23:31:44.023150 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.023115 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8"] Sep 17 23:31:44.023845 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.023825 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8f88265-41af-434f-be5b-f7f0b8f6f6d3" containerName="test-curl-7852-f8c1e6" Sep 17 23:31:44.023900 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.023850 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8f88265-41af-434f-be5b-f7f0b8f6f6d3" containerName="test-curl-7852-f8c1e6" Sep 17 23:31:44.024004 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.023991 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4e280b6-ecba-441b-bf98-5677fdd4407b" containerName="istio-proxy" Sep 17 23:31:44.024038 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.024011 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8f88265-41af-434f-be5b-f7f0b8f6f6d3" containerName="test-curl-7852-f8c1e6" Sep 17 23:31:44.028991 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.028972 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.031196 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.031175 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-78432f-openshift-default-dockercfg-lcrbw\"" Sep 17 23:31:44.037266 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.037239 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8"] Sep 17 23:31:44.149539 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.149480 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-workload-socket\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.149709 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.149588 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-credential-socket\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.149709 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.149633 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-workload-certs\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.149790 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.149709 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b44980-d812-40eb-b4ad-87d4af154876-istiod-ca-cert\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.149790 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.149732 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-istio-data\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.149790 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.149749 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx6xc\" (UniqueName: \"kubernetes.io/projected/a4b44980-d812-40eb-b4ad-87d4af154876-kube-api-access-sx6xc\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.149790 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.149784 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b44980-d812-40eb-b4ad-87d4af154876-istio-token\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.149914 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.149812 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b44980-d812-40eb-b4ad-87d4af154876-istio-podinfo\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.149914 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.149839 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-istio-envoy\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.250847 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.250806 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b44980-d812-40eb-b4ad-87d4af154876-istiod-ca-cert\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.251028 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.250871 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-istio-data\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.251028 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.250903 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx6xc\" (UniqueName: \"kubernetes.io/projected/a4b44980-d812-40eb-b4ad-87d4af154876-kube-api-access-sx6xc\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.251028 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.250958 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b44980-d812-40eb-b4ad-87d4af154876-istio-token\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.251028 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.250990 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b44980-d812-40eb-b4ad-87d4af154876-istio-podinfo\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.251222 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.251030 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-istio-envoy\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.251222 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.251063 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-workload-socket\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.251222 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.251113 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-credential-socket\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.251222 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.251159 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-workload-certs\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.251431 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.251293 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-istio-data\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.251585 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.251563 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-workload-socket\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.251651 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.251566 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-workload-certs\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.251651 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.251637 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-credential-socket\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.251805 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.251732 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b44980-d812-40eb-b4ad-87d4af154876-istiod-ca-cert\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.253461 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.253440 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-istio-envoy\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.253623 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.253605 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b44980-d812-40eb-b4ad-87d4af154876-istio-podinfo\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.265784 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.265756 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b44980-d812-40eb-b4ad-87d4af154876-istio-token\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.265908 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.265784 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx6xc\" (UniqueName: \"kubernetes.io/projected/a4b44980-d812-40eb-b4ad-87d4af154876-kube-api-access-sx6xc\") pod \"e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.341917 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.341827 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:44.622084 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.622053 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8"] Sep 17 23:31:44.626834 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.626802 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:31:44.626920 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.626872 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:31:44.626920 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:44.626902 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:31:45.610943 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:45.610897 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" event={"ID":"a4b44980-d812-40eb-b4ad-87d4af154876","Type":"ContainerStarted","Data":"ce719a216c1cae7f4a40eccf1a507fdfb84e7e7865906d9571fc3f4344b759b5"} Sep 17 23:31:45.611341 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:45.610965 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" event={"ID":"a4b44980-d812-40eb-b4ad-87d4af154876","Type":"ContainerStarted","Data":"36dedcf5d466c80fed9bbefe1a000d273649092a9e282369f8abb04121309ebc"} Sep 17 23:31:45.631629 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:45.631579 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" podStartSLOduration=1.6315639370000001 podStartE2EDuration="1.631563937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:31:45.630042573 +0000 UTC m=+1544.563949709" watchObservedRunningTime="2026-09-17 23:31:45.631563937 +0000 UTC m=+1544.565471073" Sep 17 23:31:46.342189 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:46.342155 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:46.347372 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:46.347345 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:46.617213 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:46.617101 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:46.618065 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:46.618045 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:31:51.892924 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:51.892885 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf"] Sep 17 23:31:51.896435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:51.896415 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf" Sep 17 23:31:51.898602 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:51.898585 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-78432f\"" Sep 17 23:31:51.898677 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:51.898660 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-78432f-dockercfg-ttpbl\"" Sep 17 23:31:51.906448 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:51.906424 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf"] Sep 17 23:31:52.030670 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:52.030632 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8krt\" (UniqueName: \"kubernetes.io/projected/326ab3f1-159d-4951-ac1f-629bf725e246-kube-api-access-b8krt\") pod \"payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf\" (UID: \"326ab3f1-159d-4951-ac1f-629bf725e246\") " pod="openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf" Sep 17 23:31:52.030853 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:52.030693 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/326ab3f1-159d-4951-ac1f-629bf725e246-plugins-config-volume\") pod \"payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf\" (UID: \"326ab3f1-159d-4951-ac1f-629bf725e246\") " pod="openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf" Sep 17 23:31:52.132228 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:52.132186 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/326ab3f1-159d-4951-ac1f-629bf725e246-plugins-config-volume\") pod \"payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf\" (UID: \"326ab3f1-159d-4951-ac1f-629bf725e246\") " pod="openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf" Sep 17 23:31:52.132407 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:52.132308 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8krt\" (UniqueName: \"kubernetes.io/projected/326ab3f1-159d-4951-ac1f-629bf725e246-kube-api-access-b8krt\") pod \"payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf\" (UID: \"326ab3f1-159d-4951-ac1f-629bf725e246\") " pod="openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf" Sep 17 23:31:52.132989 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:52.132966 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/326ab3f1-159d-4951-ac1f-629bf725e246-plugins-config-volume\") pod \"payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf\" (UID: \"326ab3f1-159d-4951-ac1f-629bf725e246\") " pod="openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf" Sep 17 23:31:52.139589 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:52.139564 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8krt\" (UniqueName: \"kubernetes.io/projected/326ab3f1-159d-4951-ac1f-629bf725e246-kube-api-access-b8krt\") pod \"payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf\" (UID: \"326ab3f1-159d-4951-ac1f-629bf725e246\") " pod="openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf" Sep 17 23:31:52.210339 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:52.210249 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf" Sep 17 23:31:52.486939 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:52.486912 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf"] Sep 17 23:31:52.489536 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:31:52.489472 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326ab3f1_159d_4951_ac1f_629bf725e246.slice/crio-93b5a52492e4bfdbbf7dd6832822fd3113a51b394c37150255fe94baceda360e WatchSource:0}: Error finding container 93b5a52492e4bfdbbf7dd6832822fd3113a51b394c37150255fe94baceda360e: Status 404 returned error can't find the container with id 93b5a52492e4bfdbbf7dd6832822fd3113a51b394c37150255fe94baceda360e Sep 17 23:31:52.654471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:52.654425 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf" event={"ID":"326ab3f1-159d-4951-ac1f-629bf725e246","Type":"ContainerStarted","Data":"8d8075398e8e680df27363075706e5529ed1d89c4ae80471f38295d4ccb93fd2"} Sep 17 23:31:52.654694 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:52.654492 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf" event={"ID":"326ab3f1-159d-4951-ac1f-629bf725e246","Type":"ContainerStarted","Data":"93b5a52492e4bfdbbf7dd6832822fd3113a51b394c37150255fe94baceda360e"} Sep 17 23:31:52.654694 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:52.654664 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf" Sep 17 23:31:52.673835 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:31:52.673767 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf" podStartSLOduration=1.673743169 podStartE2EDuration="1.673743169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:31:52.669665273 +0000 UTC m=+1551.603572410" watchObservedRunningTime="2026-09-17 23:31:52.673743169 +0000 UTC m=+1551.607650306" Sep 17 23:32:03.662644 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:03.662616 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf" Sep 17 23:32:05.743607 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:05.743571 2582 generic.go:356] "Generic (PLEG): container finished" podID="a4b44980-d812-40eb-b4ad-87d4af154876" containerID="ce719a216c1cae7f4a40eccf1a507fdfb84e7e7865906d9571fc3f4344b759b5" exitCode=0 Sep 17 23:32:05.744024 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:05.743646 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" event={"ID":"a4b44980-d812-40eb-b4ad-87d4af154876","Type":"ContainerDied","Data":"ce719a216c1cae7f4a40eccf1a507fdfb84e7e7865906d9571fc3f4344b759b5"} Sep 17 23:32:05.744090 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:05.744076 2582 scope.go:117] "RemoveContainer" containerID="ce719a216c1cae7f4a40eccf1a507fdfb84e7e7865906d9571fc3f4344b759b5" Sep 17 23:32:05.745266 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:05.745235 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:32:05.745389 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:05.745316 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:32:05.745389 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:05.745367 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:32:06.751269 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:06.751235 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" event={"ID":"a4b44980-d812-40eb-b4ad-87d4af154876","Type":"ContainerStarted","Data":"d5ca5ec226839b934c2fd0b4f43f869f899c83cf9f716333c4ca0c5868f9b3cc"} Sep 17 23:32:07.342765 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:07.342729 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:32:07.343079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:07.343056 2582 patch_prober.go:28] interesting pod/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 17 23:32:07.343187 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:07.343107 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" podUID="a4b44980-d812-40eb-b4ad-87d4af154876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 17 23:32:08.342622 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:08.342586 2582 patch_prober.go:28] interesting pod/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 17 23:32:08.343072 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:08.342647 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" podUID="a4b44980-d812-40eb-b4ad-87d4af154876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 17 23:32:09.343142 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:09.343104 2582 patch_prober.go:28] interesting pod/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 17 23:32:09.343564 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:09.343163 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" podUID="a4b44980-d812-40eb-b4ad-87d4af154876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 17 23:32:10.342522 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:10.342463 2582 patch_prober.go:28] interesting pod/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 17 23:32:10.342695 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:10.342568 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" podUID="a4b44980-d812-40eb-b4ad-87d4af154876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 17 23:32:11.342445 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:11.342405 2582 patch_prober.go:28] interesting pod/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 17 23:32:11.342887 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:11.342475 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" podUID="a4b44980-d812-40eb-b4ad-87d4af154876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 17 23:32:12.342472 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:12.342434 2582 patch_prober.go:28] interesting pod/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 17 23:32:12.342879 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:12.342526 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" podUID="a4b44980-d812-40eb-b4ad-87d4af154876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 17 23:32:13.347342 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:13.347311 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:32:13.347853 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:13.347631 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:32:13.348298 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:13.348279 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:32:18.460016 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.459983 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn"] Sep 17 23:32:18.463763 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.463741 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.466240 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.466220 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cf8993-openshift-default-dockercfg-qzrr5\"" Sep 17 23:32:18.474966 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.474866 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn"] Sep 17 23:32:18.505267 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.505234 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-workload-certs\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.505267 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.505268 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c79f4562-a752-4e77-a0f4-7658ea1f6835-istiod-ca-cert\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.505439 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.505289 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4wmv\" (UniqueName: \"kubernetes.io/projected/c79f4562-a752-4e77-a0f4-7658ea1f6835-kube-api-access-g4wmv\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.505439 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.505382 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-data\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.505439 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.505410 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-envoy\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.505568 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.505476 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-workload-socket\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.505568 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.505522 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-credential-socket\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.505568 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.505542 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-token\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.505667 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.505594 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-podinfo\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.607139 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.607091 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-envoy\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.607323 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.607185 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-workload-socket\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.607483 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.607459 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-credential-socket\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.607890 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.607870 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-token\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.608099 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.608083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-podinfo\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.608606 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.607929 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-credential-socket\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.608606 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.608289 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-workload-certs\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.608606 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.608323 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c79f4562-a752-4e77-a0f4-7658ea1f6835-istiod-ca-cert\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.608606 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.608351 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4wmv\" (UniqueName: \"kubernetes.io/projected/c79f4562-a752-4e77-a0f4-7658ea1f6835-kube-api-access-g4wmv\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.608606 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.608409 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-data\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.608956 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.608796 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-data\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.608956 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.607699 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-workload-socket\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.609158 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.609063 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-workload-certs\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.609799 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.609761 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c79f4562-a752-4e77-a0f4-7658ea1f6835-istiod-ca-cert\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.612535 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.612453 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-podinfo\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.623073 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.623045 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-envoy\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.624943 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.624893 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-token\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.629520 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.627127 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4wmv\" (UniqueName: \"kubernetes.io/projected/c79f4562-a752-4e77-a0f4-7658ea1f6835-kube-api-access-g4wmv\") pod \"e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:18.778928 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:18.778895 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:19.053988 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:19.053965 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn"] Sep 17 23:32:19.055340 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:32:19.055313 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79f4562_a752_4e77_a0f4_7658ea1f6835.slice/crio-91b2898018b205529e24866e4f281aafc561a6f16c73c10907ba2d6d109ad294 WatchSource:0}: Error finding container 91b2898018b205529e24866e4f281aafc561a6f16c73c10907ba2d6d109ad294: Status 404 returned error can't find the container with id 91b2898018b205529e24866e4f281aafc561a6f16c73c10907ba2d6d109ad294 Sep 17 23:32:19.057689 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:19.057657 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:32:19.057788 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:19.057718 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:32:19.057788 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:19.057751 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:32:19.859586 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:19.859540 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" event={"ID":"c79f4562-a752-4e77-a0f4-7658ea1f6835","Type":"ContainerStarted","Data":"e65d53ddfdd72815b213070c4d40dd7816ed8213f5ed0338797e43b1594bb5d9"} Sep 17 23:32:19.860022 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:19.859600 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" event={"ID":"c79f4562-a752-4e77-a0f4-7658ea1f6835","Type":"ContainerStarted","Data":"91b2898018b205529e24866e4f281aafc561a6f16c73c10907ba2d6d109ad294"} Sep 17 23:32:19.879650 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:19.879595 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" podStartSLOduration=1.879572128 podStartE2EDuration="1.879572128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:32:19.87709824 +0000 UTC m=+1578.811005387" watchObservedRunningTime="2026-09-17 23:32:19.879572128 +0000 UTC m=+1578.813479265" Sep 17 23:32:20.779953 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:20.779906 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:20.785702 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:20.785675 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:20.865382 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:20.865351 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:20.866379 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:20.866360 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:44.004136 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:44.004097 2582 generic.go:356] "Generic (PLEG): container finished" podID="c79f4562-a752-4e77-a0f4-7658ea1f6835" containerID="e65d53ddfdd72815b213070c4d40dd7816ed8213f5ed0338797e43b1594bb5d9" exitCode=0 Sep 17 23:32:44.004566 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:44.004166 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" event={"ID":"c79f4562-a752-4e77-a0f4-7658ea1f6835","Type":"ContainerDied","Data":"e65d53ddfdd72815b213070c4d40dd7816ed8213f5ed0338797e43b1594bb5d9"} Sep 17 23:32:44.004621 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:44.004602 2582 scope.go:117] "RemoveContainer" containerID="e65d53ddfdd72815b213070c4d40dd7816ed8213f5ed0338797e43b1594bb5d9" Sep 17 23:32:44.005882 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:44.005856 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:32:44.005958 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:44.005923 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:32:44.006000 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:44.005958 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:32:45.012404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:45.012321 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" event={"ID":"c79f4562-a752-4e77-a0f4-7658ea1f6835","Type":"ContainerStarted","Data":"77a20b24eee4aec3e326bdfe91838cb1c49366bf9af9e2dabb4604abfde716b9"} Sep 17 23:32:45.780358 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:45.780318 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:45.780593 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:45.780441 2582 patch_prober.go:28] interesting pod/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Sep 17 23:32:45.780593 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:45.780513 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" podUID="c79f4562-a752-4e77-a0f4-7658ea1f6835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Sep 17 23:32:46.779835 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:46.779791 2582 patch_prober.go:28] interesting pod/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Sep 17 23:32:46.780292 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:46.779868 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" podUID="c79f4562-a752-4e77-a0f4-7658ea1f6835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Sep 17 23:32:47.780143 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:47.779717 2582 patch_prober.go:28] interesting pod/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Sep 17 23:32:47.780143 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:47.779783 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" podUID="c79f4562-a752-4e77-a0f4-7658ea1f6835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Sep 17 23:32:48.779880 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:48.779844 2582 patch_prober.go:28] interesting pod/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Sep 17 23:32:48.780592 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:48.780162 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" podUID="c79f4562-a752-4e77-a0f4-7658ea1f6835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Sep 17 23:32:49.780062 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:49.780026 2582 patch_prober.go:28] interesting pod/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Sep 17 23:32:49.780348 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:49.780305 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" podUID="c79f4562-a752-4e77-a0f4-7658ea1f6835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Sep 17 23:32:50.785126 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:50.785096 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:50.785570 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:50.785419 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:50.786084 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:50.786065 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:32:54.887516 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:54.887470 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78432f-gsc27"] Sep 17 23:32:54.891034 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:54.891010 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78432f-gsc27" Sep 17 23:32:54.894219 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:54.894183 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:32:54.897330 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:54.897305 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78432f-gsc27"] Sep 17 23:32:55.001402 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:55.001377 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff7ec047-ea27-44e6-9b12-1ecdb5e19c64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-78432f-gsc27\" (UID: \"ff7ec047-ea27-44e6-9b12-1ecdb5e19c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78432f-gsc27" Sep 17 23:32:55.102367 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:55.102333 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff7ec047-ea27-44e6-9b12-1ecdb5e19c64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-78432f-gsc27\" (UID: \"ff7ec047-ea27-44e6-9b12-1ecdb5e19c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78432f-gsc27" Sep 17 23:32:55.103006 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:55.102984 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff7ec047-ea27-44e6-9b12-1ecdb5e19c64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-78432f-gsc27\" (UID: \"ff7ec047-ea27-44e6-9b12-1ecdb5e19c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78432f-gsc27" Sep 17 23:32:55.204685 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:55.204593 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78432f-gsc27" Sep 17 23:32:55.475583 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:55.475554 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78432f-gsc27"] Sep 17 23:32:55.478293 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:32:55.478266 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7ec047_ea27_44e6_9b12_1ecdb5e19c64.slice/crio-87fdd5b4353a9407270a2711fbb403bc4e6e4731a79047b06c989f2f643f288e WatchSource:0}: Error finding container 87fdd5b4353a9407270a2711fbb403bc4e6e4731a79047b06c989f2f643f288e: Status 404 returned error can't find the container with id 87fdd5b4353a9407270a2711fbb403bc4e6e4731a79047b06c989f2f643f288e Sep 17 23:32:56.081030 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:56.080945 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78432f-gsc27" event={"ID":"ff7ec047-ea27-44e6-9b12-1ecdb5e19c64","Type":"ContainerStarted","Data":"4cdecc6def40059031b39c4e7d7092bd7bc7cfc6ac9f775c80a8dec39675a2f0"} Sep 17 23:32:56.081030 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:56.080996 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78432f-gsc27" event={"ID":"ff7ec047-ea27-44e6-9b12-1ecdb5e19c64","Type":"ContainerStarted","Data":"87fdd5b4353a9407270a2711fbb403bc4e6e4731a79047b06c989f2f643f288e"} Sep 17 23:32:56.104517 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:56.101209 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78432f-gsc27" podStartSLOduration=2.101183313 podStartE2EDuration="2.101183313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:32:56.094955297 +0000 UTC m=+1615.028862432" watchObservedRunningTime="2026-09-17 23:32:56.101183313 +0000 UTC m=+1615.035090443" Sep 17 23:32:57.090715 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:57.090674 2582 generic.go:356] "Generic (PLEG): container finished" podID="ff7ec047-ea27-44e6-9b12-1ecdb5e19c64" containerID="4cdecc6def40059031b39c4e7d7092bd7bc7cfc6ac9f775c80a8dec39675a2f0" exitCode=0 Sep 17 23:32:57.091082 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:57.090712 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78432f-gsc27" event={"ID":"ff7ec047-ea27-44e6-9b12-1ecdb5e19c64","Type":"ContainerDied","Data":"4cdecc6def40059031b39c4e7d7092bd7bc7cfc6ac9f775c80a8dec39675a2f0"} Sep 17 23:32:58.232425 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:58.232400 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78432f-gsc27" Sep 17 23:32:58.335167 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:58.335135 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff7ec047-ea27-44e6-9b12-1ecdb5e19c64-maas-api-service-ca\") pod \"ff7ec047-ea27-44e6-9b12-1ecdb5e19c64\" (UID: \"ff7ec047-ea27-44e6-9b12-1ecdb5e19c64\") " Sep 17 23:32:58.335482 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:58.335463 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff7ec047-ea27-44e6-9b12-1ecdb5e19c64-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ff7ec047-ea27-44e6-9b12-1ecdb5e19c64" (UID: "ff7ec047-ea27-44e6-9b12-1ecdb5e19c64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:32:58.335611 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:58.335594 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff7ec047-ea27-44e6-9b12-1ecdb5e19c64-maas-api-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:32:59.105971 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:59.105932 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78432f-gsc27" event={"ID":"ff7ec047-ea27-44e6-9b12-1ecdb5e19c64","Type":"ContainerDied","Data":"87fdd5b4353a9407270a2711fbb403bc4e6e4731a79047b06c989f2f643f288e"} Sep 17 23:32:59.106153 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:59.105996 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87fdd5b4353a9407270a2711fbb403bc4e6e4731a79047b06c989f2f643f288e" Sep 17 23:32:59.106153 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:32:59.105951 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78432f-gsc27" Sep 17 23:33:03.548740 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:03.548708 2582 scope.go:117] "RemoveContainer" containerID="7101ed48e08d0feee0a3b9cb93c9ad8835720975e1c6cbb402c7f0f1b9809f17" Sep 17 23:33:10.302716 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:10.302683 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf"] Sep 17 23:33:10.303160 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:10.303001 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf" podUID="326ab3f1-159d-4951-ac1f-629bf725e246" containerName="payload-processing" containerID="cri-o://8d8075398e8e680df27363075706e5529ed1d89c4ae80471f38295d4ccb93fd2" gracePeriod=30 Sep 17 23:33:10.468147 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:10.468083 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78432f-gsc27"] Sep 17 23:33:10.472768 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:10.472737 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78432f-gsc27"] Sep 17 23:33:10.560570 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:10.560513 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf" Sep 17 23:33:10.663724 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:10.663694 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/326ab3f1-159d-4951-ac1f-629bf725e246-plugins-config-volume\") pod \"326ab3f1-159d-4951-ac1f-629bf725e246\" (UID: \"326ab3f1-159d-4951-ac1f-629bf725e246\") " Sep 17 23:33:10.663908 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:10.663781 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8krt\" (UniqueName: \"kubernetes.io/projected/326ab3f1-159d-4951-ac1f-629bf725e246-kube-api-access-b8krt\") pod \"326ab3f1-159d-4951-ac1f-629bf725e246\" (UID: \"326ab3f1-159d-4951-ac1f-629bf725e246\") " Sep 17 23:33:10.664128 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:10.664089 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326ab3f1-159d-4951-ac1f-629bf725e246-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "326ab3f1-159d-4951-ac1f-629bf725e246" (UID: "326ab3f1-159d-4951-ac1f-629bf725e246"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:33:10.664256 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:10.664204 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/326ab3f1-159d-4951-ac1f-629bf725e246-plugins-config-volume\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:10.665880 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:10.665858 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326ab3f1-159d-4951-ac1f-629bf725e246-kube-api-access-b8krt" (OuterVolumeSpecName: "kube-api-access-b8krt") pod "326ab3f1-159d-4951-ac1f-629bf725e246" (UID: "326ab3f1-159d-4951-ac1f-629bf725e246"). InnerVolumeSpecName "kube-api-access-b8krt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:33:10.765389 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:10.765356 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8krt\" (UniqueName: \"kubernetes.io/projected/326ab3f1-159d-4951-ac1f-629bf725e246-kube-api-access-b8krt\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:11.179923 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:11.179888 2582 generic.go:356] "Generic (PLEG): container finished" podID="326ab3f1-159d-4951-ac1f-629bf725e246" containerID="8d8075398e8e680df27363075706e5529ed1d89c4ae80471f38295d4ccb93fd2" exitCode=0 Sep 17 23:33:11.180080 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:11.179936 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf" Sep 17 23:33:11.180080 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:11.179973 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf" event={"ID":"326ab3f1-159d-4951-ac1f-629bf725e246","Type":"ContainerDied","Data":"8d8075398e8e680df27363075706e5529ed1d89c4ae80471f38295d4ccb93fd2"} Sep 17 23:33:11.180080 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:11.180031 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf" event={"ID":"326ab3f1-159d-4951-ac1f-629bf725e246","Type":"ContainerDied","Data":"93b5a52492e4bfdbbf7dd6832822fd3113a51b394c37150255fe94baceda360e"} Sep 17 23:33:11.180080 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:11.180059 2582 scope.go:117] "RemoveContainer" containerID="8d8075398e8e680df27363075706e5529ed1d89c4ae80471f38295d4ccb93fd2" Sep 17 23:33:11.188993 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:11.188979 2582 scope.go:117] "RemoveContainer" containerID="8d8075398e8e680df27363075706e5529ed1d89c4ae80471f38295d4ccb93fd2" Sep 17 23:33:11.189253 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:33:11.189236 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d8075398e8e680df27363075706e5529ed1d89c4ae80471f38295d4ccb93fd2\": container with ID starting with 8d8075398e8e680df27363075706e5529ed1d89c4ae80471f38295d4ccb93fd2 not found: ID does not exist" containerID="8d8075398e8e680df27363075706e5529ed1d89c4ae80471f38295d4ccb93fd2" Sep 17 23:33:11.189292 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:11.189261 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d8075398e8e680df27363075706e5529ed1d89c4ae80471f38295d4ccb93fd2"} err="failed to get container status \"8d8075398e8e680df27363075706e5529ed1d89c4ae80471f38295d4ccb93fd2\": rpc error: code = NotFound desc = could not find container \"8d8075398e8e680df27363075706e5529ed1d89c4ae80471f38295d4ccb93fd2\": container with ID starting with 8d8075398e8e680df27363075706e5529ed1d89c4ae80471f38295d4ccb93fd2 not found: ID does not exist" Sep 17 23:33:11.200484 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:11.200463 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf"] Sep 17 23:33:11.204733 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:11.204712 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-78432f-769f8cc79-gdxcf"] Sep 17 23:33:11.293940 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:11.293912 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="326ab3f1-159d-4951-ac1f-629bf725e246" path="/var/lib/kubelet/pods/326ab3f1-159d-4951-ac1f-629bf725e246/volumes" Sep 17 23:33:11.294255 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:11.294242 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff7ec047-ea27-44e6-9b12-1ecdb5e19c64" path="/var/lib/kubelet/pods/ff7ec047-ea27-44e6-9b12-1ecdb5e19c64/volumes" Sep 17 23:33:16.866652 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:16.866608 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8"] Sep 17 23:33:16.867726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:16.866837 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" podUID="a4b44980-d812-40eb-b4ad-87d4af154876" containerName="istio-proxy" containerID="cri-o://d5ca5ec226839b934c2fd0b4f43f869f899c83cf9f716333c4ca0c5868f9b3cc" gracePeriod=30 Sep 17 23:33:17.222616 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:17.222582 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz"] Sep 17 23:33:17.223260 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:17.223243 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="326ab3f1-159d-4951-ac1f-629bf725e246" containerName="payload-processing" Sep 17 23:33:17.223309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:17.223264 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="326ab3f1-159d-4951-ac1f-629bf725e246" containerName="payload-processing" Sep 17 23:33:17.223309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:17.223280 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff7ec047-ea27-44e6-9b12-1ecdb5e19c64" containerName="revoke-keys" Sep 17 23:33:17.223309 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:17.223285 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff7ec047-ea27-44e6-9b12-1ecdb5e19c64" containerName="revoke-keys" Sep 17 23:33:17.223400 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:17.223368 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="326ab3f1-159d-4951-ac1f-629bf725e246" containerName="payload-processing" Sep 17 23:33:17.223400 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:17.223378 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff7ec047-ea27-44e6-9b12-1ecdb5e19c64" containerName="revoke-keys" Sep 17 23:33:17.226211 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:17.226194 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz" Sep 17 23:33:17.228427 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:17.228409 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:33:17.233077 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:17.233055 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz"] Sep 17 23:33:17.326125 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:17.326100 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93f62a2-e08d-4605-a234-9fc62d46b231-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz\" (UID: \"d93f62a2-e08d-4605-a234-9fc62d46b231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz" Sep 17 23:33:17.427460 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:17.427428 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93f62a2-e08d-4605-a234-9fc62d46b231-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz\" (UID: \"d93f62a2-e08d-4605-a234-9fc62d46b231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz" Sep 17 23:33:17.428083 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:17.428064 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93f62a2-e08d-4605-a234-9fc62d46b231-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz\" (UID: \"d93f62a2-e08d-4605-a234-9fc62d46b231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz" Sep 17 23:33:17.537607 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:17.537536 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz" Sep 17 23:33:17.809883 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:17.809855 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz"] Sep 17 23:33:17.811355 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:33:17.811330 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93f62a2_e08d_4605_a234_9fc62d46b231.slice/crio-b478105b80eb789a2c918aecc7a8b96d4873dd5550bfec59c2015e17c6367c61 WatchSource:0}: Error finding container b478105b80eb789a2c918aecc7a8b96d4873dd5550bfec59c2015e17c6367c61: Status 404 returned error can't find the container with id b478105b80eb789a2c918aecc7a8b96d4873dd5550bfec59c2015e17c6367c61 Sep 17 23:33:18.223946 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:18.223914 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz" event={"ID":"d93f62a2-e08d-4605-a234-9fc62d46b231","Type":"ContainerStarted","Data":"7f60e4b95230295425dd5eda8374a29a302c9ff228afe0e0091556a955f58e7f"} Sep 17 23:33:18.224301 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:18.223968 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz" event={"ID":"d93f62a2-e08d-4605-a234-9fc62d46b231","Type":"ContainerStarted","Data":"b478105b80eb789a2c918aecc7a8b96d4873dd5550bfec59c2015e17c6367c61"} Sep 17 23:33:18.240481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:18.240434 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz" podStartSLOduration=1.240420685 podStartE2EDuration="1.240420685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:33:18.23769004 +0000 UTC m=+1637.171597175" watchObservedRunningTime="2026-09-17 23:33:18.240420685 +0000 UTC m=+1637.174327819" Sep 17 23:33:19.231860 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:19.231822 2582 generic.go:356] "Generic (PLEG): container finished" podID="d93f62a2-e08d-4605-a234-9fc62d46b231" containerID="7f60e4b95230295425dd5eda8374a29a302c9ff228afe0e0091556a955f58e7f" exitCode=0 Sep 17 23:33:19.232213 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:19.231900 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz" event={"ID":"d93f62a2-e08d-4605-a234-9fc62d46b231","Type":"ContainerDied","Data":"7f60e4b95230295425dd5eda8374a29a302c9ff228afe0e0091556a955f58e7f"} Sep 17 23:33:20.376679 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:20.376653 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz" Sep 17 23:33:20.458593 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:20.458562 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93f62a2-e08d-4605-a234-9fc62d46b231-maas-api-service-ca\") pod \"d93f62a2-e08d-4605-a234-9fc62d46b231\" (UID: \"d93f62a2-e08d-4605-a234-9fc62d46b231\") " Sep 17 23:33:20.458975 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:20.458952 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d93f62a2-e08d-4605-a234-9fc62d46b231-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d93f62a2-e08d-4605-a234-9fc62d46b231" (UID: "d93f62a2-e08d-4605-a234-9fc62d46b231"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:33:20.559864 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:20.559796 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d93f62a2-e08d-4605-a234-9fc62d46b231-maas-api-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:21.249221 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:21.249183 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz" Sep 17 23:33:21.249395 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:21.249219 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz" event={"ID":"d93f62a2-e08d-4605-a234-9fc62d46b231","Type":"ContainerDied","Data":"b478105b80eb789a2c918aecc7a8b96d4873dd5550bfec59c2015e17c6367c61"} Sep 17 23:33:21.249395 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:21.249265 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b478105b80eb789a2c918aecc7a8b96d4873dd5550bfec59c2015e17c6367c61" Sep 17 23:33:22.119773 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.119752 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:33:22.175664 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.175604 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-credential-socket\") pod \"a4b44980-d812-40eb-b4ad-87d4af154876\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " Sep 17 23:33:22.175664 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.175634 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx6xc\" (UniqueName: \"kubernetes.io/projected/a4b44980-d812-40eb-b4ad-87d4af154876-kube-api-access-sx6xc\") pod \"a4b44980-d812-40eb-b4ad-87d4af154876\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " Sep 17 23:33:22.175838 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.175672 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b44980-d812-40eb-b4ad-87d4af154876-istio-token\") pod \"a4b44980-d812-40eb-b4ad-87d4af154876\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " Sep 17 23:33:22.175838 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.175688 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b44980-d812-40eb-b4ad-87d4af154876-istio-podinfo\") pod \"a4b44980-d812-40eb-b4ad-87d4af154876\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " Sep 17 23:33:22.175838 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.175824 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-workload-socket\") pod \"a4b44980-d812-40eb-b4ad-87d4af154876\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " Sep 17 23:33:22.175975 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.175883 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-workload-certs\") pod \"a4b44980-d812-40eb-b4ad-87d4af154876\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " Sep 17 23:33:22.175975 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.175906 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b44980-d812-40eb-b4ad-87d4af154876-istiod-ca-cert\") pod \"a4b44980-d812-40eb-b4ad-87d4af154876\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " Sep 17 23:33:22.175975 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.175939 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-istio-envoy\") pod \"a4b44980-d812-40eb-b4ad-87d4af154876\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " Sep 17 23:33:22.175975 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.175946 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a4b44980-d812-40eb-b4ad-87d4af154876" (UID: "a4b44980-d812-40eb-b4ad-87d4af154876"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:33:22.176181 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.176093 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-istio-data\") pod \"a4b44980-d812-40eb-b4ad-87d4af154876\" (UID: \"a4b44980-d812-40eb-b4ad-87d4af154876\") " Sep 17 23:33:22.176566 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.176299 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a4b44980-d812-40eb-b4ad-87d4af154876" (UID: "a4b44980-d812-40eb-b4ad-87d4af154876"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:33:22.176566 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.176314 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a4b44980-d812-40eb-b4ad-87d4af154876" (UID: "a4b44980-d812-40eb-b4ad-87d4af154876"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:33:22.176566 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.176405 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b44980-d812-40eb-b4ad-87d4af154876-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a4b44980-d812-40eb-b4ad-87d4af154876" (UID: "a4b44980-d812-40eb-b4ad-87d4af154876"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:33:22.176566 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.176521 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-istio-data" (OuterVolumeSpecName: "istio-data") pod "a4b44980-d812-40eb-b4ad-87d4af154876" (UID: "a4b44980-d812-40eb-b4ad-87d4af154876"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:33:22.176566 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.176560 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-credential-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:22.176828 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.176581 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-workload-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:22.176828 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.176594 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-workload-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:22.176828 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.176605 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b44980-d812-40eb-b4ad-87d4af154876-istiod-ca-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:22.178021 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.177997 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a4b44980-d812-40eb-b4ad-87d4af154876" (UID: "a4b44980-d812-40eb-b4ad-87d4af154876"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:33:22.178301 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.178285 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b44980-d812-40eb-b4ad-87d4af154876-kube-api-access-sx6xc" (OuterVolumeSpecName: "kube-api-access-sx6xc") pod "a4b44980-d812-40eb-b4ad-87d4af154876" (UID: "a4b44980-d812-40eb-b4ad-87d4af154876"). InnerVolumeSpecName "kube-api-access-sx6xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:33:22.178366 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.178297 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4b44980-d812-40eb-b4ad-87d4af154876-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a4b44980-d812-40eb-b4ad-87d4af154876" (UID: "a4b44980-d812-40eb-b4ad-87d4af154876"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:33:22.178366 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.178355 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b44980-d812-40eb-b4ad-87d4af154876-istio-token" (OuterVolumeSpecName: "istio-token") pod "a4b44980-d812-40eb-b4ad-87d4af154876" (UID: "a4b44980-d812-40eb-b4ad-87d4af154876"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:33:22.256313 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.256285 2582 generic.go:356] "Generic (PLEG): container finished" podID="a4b44980-d812-40eb-b4ad-87d4af154876" containerID="d5ca5ec226839b934c2fd0b4f43f869f899c83cf9f716333c4ca0c5868f9b3cc" exitCode=0 Sep 17 23:33:22.256417 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.256334 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" Sep 17 23:33:22.256417 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.256368 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" event={"ID":"a4b44980-d812-40eb-b4ad-87d4af154876","Type":"ContainerDied","Data":"d5ca5ec226839b934c2fd0b4f43f869f899c83cf9f716333c4ca0c5868f9b3cc"} Sep 17 23:33:22.256510 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.256417 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8" event={"ID":"a4b44980-d812-40eb-b4ad-87d4af154876","Type":"ContainerDied","Data":"36dedcf5d466c80fed9bbefe1a000d273649092a9e282369f8abb04121309ebc"} Sep 17 23:33:22.256510 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.256439 2582 scope.go:117] "RemoveContainer" containerID="d5ca5ec226839b934c2fd0b4f43f869f899c83cf9f716333c4ca0c5868f9b3cc" Sep 17 23:33:22.266087 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.266071 2582 scope.go:117] "RemoveContainer" containerID="ce719a216c1cae7f4a40eccf1a507fdfb84e7e7865906d9571fc3f4344b759b5" Sep 17 23:33:22.276708 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.276691 2582 scope.go:117] "RemoveContainer" containerID="d5ca5ec226839b934c2fd0b4f43f869f899c83cf9f716333c4ca0c5868f9b3cc" Sep 17 23:33:22.277056 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:33:22.277036 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5ca5ec226839b934c2fd0b4f43f869f899c83cf9f716333c4ca0c5868f9b3cc\": container with ID starting with d5ca5ec226839b934c2fd0b4f43f869f899c83cf9f716333c4ca0c5868f9b3cc not found: ID does not exist" containerID="d5ca5ec226839b934c2fd0b4f43f869f899c83cf9f716333c4ca0c5868f9b3cc" Sep 17 23:33:22.277146 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.277063 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5ca5ec226839b934c2fd0b4f43f869f899c83cf9f716333c4ca0c5868f9b3cc"} err="failed to get container status \"d5ca5ec226839b934c2fd0b4f43f869f899c83cf9f716333c4ca0c5868f9b3cc\": rpc error: code = NotFound desc = could not find container \"d5ca5ec226839b934c2fd0b4f43f869f899c83cf9f716333c4ca0c5868f9b3cc\": container with ID starting with d5ca5ec226839b934c2fd0b4f43f869f899c83cf9f716333c4ca0c5868f9b3cc not found: ID does not exist" Sep 17 23:33:22.277146 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.277081 2582 scope.go:117] "RemoveContainer" containerID="ce719a216c1cae7f4a40eccf1a507fdfb84e7e7865906d9571fc3f4344b759b5" Sep 17 23:33:22.277272 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.277244 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b44980-d812-40eb-b4ad-87d4af154876-istio-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:22.277386 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.277276 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b44980-d812-40eb-b4ad-87d4af154876-istio-podinfo\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:22.277386 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.277290 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-istio-envoy\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:22.277386 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.277301 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b44980-d812-40eb-b4ad-87d4af154876-istio-data\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:22.277386 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.277316 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx6xc\" (UniqueName: \"kubernetes.io/projected/a4b44980-d812-40eb-b4ad-87d4af154876-kube-api-access-sx6xc\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:22.277386 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:33:22.277378 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce719a216c1cae7f4a40eccf1a507fdfb84e7e7865906d9571fc3f4344b759b5\": container with ID starting with ce719a216c1cae7f4a40eccf1a507fdfb84e7e7865906d9571fc3f4344b759b5 not found: ID does not exist" containerID="ce719a216c1cae7f4a40eccf1a507fdfb84e7e7865906d9571fc3f4344b759b5" Sep 17 23:33:22.277666 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.277403 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce719a216c1cae7f4a40eccf1a507fdfb84e7e7865906d9571fc3f4344b759b5"} err="failed to get container status \"ce719a216c1cae7f4a40eccf1a507fdfb84e7e7865906d9571fc3f4344b759b5\": rpc error: code = NotFound desc = could not find container \"ce719a216c1cae7f4a40eccf1a507fdfb84e7e7865906d9571fc3f4344b759b5\": container with ID starting with ce719a216c1cae7f4a40eccf1a507fdfb84e7e7865906d9571fc3f4344b759b5 not found: ID does not exist" Sep 17 23:33:22.278576 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.278554 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8"] Sep 17 23:33:22.281190 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:22.281161 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-78432f-openshift-default-6bd4c49c9b-rp9l8"] Sep 17 23:33:23.294611 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:23.294582 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4b44980-d812-40eb-b4ad-87d4af154876" path="/var/lib/kubelet/pods/a4b44980-d812-40eb-b4ad-87d4af154876/volumes" Sep 17 23:33:25.139806 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:25.139778 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7852-6c1bab"] Sep 17 23:33:25.140228 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:25.140216 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4b44980-d812-40eb-b4ad-87d4af154876" containerName="istio-proxy" Sep 17 23:33:25.140290 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:25.140229 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4b44980-d812-40eb-b4ad-87d4af154876" containerName="istio-proxy" Sep 17 23:33:25.140290 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:25.140240 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d93f62a2-e08d-4605-a234-9fc62d46b231" containerName="revoke-keys" Sep 17 23:33:25.140290 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:25.140246 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d93f62a2-e08d-4605-a234-9fc62d46b231" containerName="revoke-keys" Sep 17 23:33:25.140290 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:25.140254 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4b44980-d812-40eb-b4ad-87d4af154876" containerName="istio-proxy" Sep 17 23:33:25.140290 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:25.140260 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4b44980-d812-40eb-b4ad-87d4af154876" containerName="istio-proxy" Sep 17 23:33:25.140436 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:25.140329 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4b44980-d812-40eb-b4ad-87d4af154876" containerName="istio-proxy" Sep 17 23:33:25.140436 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:25.140339 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d93f62a2-e08d-4605-a234-9fc62d46b231" containerName="revoke-keys" Sep 17 23:33:25.142629 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:25.142615 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-6c1bab" Sep 17 23:33:25.145481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:25.145454 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m6p2p\"" Sep 17 23:33:25.150516 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:25.147776 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-6c1bab"] Sep 17 23:33:25.204817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:25.204787 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-276jp\" (UniqueName: \"kubernetes.io/projected/af5d1f01-f9c6-4482-b7bb-331ce22a5810-kube-api-access-276jp\") pod \"test-curl-7852-6c1bab\" (UID: \"af5d1f01-f9c6-4482-b7bb-331ce22a5810\") " pod="opendatahub/test-curl-7852-6c1bab" Sep 17 23:33:25.305637 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:25.305595 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-276jp\" (UniqueName: \"kubernetes.io/projected/af5d1f01-f9c6-4482-b7bb-331ce22a5810-kube-api-access-276jp\") pod \"test-curl-7852-6c1bab\" (UID: \"af5d1f01-f9c6-4482-b7bb-331ce22a5810\") " pod="opendatahub/test-curl-7852-6c1bab" Sep 17 23:33:25.314312 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:25.314290 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-276jp\" (UniqueName: \"kubernetes.io/projected/af5d1f01-f9c6-4482-b7bb-331ce22a5810-kube-api-access-276jp\") pod \"test-curl-7852-6c1bab\" (UID: \"af5d1f01-f9c6-4482-b7bb-331ce22a5810\") " pod="opendatahub/test-curl-7852-6c1bab" Sep 17 23:33:25.457312 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:25.457242 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-6c1bab" Sep 17 23:33:25.725156 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:25.725126 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-6c1bab"] Sep 17 23:33:25.726971 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:33:25.726946 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf5d1f01_f9c6_4482_b7bb_331ce22a5810.slice/crio-94f5502d96e97bfebd7f4bc45246a44fc4585bb595400454cacb896bc01c0407 WatchSource:0}: Error finding container 94f5502d96e97bfebd7f4bc45246a44fc4585bb595400454cacb896bc01c0407: Status 404 returned error can't find the container with id 94f5502d96e97bfebd7f4bc45246a44fc4585bb595400454cacb896bc01c0407 Sep 17 23:33:26.284246 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:26.284210 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-6c1bab" event={"ID":"af5d1f01-f9c6-4482-b7bb-331ce22a5810","Type":"ContainerStarted","Data":"6abd4f3fd8919e34078c1e9e46a58bdf64836314bb79340c7ad6f9dd69660029"} Sep 17 23:33:26.284628 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:26.284263 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-6c1bab" event={"ID":"af5d1f01-f9c6-4482-b7bb-331ce22a5810","Type":"ContainerStarted","Data":"94f5502d96e97bfebd7f4bc45246a44fc4585bb595400454cacb896bc01c0407"} Sep 17 23:33:26.299707 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:26.299656 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7852-6c1bab" podStartSLOduration=1.299642157 podStartE2EDuration="1.299642157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:33:26.296933012 +0000 UTC m=+1645.230840147" watchObservedRunningTime="2026-09-17 23:33:26.299642157 +0000 UTC m=+1645.233549291" Sep 17 23:33:26.888805 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:26.888774 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7852-6c1bab"] Sep 17 23:33:26.892454 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:26.892428 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7852-6c1bab"] Sep 17 23:33:27.012517 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:27.012480 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7852-89321d"] Sep 17 23:33:27.012936 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:27.012923 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af5d1f01-f9c6-4482-b7bb-331ce22a5810" containerName="test-curl-7852-6c1bab" Sep 17 23:33:27.012981 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:27.012938 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="af5d1f01-f9c6-4482-b7bb-331ce22a5810" containerName="test-curl-7852-6c1bab" Sep 17 23:33:27.013024 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:27.013015 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4b44980-d812-40eb-b4ad-87d4af154876" containerName="istio-proxy" Sep 17 23:33:27.013059 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:27.013027 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="af5d1f01-f9c6-4482-b7bb-331ce22a5810" containerName="test-curl-7852-6c1bab" Sep 17 23:33:27.015176 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:27.015161 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-89321d" Sep 17 23:33:27.022003 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:27.021980 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-89321d"] Sep 17 23:33:27.123209 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:27.123181 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbkt8\" (UniqueName: \"kubernetes.io/projected/420e770a-454a-424d-85dc-abab75441ee3-kube-api-access-sbkt8\") pod \"test-curl-7852-89321d\" (UID: \"420e770a-454a-424d-85dc-abab75441ee3\") " pod="opendatahub/test-curl-7852-89321d" Sep 17 23:33:27.224181 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:27.224156 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbkt8\" (UniqueName: \"kubernetes.io/projected/420e770a-454a-424d-85dc-abab75441ee3-kube-api-access-sbkt8\") pod \"test-curl-7852-89321d\" (UID: \"420e770a-454a-424d-85dc-abab75441ee3\") " pod="opendatahub/test-curl-7852-89321d" Sep 17 23:33:27.232044 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:27.232019 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbkt8\" (UniqueName: \"kubernetes.io/projected/420e770a-454a-424d-85dc-abab75441ee3-kube-api-access-sbkt8\") pod \"test-curl-7852-89321d\" (UID: \"420e770a-454a-424d-85dc-abab75441ee3\") " pod="opendatahub/test-curl-7852-89321d" Sep 17 23:33:27.291656 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:33:27.291583 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7852-6c1bab" container="test-curl-7852-6c1bab" Sep 17 23:33:27.325138 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:27.325108 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-89321d" Sep 17 23:33:27.595827 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:27.595798 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-89321d"] Sep 17 23:33:27.597397 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:33:27.597361 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod420e770a_454a_424d_85dc_abab75441ee3.slice/crio-99608323f765a82cc2c39bdec70dfd3facbf033b18ff71774fe9bba0990597e1 WatchSource:0}: Error finding container 99608323f765a82cc2c39bdec70dfd3facbf033b18ff71774fe9bba0990597e1: Status 404 returned error can't find the container with id 99608323f765a82cc2c39bdec70dfd3facbf033b18ff71774fe9bba0990597e1 Sep 17 23:33:28.299383 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:28.299349 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-89321d" event={"ID":"420e770a-454a-424d-85dc-abab75441ee3","Type":"ContainerStarted","Data":"79943ae347635e33fae504d139361bc45c670f480231d00271316cb99ff43462"} Sep 17 23:33:28.299383 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:28.299391 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-89321d" event={"ID":"420e770a-454a-424d-85dc-abab75441ee3","Type":"ContainerStarted","Data":"99608323f765a82cc2c39bdec70dfd3facbf033b18ff71774fe9bba0990597e1"} Sep 17 23:33:28.299817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:28.299384 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7852-6c1bab" podUID="af5d1f01-f9c6-4482-b7bb-331ce22a5810" containerName="test-curl-7852-6c1bab" containerID="cri-o://6abd4f3fd8919e34078c1e9e46a58bdf64836314bb79340c7ad6f9dd69660029" gracePeriod=2 Sep 17 23:33:28.311876 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:28.311823 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7852-89321d" podStartSLOduration=2.311805245 podStartE2EDuration="2.311805245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:33:28.31142809 +0000 UTC m=+1647.245335229" watchObservedRunningTime="2026-09-17 23:33:28.311805245 +0000 UTC m=+1647.245712382" Sep 17 23:33:28.911334 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:28.911302 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7852-89321d"] Sep 17 23:33:28.918057 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:28.917996 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7852-89321d"] Sep 17 23:33:29.305096 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:33:29.305067 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7852-89321d" container="test-curl-7852-89321d" Sep 17 23:33:30.311660 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:30.311606 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7852-89321d" podUID="420e770a-454a-424d-85dc-abab75441ee3" containerName="test-curl-7852-89321d" containerID="cri-o://79943ae347635e33fae504d139361bc45c670f480231d00271316cb99ff43462" gracePeriod=2 Sep 17 23:33:30.439932 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:30.439911 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-6c1bab" Sep 17 23:33:30.554624 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:30.554564 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-276jp\" (UniqueName: \"kubernetes.io/projected/af5d1f01-f9c6-4482-b7bb-331ce22a5810-kube-api-access-276jp\") pod \"af5d1f01-f9c6-4482-b7bb-331ce22a5810\" (UID: \"af5d1f01-f9c6-4482-b7bb-331ce22a5810\") " Sep 17 23:33:30.556444 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:30.556416 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5d1f01-f9c6-4482-b7bb-331ce22a5810-kube-api-access-276jp" (OuterVolumeSpecName: "kube-api-access-276jp") pod "af5d1f01-f9c6-4482-b7bb-331ce22a5810" (UID: "af5d1f01-f9c6-4482-b7bb-331ce22a5810"). InnerVolumeSpecName "kube-api-access-276jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:33:30.655272 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:30.655251 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-276jp\" (UniqueName: \"kubernetes.io/projected/af5d1f01-f9c6-4482-b7bb-331ce22a5810-kube-api-access-276jp\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:31.293823 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:31.293789 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af5d1f01-f9c6-4482-b7bb-331ce22a5810" path="/var/lib/kubelet/pods/af5d1f01-f9c6-4482-b7bb-331ce22a5810/volumes" Sep 17 23:33:31.317657 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:31.317628 2582 generic.go:356] "Generic (PLEG): container finished" podID="af5d1f01-f9c6-4482-b7bb-331ce22a5810" containerID="6abd4f3fd8919e34078c1e9e46a58bdf64836314bb79340c7ad6f9dd69660029" exitCode=137 Sep 17 23:33:31.317986 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:31.317670 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-6c1bab" Sep 17 23:33:31.317986 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:31.317724 2582 scope.go:117] "RemoveContainer" containerID="6abd4f3fd8919e34078c1e9e46a58bdf64836314bb79340c7ad6f9dd69660029" Sep 17 23:33:31.326383 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:31.326366 2582 scope.go:117] "RemoveContainer" containerID="6abd4f3fd8919e34078c1e9e46a58bdf64836314bb79340c7ad6f9dd69660029" Sep 17 23:33:31.326667 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:33:31.326649 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6abd4f3fd8919e34078c1e9e46a58bdf64836314bb79340c7ad6f9dd69660029\": container with ID starting with 6abd4f3fd8919e34078c1e9e46a58bdf64836314bb79340c7ad6f9dd69660029 not found: ID does not exist" containerID="6abd4f3fd8919e34078c1e9e46a58bdf64836314bb79340c7ad6f9dd69660029" Sep 17 23:33:31.326739 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:31.326682 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6abd4f3fd8919e34078c1e9e46a58bdf64836314bb79340c7ad6f9dd69660029"} err="failed to get container status \"6abd4f3fd8919e34078c1e9e46a58bdf64836314bb79340c7ad6f9dd69660029\": rpc error: code = NotFound desc = could not find container \"6abd4f3fd8919e34078c1e9e46a58bdf64836314bb79340c7ad6f9dd69660029\": container with ID starting with 6abd4f3fd8919e34078c1e9e46a58bdf64836314bb79340c7ad6f9dd69660029 not found: ID does not exist" Sep 17 23:33:32.485523 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:32.485471 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-89321d" Sep 17 23:33:32.574840 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:32.574770 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbkt8\" (UniqueName: \"kubernetes.io/projected/420e770a-454a-424d-85dc-abab75441ee3-kube-api-access-sbkt8\") pod \"420e770a-454a-424d-85dc-abab75441ee3\" (UID: \"420e770a-454a-424d-85dc-abab75441ee3\") " Sep 17 23:33:32.577038 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:32.577013 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420e770a-454a-424d-85dc-abab75441ee3-kube-api-access-sbkt8" (OuterVolumeSpecName: "kube-api-access-sbkt8") pod "420e770a-454a-424d-85dc-abab75441ee3" (UID: "420e770a-454a-424d-85dc-abab75441ee3"). InnerVolumeSpecName "kube-api-access-sbkt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:33:32.675926 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:32.675897 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbkt8\" (UniqueName: \"kubernetes.io/projected/420e770a-454a-424d-85dc-abab75441ee3-kube-api-access-sbkt8\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:33.026326 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:33.026292 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz"] Sep 17 23:33:33.029167 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:33.029140 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cf8993-jtfgz"] Sep 17 23:33:33.295026 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:33.294944 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="420e770a-454a-424d-85dc-abab75441ee3" path="/var/lib/kubelet/pods/420e770a-454a-424d-85dc-abab75441ee3/volumes" Sep 17 23:33:33.295371 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:33.295351 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d93f62a2-e08d-4605-a234-9fc62d46b231" path="/var/lib/kubelet/pods/d93f62a2-e08d-4605-a234-9fc62d46b231/volumes" Sep 17 23:33:33.332665 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:33.332634 2582 generic.go:356] "Generic (PLEG): container finished" podID="420e770a-454a-424d-85dc-abab75441ee3" containerID="79943ae347635e33fae504d139361bc45c670f480231d00271316cb99ff43462" exitCode=137 Sep 17 23:33:33.332777 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:33.332672 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-89321d" Sep 17 23:33:33.332777 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:33.332702 2582 scope.go:117] "RemoveContainer" containerID="79943ae347635e33fae504d139361bc45c670f480231d00271316cb99ff43462" Sep 17 23:33:33.341390 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:33.341371 2582 scope.go:117] "RemoveContainer" containerID="79943ae347635e33fae504d139361bc45c670f480231d00271316cb99ff43462" Sep 17 23:33:33.341644 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:33:33.341629 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79943ae347635e33fae504d139361bc45c670f480231d00271316cb99ff43462\": container with ID starting with 79943ae347635e33fae504d139361bc45c670f480231d00271316cb99ff43462 not found: ID does not exist" containerID="79943ae347635e33fae504d139361bc45c670f480231d00271316cb99ff43462" Sep 17 23:33:33.341682 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:33.341651 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79943ae347635e33fae504d139361bc45c670f480231d00271316cb99ff43462"} err="failed to get container status \"79943ae347635e33fae504d139361bc45c670f480231d00271316cb99ff43462\": rpc error: code = NotFound desc = could not find container \"79943ae347635e33fae504d139361bc45c670f480231d00271316cb99ff43462\": container with ID starting with 79943ae347635e33fae504d139361bc45c670f480231d00271316cb99ff43462 not found: ID does not exist" Sep 17 23:33:39.337949 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:39.337900 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn"] Sep 17 23:33:39.338424 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:39.338159 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" podUID="c79f4562-a752-4e77-a0f4-7658ea1f6835" containerName="istio-proxy" containerID="cri-o://77a20b24eee4aec3e326bdfe91838cb1c49366bf9af9e2dabb4604abfde716b9" gracePeriod=30 Sep 17 23:33:39.693020 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:39.692943 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d69250-zzzms"] Sep 17 23:33:39.693485 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:39.693471 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="420e770a-454a-424d-85dc-abab75441ee3" containerName="test-curl-7852-89321d" Sep 17 23:33:39.693558 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:39.693487 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="420e770a-454a-424d-85dc-abab75441ee3" containerName="test-curl-7852-89321d" Sep 17 23:33:39.693600 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:39.693592 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="420e770a-454a-424d-85dc-abab75441ee3" containerName="test-curl-7852-89321d" Sep 17 23:33:39.696518 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:39.696483 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d69250-zzzms" Sep 17 23:33:39.698995 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:39.698971 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:33:39.703570 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:39.703523 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d69250-zzzms"] Sep 17 23:33:39.848265 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:39.848228 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafda0c6-6d75-4141-9671-442162b196ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d69250-zzzms\" (UID: \"cafda0c6-6d75-4141-9671-442162b196ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d69250-zzzms" Sep 17 23:33:39.949823 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:39.949736 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafda0c6-6d75-4141-9671-442162b196ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d69250-zzzms\" (UID: \"cafda0c6-6d75-4141-9671-442162b196ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d69250-zzzms" Sep 17 23:33:39.950335 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:39.950316 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafda0c6-6d75-4141-9671-442162b196ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d69250-zzzms\" (UID: \"cafda0c6-6d75-4141-9671-442162b196ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d69250-zzzms" Sep 17 23:33:40.009631 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:40.009595 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d69250-zzzms" Sep 17 23:33:40.284368 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:40.284344 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d69250-zzzms"] Sep 17 23:33:40.286198 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:33:40.286170 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafda0c6_6d75_4141_9671_442162b196ea.slice/crio-d7b1bc5d5ac3d39a396965a01338ca3274e2ab1ceef5eb25472fe2bc19e40e2b WatchSource:0}: Error finding container d7b1bc5d5ac3d39a396965a01338ca3274e2ab1ceef5eb25472fe2bc19e40e2b: Status 404 returned error can't find the container with id d7b1bc5d5ac3d39a396965a01338ca3274e2ab1ceef5eb25472fe2bc19e40e2b Sep 17 23:33:40.376534 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:40.376479 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d69250-zzzms" event={"ID":"cafda0c6-6d75-4141-9671-442162b196ea","Type":"ContainerStarted","Data":"d7b1bc5d5ac3d39a396965a01338ca3274e2ab1ceef5eb25472fe2bc19e40e2b"} Sep 17 23:33:41.384445 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:41.384361 2582 generic.go:356] "Generic (PLEG): container finished" podID="cafda0c6-6d75-4141-9671-442162b196ea" containerID="c0545a4429d0c0a621b447c6e4651461cb862f85609d111a5f4667afc65c730e" exitCode=0 Sep 17 23:33:41.384867 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:41.384454 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d69250-zzzms" event={"ID":"cafda0c6-6d75-4141-9671-442162b196ea","Type":"ContainerDied","Data":"c0545a4429d0c0a621b447c6e4651461cb862f85609d111a5f4667afc65c730e"} Sep 17 23:33:42.534232 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:42.534210 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d69250-zzzms" Sep 17 23:33:42.680185 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:42.680101 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafda0c6-6d75-4141-9671-442162b196ea-maas-api-service-ca\") pod \"cafda0c6-6d75-4141-9671-442162b196ea\" (UID: \"cafda0c6-6d75-4141-9671-442162b196ea\") " Sep 17 23:33:42.680465 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:42.680442 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafda0c6-6d75-4141-9671-442162b196ea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cafda0c6-6d75-4141-9671-442162b196ea" (UID: "cafda0c6-6d75-4141-9671-442162b196ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:33:42.781824 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:42.781791 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafda0c6-6d75-4141-9671-442162b196ea-maas-api-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:43.402977 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:43.402944 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d69250-zzzms" event={"ID":"cafda0c6-6d75-4141-9671-442162b196ea","Type":"ContainerDied","Data":"d7b1bc5d5ac3d39a396965a01338ca3274e2ab1ceef5eb25472fe2bc19e40e2b"} Sep 17 23:33:43.403142 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:43.402996 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7b1bc5d5ac3d39a396965a01338ca3274e2ab1ceef5eb25472fe2bc19e40e2b" Sep 17 23:33:43.403142 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:43.402954 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d69250-zzzms" Sep 17 23:33:44.607259 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.606994 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:33:44.703434 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.703401 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4wmv\" (UniqueName: \"kubernetes.io/projected/c79f4562-a752-4e77-a0f4-7658ea1f6835-kube-api-access-g4wmv\") pod \"c79f4562-a752-4e77-a0f4-7658ea1f6835\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " Sep 17 23:33:44.703662 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.703450 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-credential-socket\") pod \"c79f4562-a752-4e77-a0f4-7658ea1f6835\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " Sep 17 23:33:44.703662 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.703490 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-workload-socket\") pod \"c79f4562-a752-4e77-a0f4-7658ea1f6835\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " Sep 17 23:33:44.703662 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.703566 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-workload-certs\") pod \"c79f4562-a752-4e77-a0f4-7658ea1f6835\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " Sep 17 23:33:44.703662 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.703625 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-token\") pod \"c79f4562-a752-4e77-a0f4-7658ea1f6835\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " Sep 17 23:33:44.703876 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.703704 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-envoy\") pod \"c79f4562-a752-4e77-a0f4-7658ea1f6835\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " Sep 17 23:33:44.703876 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.703734 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c79f4562-a752-4e77-a0f4-7658ea1f6835-istiod-ca-cert\") pod \"c79f4562-a752-4e77-a0f4-7658ea1f6835\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " Sep 17 23:33:44.703876 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.703756 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c79f4562-a752-4e77-a0f4-7658ea1f6835" (UID: "c79f4562-a752-4e77-a0f4-7658ea1f6835"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:33:44.703876 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.703785 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-data\") pod \"c79f4562-a752-4e77-a0f4-7658ea1f6835\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " Sep 17 23:33:44.703876 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.703814 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-podinfo\") pod \"c79f4562-a752-4e77-a0f4-7658ea1f6835\" (UID: \"c79f4562-a752-4e77-a0f4-7658ea1f6835\") " Sep 17 23:33:44.703876 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.703818 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c79f4562-a752-4e77-a0f4-7658ea1f6835" (UID: "c79f4562-a752-4e77-a0f4-7658ea1f6835"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:33:44.703876 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.703848 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c79f4562-a752-4e77-a0f4-7658ea1f6835" (UID: "c79f4562-a752-4e77-a0f4-7658ea1f6835"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:33:44.704182 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.703986 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-data" (OuterVolumeSpecName: "istio-data") pod "c79f4562-a752-4e77-a0f4-7658ea1f6835" (UID: "c79f4562-a752-4e77-a0f4-7658ea1f6835"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:33:44.704182 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.704043 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c79f4562-a752-4e77-a0f4-7658ea1f6835-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c79f4562-a752-4e77-a0f4-7658ea1f6835" (UID: "c79f4562-a752-4e77-a0f4-7658ea1f6835"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:33:44.704293 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.704196 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-workload-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:44.704293 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.704215 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c79f4562-a752-4e77-a0f4-7658ea1f6835-istiod-ca-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:44.704293 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.704224 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-data\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:44.704293 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.704232 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-credential-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:44.704293 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.704240 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-workload-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:44.705937 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.705914 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c79f4562-a752-4e77-a0f4-7658ea1f6835" (UID: "c79f4562-a752-4e77-a0f4-7658ea1f6835"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:33:44.706068 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.705993 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c79f4562-a752-4e77-a0f4-7658ea1f6835" (UID: "c79f4562-a752-4e77-a0f4-7658ea1f6835"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:33:44.706173 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.706155 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79f4562-a752-4e77-a0f4-7658ea1f6835-kube-api-access-g4wmv" (OuterVolumeSpecName: "kube-api-access-g4wmv") pod "c79f4562-a752-4e77-a0f4-7658ea1f6835" (UID: "c79f4562-a752-4e77-a0f4-7658ea1f6835"). InnerVolumeSpecName "kube-api-access-g4wmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:33:44.706233 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.706177 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-token" (OuterVolumeSpecName: "istio-token") pod "c79f4562-a752-4e77-a0f4-7658ea1f6835" (UID: "c79f4562-a752-4e77-a0f4-7658ea1f6835"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:33:44.805299 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.805260 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-envoy\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:44.805299 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.805286 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-podinfo\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:44.805299 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.805296 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4wmv\" (UniqueName: \"kubernetes.io/projected/c79f4562-a752-4e77-a0f4-7658ea1f6835-kube-api-access-g4wmv\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:44.805299 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:44.805306 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c79f4562-a752-4e77-a0f4-7658ea1f6835-istio-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:45.421301 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:45.421267 2582 generic.go:356] "Generic (PLEG): container finished" podID="c79f4562-a752-4e77-a0f4-7658ea1f6835" containerID="77a20b24eee4aec3e326bdfe91838cb1c49366bf9af9e2dabb4604abfde716b9" exitCode=0 Sep 17 23:33:45.421476 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:45.421306 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" event={"ID":"c79f4562-a752-4e77-a0f4-7658ea1f6835","Type":"ContainerDied","Data":"77a20b24eee4aec3e326bdfe91838cb1c49366bf9af9e2dabb4604abfde716b9"} Sep 17 23:33:45.421476 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:45.421340 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" Sep 17 23:33:45.421476 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:45.421385 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn" event={"ID":"c79f4562-a752-4e77-a0f4-7658ea1f6835","Type":"ContainerDied","Data":"91b2898018b205529e24866e4f281aafc561a6f16c73c10907ba2d6d109ad294"} Sep 17 23:33:45.421476 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:45.421409 2582 scope.go:117] "RemoveContainer" containerID="77a20b24eee4aec3e326bdfe91838cb1c49366bf9af9e2dabb4604abfde716b9" Sep 17 23:33:45.431363 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:45.431346 2582 scope.go:117] "RemoveContainer" containerID="e65d53ddfdd72815b213070c4d40dd7816ed8213f5ed0338797e43b1594bb5d9" Sep 17 23:33:45.439566 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:45.439542 2582 scope.go:117] "RemoveContainer" containerID="77a20b24eee4aec3e326bdfe91838cb1c49366bf9af9e2dabb4604abfde716b9" Sep 17 23:33:45.439810 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:33:45.439791 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a20b24eee4aec3e326bdfe91838cb1c49366bf9af9e2dabb4604abfde716b9\": container with ID starting with 77a20b24eee4aec3e326bdfe91838cb1c49366bf9af9e2dabb4604abfde716b9 not found: ID does not exist" containerID="77a20b24eee4aec3e326bdfe91838cb1c49366bf9af9e2dabb4604abfde716b9" Sep 17 23:33:45.439869 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:45.439817 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a20b24eee4aec3e326bdfe91838cb1c49366bf9af9e2dabb4604abfde716b9"} err="failed to get container status \"77a20b24eee4aec3e326bdfe91838cb1c49366bf9af9e2dabb4604abfde716b9\": rpc error: code = NotFound desc = could not find container \"77a20b24eee4aec3e326bdfe91838cb1c49366bf9af9e2dabb4604abfde716b9\": container with ID starting with 77a20b24eee4aec3e326bdfe91838cb1c49366bf9af9e2dabb4604abfde716b9 not found: ID does not exist" Sep 17 23:33:45.439869 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:45.439834 2582 scope.go:117] "RemoveContainer" containerID="e65d53ddfdd72815b213070c4d40dd7816ed8213f5ed0338797e43b1594bb5d9" Sep 17 23:33:45.440082 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:33:45.440062 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65d53ddfdd72815b213070c4d40dd7816ed8213f5ed0338797e43b1594bb5d9\": container with ID starting with e65d53ddfdd72815b213070c4d40dd7816ed8213f5ed0338797e43b1594bb5d9 not found: ID does not exist" containerID="e65d53ddfdd72815b213070c4d40dd7816ed8213f5ed0338797e43b1594bb5d9" Sep 17 23:33:45.440197 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:45.440086 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65d53ddfdd72815b213070c4d40dd7816ed8213f5ed0338797e43b1594bb5d9"} err="failed to get container status \"e65d53ddfdd72815b213070c4d40dd7816ed8213f5ed0338797e43b1594bb5d9\": rpc error: code = NotFound desc = could not find container \"e65d53ddfdd72815b213070c4d40dd7816ed8213f5ed0338797e43b1594bb5d9\": container with ID starting with e65d53ddfdd72815b213070c4d40dd7816ed8213f5ed0338797e43b1594bb5d9 not found: ID does not exist" Sep 17 23:33:45.443091 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:45.443072 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn"] Sep 17 23:33:45.447207 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:45.447187 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cf8993-openshift-default-75558d8c5d-rh7qn"] Sep 17 23:33:47.295024 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:47.294981 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c79f4562-a752-4e77-a0f4-7658ea1f6835" path="/var/lib/kubelet/pods/c79f4562-a752-4e77-a0f4-7658ea1f6835/volumes" Sep 17 23:33:50.311109 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:50.311075 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl"] Sep 17 23:33:50.311661 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:50.311341 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl" podUID="825f8f34-8b44-4123-962e-4e20a859326d" containerName="payload-processing" containerID="cri-o://96fec26353d1b193685be39c38e404719cd493802892d2d52778e6a93465417b" gracePeriod=30 Sep 17 23:33:50.456190 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:50.456157 2582 generic.go:356] "Generic (PLEG): container finished" podID="825f8f34-8b44-4123-962e-4e20a859326d" containerID="96fec26353d1b193685be39c38e404719cd493802892d2d52778e6a93465417b" exitCode=0 Sep 17 23:33:50.456349 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:50.456244 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl" event={"ID":"825f8f34-8b44-4123-962e-4e20a859326d","Type":"ContainerDied","Data":"96fec26353d1b193685be39c38e404719cd493802892d2d52778e6a93465417b"} Sep 17 23:33:50.483227 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:50.481175 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d69250-zzzms"] Sep 17 23:33:50.484557 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:50.484491 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d69250-zzzms"] Sep 17 23:33:50.562719 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:50.562659 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl" Sep 17 23:33:50.670530 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:50.670479 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825f8f34-8b44-4123-962e-4e20a859326d-plugins-config-volume\") pod \"825f8f34-8b44-4123-962e-4e20a859326d\" (UID: \"825f8f34-8b44-4123-962e-4e20a859326d\") " Sep 17 23:33:50.670793 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:50.670773 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks2fj\" (UniqueName: \"kubernetes.io/projected/825f8f34-8b44-4123-962e-4e20a859326d-kube-api-access-ks2fj\") pod \"825f8f34-8b44-4123-962e-4e20a859326d\" (UID: \"825f8f34-8b44-4123-962e-4e20a859326d\") " Sep 17 23:33:50.671187 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:50.670867 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825f8f34-8b44-4123-962e-4e20a859326d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "825f8f34-8b44-4123-962e-4e20a859326d" (UID: "825f8f34-8b44-4123-962e-4e20a859326d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:33:50.671548 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:50.671529 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825f8f34-8b44-4123-962e-4e20a859326d-plugins-config-volume\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:50.673413 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:50.673341 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825f8f34-8b44-4123-962e-4e20a859326d-kube-api-access-ks2fj" (OuterVolumeSpecName: "kube-api-access-ks2fj") pod "825f8f34-8b44-4123-962e-4e20a859326d" (UID: "825f8f34-8b44-4123-962e-4e20a859326d"). InnerVolumeSpecName "kube-api-access-ks2fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:33:50.773366 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:50.773318 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks2fj\" (UniqueName: \"kubernetes.io/projected/825f8f34-8b44-4123-962e-4e20a859326d-kube-api-access-ks2fj\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:33:51.294771 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:51.294736 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cafda0c6-6d75-4141-9671-442162b196ea" path="/var/lib/kubelet/pods/cafda0c6-6d75-4141-9671-442162b196ea/volumes" Sep 17 23:33:51.463254 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:51.463225 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl" event={"ID":"825f8f34-8b44-4123-962e-4e20a859326d","Type":"ContainerDied","Data":"1a0e99d71b9769a837342e14d06f157264083ad7a8d96ee0421685280b56ac6b"} Sep 17 23:33:51.463688 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:51.463264 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl" Sep 17 23:33:51.463688 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:51.463282 2582 scope.go:117] "RemoveContainer" containerID="96fec26353d1b193685be39c38e404719cd493802892d2d52778e6a93465417b" Sep 17 23:33:51.481433 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:51.481396 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl"] Sep 17 23:33:51.485272 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:51.485248 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d69250-7c976cc9df-99mpl"] Sep 17 23:33:53.294726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:53.294693 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="825f8f34-8b44-4123-962e-4e20a859326d" path="/var/lib/kubelet/pods/825f8f34-8b44-4123-962e-4e20a859326d/volumes" Sep 17 23:33:56.892577 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:56.892543 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc"] Sep 17 23:33:56.893031 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:56.892785 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" podUID="64b1a682-e6fa-460e-a462-14827fb346a4" containerName="istio-proxy" containerID="cri-o://8bf6aadeb6bc1acbfd3136a4430df35e93f5d6f2e6148afd19bfc0436715c611" gracePeriod=30 Sep 17 23:33:57.274008 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:57.273965 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6"] Sep 17 23:33:57.274687 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:57.274662 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c79f4562-a752-4e77-a0f4-7658ea1f6835" containerName="istio-proxy" Sep 17 23:33:57.274793 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:57.274690 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c79f4562-a752-4e77-a0f4-7658ea1f6835" containerName="istio-proxy" Sep 17 23:33:57.274793 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:57.274722 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="825f8f34-8b44-4123-962e-4e20a859326d" containerName="payload-processing" Sep 17 23:33:57.274793 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:57.274731 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="825f8f34-8b44-4123-962e-4e20a859326d" containerName="payload-processing" Sep 17 23:33:57.274793 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:57.274749 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cafda0c6-6d75-4141-9671-442162b196ea" containerName="revoke-keys" Sep 17 23:33:57.274793 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:57.274759 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cafda0c6-6d75-4141-9671-442162b196ea" containerName="revoke-keys" Sep 17 23:33:57.274982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:57.274894 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c79f4562-a752-4e77-a0f4-7658ea1f6835" containerName="istio-proxy" Sep 17 23:33:57.274982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:57.274907 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="825f8f34-8b44-4123-962e-4e20a859326d" containerName="payload-processing" Sep 17 23:33:57.274982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:57.274916 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cafda0c6-6d75-4141-9671-442162b196ea" containerName="revoke-keys" Sep 17 23:33:57.274982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:57.274927 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c79f4562-a752-4e77-a0f4-7658ea1f6835" containerName="istio-proxy" Sep 17 23:33:57.277559 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:57.277537 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6" Sep 17 23:33:57.280346 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:57.280318 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:33:57.283344 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:57.283276 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6"] Sep 17 23:33:57.343295 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:57.343252 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33cd46e0-8439-4167-916a-1b5f17676cc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6\" (UID: \"33cd46e0-8439-4167-916a-1b5f17676cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6" Sep 17 23:33:57.444363 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:57.444327 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33cd46e0-8439-4167-916a-1b5f17676cc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6\" (UID: \"33cd46e0-8439-4167-916a-1b5f17676cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6" Sep 17 23:33:57.444990 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:57.444959 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33cd46e0-8439-4167-916a-1b5f17676cc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6\" (UID: \"33cd46e0-8439-4167-916a-1b5f17676cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6" Sep 17 23:33:57.591621 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:57.591544 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6" Sep 17 23:33:57.864223 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:57.864186 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6"] Sep 17 23:33:57.865746 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:33:57.865715 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33cd46e0_8439_4167_916a_1b5f17676cc0.slice/crio-3f2d078bdc2c8c7cc01cb7dc19b9dccc0c810daba95b548540827dc69321c7e5 WatchSource:0}: Error finding container 3f2d078bdc2c8c7cc01cb7dc19b9dccc0c810daba95b548540827dc69321c7e5: Status 404 returned error can't find the container with id 3f2d078bdc2c8c7cc01cb7dc19b9dccc0c810daba95b548540827dc69321c7e5 Sep 17 23:33:58.508287 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:58.508247 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6" event={"ID":"33cd46e0-8439-4167-916a-1b5f17676cc0","Type":"ContainerStarted","Data":"6fbd1ed76b03dfe2cb2df21675a2de8840668d7e1cb3b850863f5e16fd6f3f5d"} Sep 17 23:33:58.508763 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:58.508298 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6" event={"ID":"33cd46e0-8439-4167-916a-1b5f17676cc0","Type":"ContainerStarted","Data":"3f2d078bdc2c8c7cc01cb7dc19b9dccc0c810daba95b548540827dc69321c7e5"} Sep 17 23:33:58.523576 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:58.523528 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6" podStartSLOduration=1.523513125 podStartE2EDuration="1.523513125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:33:58.521608596 +0000 UTC m=+1677.455515733" watchObservedRunningTime="2026-09-17 23:33:58.523513125 +0000 UTC m=+1677.457420252" Sep 17 23:33:59.517241 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:59.517210 2582 generic.go:356] "Generic (PLEG): container finished" podID="33cd46e0-8439-4167-916a-1b5f17676cc0" containerID="6fbd1ed76b03dfe2cb2df21675a2de8840668d7e1cb3b850863f5e16fd6f3f5d" exitCode=0 Sep 17 23:33:59.517644 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:59.517271 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6" event={"ID":"33cd46e0-8439-4167-916a-1b5f17676cc0","Type":"ContainerDied","Data":"6fbd1ed76b03dfe2cb2df21675a2de8840668d7e1cb3b850863f5e16fd6f3f5d"} Sep 17 23:33:59.834216 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:59.834139 2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 23:33:59.834354 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:33:59.834226 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" podUID="64b1a682-e6fa-460e-a462-14827fb346a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 23:34:00.667263 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:00.667240 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6" Sep 17 23:34:00.780464 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:00.780384 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33cd46e0-8439-4167-916a-1b5f17676cc0-maas-api-service-ca\") pod \"33cd46e0-8439-4167-916a-1b5f17676cc0\" (UID: \"33cd46e0-8439-4167-916a-1b5f17676cc0\") " Sep 17 23:34:00.780799 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:00.780779 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33cd46e0-8439-4167-916a-1b5f17676cc0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33cd46e0-8439-4167-916a-1b5f17676cc0" (UID: "33cd46e0-8439-4167-916a-1b5f17676cc0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:34:00.882096 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:00.882068 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33cd46e0-8439-4167-916a-1b5f17676cc0-maas-api-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:34:01.532691 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:01.532663 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6" Sep 17 23:34:01.532860 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:01.532693 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6" event={"ID":"33cd46e0-8439-4167-916a-1b5f17676cc0","Type":"ContainerDied","Data":"3f2d078bdc2c8c7cc01cb7dc19b9dccc0c810daba95b548540827dc69321c7e5"} Sep 17 23:34:01.532860 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:01.532745 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f2d078bdc2c8c7cc01cb7dc19b9dccc0c810daba95b548540827dc69321c7e5" Sep 17 23:34:02.153082 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.153059 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:34:02.293528 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.293486 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-istio-data\") pod \"64b1a682-e6fa-460e-a462-14827fb346a4\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " Sep 17 23:34:02.293716 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.293567 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-credential-socket\") pod \"64b1a682-e6fa-460e-a462-14827fb346a4\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " Sep 17 23:34:02.293716 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.293613 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-istio-envoy\") pod \"64b1a682-e6fa-460e-a462-14827fb346a4\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " Sep 17 23:34:02.293716 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.293631 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64b1a682-e6fa-460e-a462-14827fb346a4-istiod-ca-cert\") pod \"64b1a682-e6fa-460e-a462-14827fb346a4\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " Sep 17 23:34:02.293716 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.293704 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9vlv\" (UniqueName: \"kubernetes.io/projected/64b1a682-e6fa-460e-a462-14827fb346a4-kube-api-access-x9vlv\") pod \"64b1a682-e6fa-460e-a462-14827fb346a4\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " Sep 17 23:34:02.294014 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.293734 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64b1a682-e6fa-460e-a462-14827fb346a4-istio-token\") pod \"64b1a682-e6fa-460e-a462-14827fb346a4\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " Sep 17 23:34:02.294014 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.293768 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-workload-socket\") pod \"64b1a682-e6fa-460e-a462-14827fb346a4\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " Sep 17 23:34:02.294014 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.293790 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-workload-certs\") pod \"64b1a682-e6fa-460e-a462-14827fb346a4\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " Sep 17 23:34:02.294014 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.293782 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-istio-data" (OuterVolumeSpecName: "istio-data") pod "64b1a682-e6fa-460e-a462-14827fb346a4" (UID: "64b1a682-e6fa-460e-a462-14827fb346a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:34:02.294014 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.293800 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "64b1a682-e6fa-460e-a462-14827fb346a4" (UID: "64b1a682-e6fa-460e-a462-14827fb346a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:34:02.294283 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.294072 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64b1a682-e6fa-460e-a462-14827fb346a4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "64b1a682-e6fa-460e-a462-14827fb346a4" (UID: "64b1a682-e6fa-460e-a462-14827fb346a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:34:02.294283 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.294160 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "64b1a682-e6fa-460e-a462-14827fb346a4" (UID: "64b1a682-e6fa-460e-a462-14827fb346a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:34:02.294283 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.294184 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64b1a682-e6fa-460e-a462-14827fb346a4-istio-podinfo\") pod \"64b1a682-e6fa-460e-a462-14827fb346a4\" (UID: \"64b1a682-e6fa-460e-a462-14827fb346a4\") " Sep 17 23:34:02.294704 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.294648 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64b1a682-e6fa-460e-a462-14827fb346a4-istiod-ca-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:34:02.294704 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.294676 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-workload-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:34:02.294704 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.294691 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-istio-data\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:34:02.294909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.294705 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-credential-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:34:02.294972 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.294937 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "64b1a682-e6fa-460e-a462-14827fb346a4" (UID: "64b1a682-e6fa-460e-a462-14827fb346a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:34:02.295916 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.295889 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b1a682-e6fa-460e-a462-14827fb346a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "64b1a682-e6fa-460e-a462-14827fb346a4" (UID: "64b1a682-e6fa-460e-a462-14827fb346a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:34:02.296020 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.295987 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "64b1a682-e6fa-460e-a462-14827fb346a4" (UID: "64b1a682-e6fa-460e-a462-14827fb346a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:34:02.296093 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.296071 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b1a682-e6fa-460e-a462-14827fb346a4-kube-api-access-x9vlv" (OuterVolumeSpecName: "kube-api-access-x9vlv") pod "64b1a682-e6fa-460e-a462-14827fb346a4" (UID: "64b1a682-e6fa-460e-a462-14827fb346a4"). InnerVolumeSpecName "kube-api-access-x9vlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:34:02.296227 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.296209 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64b1a682-e6fa-460e-a462-14827fb346a4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "64b1a682-e6fa-460e-a462-14827fb346a4" (UID: "64b1a682-e6fa-460e-a462-14827fb346a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:34:02.396058 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.396023 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-istio-envoy\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:34:02.396058 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.396057 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9vlv\" (UniqueName: \"kubernetes.io/projected/64b1a682-e6fa-460e-a462-14827fb346a4-kube-api-access-x9vlv\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:34:02.396302 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.396071 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64b1a682-e6fa-460e-a462-14827fb346a4-istio-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:34:02.396302 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.396085 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64b1a682-e6fa-460e-a462-14827fb346a4-workload-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:34:02.396302 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.396098 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64b1a682-e6fa-460e-a462-14827fb346a4-istio-podinfo\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:34:02.547587 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.547481 2582 generic.go:356] "Generic (PLEG): container finished" podID="64b1a682-e6fa-460e-a462-14827fb346a4" containerID="8bf6aadeb6bc1acbfd3136a4430df35e93f5d6f2e6148afd19bfc0436715c611" exitCode=0 Sep 17 23:34:02.547587 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.547552 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" event={"ID":"64b1a682-e6fa-460e-a462-14827fb346a4","Type":"ContainerDied","Data":"8bf6aadeb6bc1acbfd3136a4430df35e93f5d6f2e6148afd19bfc0436715c611"} Sep 17 23:34:02.547587 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.547556 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" Sep 17 23:34:02.547831 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.547614 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc" event={"ID":"64b1a682-e6fa-460e-a462-14827fb346a4","Type":"ContainerDied","Data":"37078798fbd6bdb6adfb58acb766f44c23109a89f82b4b6f6cf4f57e6e50c1e5"} Sep 17 23:34:02.547831 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.547635 2582 scope.go:117] "RemoveContainer" containerID="8bf6aadeb6bc1acbfd3136a4430df35e93f5d6f2e6148afd19bfc0436715c611" Sep 17 23:34:02.557858 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.557839 2582 scope.go:117] "RemoveContainer" containerID="8644db3c01e483f9bffe6d7d7fdae0808f3fc25c9fd6b3880ab7f439f4a4f33b" Sep 17 23:34:02.566677 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.566660 2582 scope.go:117] "RemoveContainer" containerID="8bf6aadeb6bc1acbfd3136a4430df35e93f5d6f2e6148afd19bfc0436715c611" Sep 17 23:34:02.566946 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:34:02.566925 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf6aadeb6bc1acbfd3136a4430df35e93f5d6f2e6148afd19bfc0436715c611\": container with ID starting with 8bf6aadeb6bc1acbfd3136a4430df35e93f5d6f2e6148afd19bfc0436715c611 not found: ID does not exist" containerID="8bf6aadeb6bc1acbfd3136a4430df35e93f5d6f2e6148afd19bfc0436715c611" Sep 17 23:34:02.566994 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.566954 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf6aadeb6bc1acbfd3136a4430df35e93f5d6f2e6148afd19bfc0436715c611"} err="failed to get container status \"8bf6aadeb6bc1acbfd3136a4430df35e93f5d6f2e6148afd19bfc0436715c611\": rpc error: code = NotFound desc = could not find container \"8bf6aadeb6bc1acbfd3136a4430df35e93f5d6f2e6148afd19bfc0436715c611\": container with ID starting with 8bf6aadeb6bc1acbfd3136a4430df35e93f5d6f2e6148afd19bfc0436715c611 not found: ID does not exist" Sep 17 23:34:02.566994 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.566971 2582 scope.go:117] "RemoveContainer" containerID="8644db3c01e483f9bffe6d7d7fdae0808f3fc25c9fd6b3880ab7f439f4a4f33b" Sep 17 23:34:02.567244 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:34:02.567224 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8644db3c01e483f9bffe6d7d7fdae0808f3fc25c9fd6b3880ab7f439f4a4f33b\": container with ID starting with 8644db3c01e483f9bffe6d7d7fdae0808f3fc25c9fd6b3880ab7f439f4a4f33b not found: ID does not exist" containerID="8644db3c01e483f9bffe6d7d7fdae0808f3fc25c9fd6b3880ab7f439f4a4f33b" Sep 17 23:34:02.567346 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.567252 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8644db3c01e483f9bffe6d7d7fdae0808f3fc25c9fd6b3880ab7f439f4a4f33b"} err="failed to get container status \"8644db3c01e483f9bffe6d7d7fdae0808f3fc25c9fd6b3880ab7f439f4a4f33b\": rpc error: code = NotFound desc = could not find container \"8644db3c01e483f9bffe6d7d7fdae0808f3fc25c9fd6b3880ab7f439f4a4f33b\": container with ID starting with 8644db3c01e483f9bffe6d7d7fdae0808f3fc25c9fd6b3880ab7f439f4a4f33b not found: ID does not exist" Sep 17 23:34:02.569623 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.569598 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc"] Sep 17 23:34:02.573774 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:02.573752 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d69250-openshift-default-6db8cc5898-55vhc"] Sep 17 23:34:03.294175 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:03.294141 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64b1a682-e6fa-460e-a462-14827fb346a4" path="/var/lib/kubelet/pods/64b1a682-e6fa-460e-a462-14827fb346a4/volumes" Sep 17 23:34:03.571054 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:03.570974 2582 scope.go:117] "RemoveContainer" containerID="97cd8986b9c01c48f04d13219631d424f0cbf637427e3855e278e69e9568bf02" Sep 17 23:34:08.059002 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.058969 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7"] Sep 17 23:34:08.059943 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.059910 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7" podUID="04502971-5f33-4f73-b07e-5025585c6be3" containerName="payload-pre-processing" containerID="cri-o://e1f954028cc9acd1d4771e1501639cdb8777fb0caa787fb6ed83eb58b268505e" gracePeriod=30 Sep 17 23:34:08.217715 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.217684 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6"] Sep 17 23:34:08.221067 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.221042 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fd82a9-7vds6"] Sep 17 23:34:08.332442 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.332408 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7" Sep 17 23:34:08.451332 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.451301 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04502971-5f33-4f73-b07e-5025585c6be3-plugins-config-volume\") pod \"04502971-5f33-4f73-b07e-5025585c6be3\" (UID: \"04502971-5f33-4f73-b07e-5025585c6be3\") " Sep 17 23:34:08.451570 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.451351 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v87bb\" (UniqueName: \"kubernetes.io/projected/04502971-5f33-4f73-b07e-5025585c6be3-kube-api-access-v87bb\") pod \"04502971-5f33-4f73-b07e-5025585c6be3\" (UID: \"04502971-5f33-4f73-b07e-5025585c6be3\") " Sep 17 23:34:08.451680 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.451654 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04502971-5f33-4f73-b07e-5025585c6be3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "04502971-5f33-4f73-b07e-5025585c6be3" (UID: "04502971-5f33-4f73-b07e-5025585c6be3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:34:08.451850 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.451831 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04502971-5f33-4f73-b07e-5025585c6be3-plugins-config-volume\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:34:08.453653 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.453624 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04502971-5f33-4f73-b07e-5025585c6be3-kube-api-access-v87bb" (OuterVolumeSpecName: "kube-api-access-v87bb") pod "04502971-5f33-4f73-b07e-5025585c6be3" (UID: "04502971-5f33-4f73-b07e-5025585c6be3"). InnerVolumeSpecName "kube-api-access-v87bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:34:08.552655 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.552622 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v87bb\" (UniqueName: \"kubernetes.io/projected/04502971-5f33-4f73-b07e-5025585c6be3-kube-api-access-v87bb\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:34:08.586899 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.586807 2582 generic.go:356] "Generic (PLEG): container finished" podID="04502971-5f33-4f73-b07e-5025585c6be3" containerID="e1f954028cc9acd1d4771e1501639cdb8777fb0caa787fb6ed83eb58b268505e" exitCode=0 Sep 17 23:34:08.587055 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.586911 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7" Sep 17 23:34:08.587055 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.586947 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7" event={"ID":"04502971-5f33-4f73-b07e-5025585c6be3","Type":"ContainerDied","Data":"e1f954028cc9acd1d4771e1501639cdb8777fb0caa787fb6ed83eb58b268505e"} Sep 17 23:34:08.587055 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.587008 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7" event={"ID":"04502971-5f33-4f73-b07e-5025585c6be3","Type":"ContainerDied","Data":"a73d47cb173cc5ccaf7191012fd41623d2d6e3cece59771f4839f275025b37c1"} Sep 17 23:34:08.587055 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.587036 2582 scope.go:117] "RemoveContainer" containerID="e1f954028cc9acd1d4771e1501639cdb8777fb0caa787fb6ed83eb58b268505e" Sep 17 23:34:08.597450 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.597429 2582 scope.go:117] "RemoveContainer" containerID="e1f954028cc9acd1d4771e1501639cdb8777fb0caa787fb6ed83eb58b268505e" Sep 17 23:34:08.597754 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:34:08.597733 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f954028cc9acd1d4771e1501639cdb8777fb0caa787fb6ed83eb58b268505e\": container with ID starting with e1f954028cc9acd1d4771e1501639cdb8777fb0caa787fb6ed83eb58b268505e not found: ID does not exist" containerID="e1f954028cc9acd1d4771e1501639cdb8777fb0caa787fb6ed83eb58b268505e" Sep 17 23:34:08.597808 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.597763 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f954028cc9acd1d4771e1501639cdb8777fb0caa787fb6ed83eb58b268505e"} err="failed to get container status \"e1f954028cc9acd1d4771e1501639cdb8777fb0caa787fb6ed83eb58b268505e\": rpc error: code = NotFound desc = could not find container \"e1f954028cc9acd1d4771e1501639cdb8777fb0caa787fb6ed83eb58b268505e\": container with ID starting with e1f954028cc9acd1d4771e1501639cdb8777fb0caa787fb6ed83eb58b268505e not found: ID does not exist" Sep 17 23:34:08.610313 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.610288 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7"] Sep 17 23:34:08.612429 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:08.612407 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fd82a9-75ff44b7b6-d9vt7"] Sep 17 23:34:09.294248 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:09.294216 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04502971-5f33-4f73-b07e-5025585c6be3" path="/var/lib/kubelet/pods/04502971-5f33-4f73-b07e-5025585c6be3/volumes" Sep 17 23:34:09.294630 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:34:09.294568 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33cd46e0-8439-4167-916a-1b5f17676cc0" path="/var/lib/kubelet/pods/33cd46e0-8439-4167-916a-1b5f17676cc0/volumes" Sep 17 23:35:28.864211 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.864174 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm"] Sep 17 23:35:28.864681 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.864605 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04502971-5f33-4f73-b07e-5025585c6be3" containerName="payload-pre-processing" Sep 17 23:35:28.864681 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.864617 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="04502971-5f33-4f73-b07e-5025585c6be3" containerName="payload-pre-processing" Sep 17 23:35:28.864681 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.864628 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64b1a682-e6fa-460e-a462-14827fb346a4" containerName="istio-proxy" Sep 17 23:35:28.864681 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.864633 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="64b1a682-e6fa-460e-a462-14827fb346a4" containerName="istio-proxy" Sep 17 23:35:28.864681 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.864643 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33cd46e0-8439-4167-916a-1b5f17676cc0" containerName="revoke-keys" Sep 17 23:35:28.864681 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.864648 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="33cd46e0-8439-4167-916a-1b5f17676cc0" containerName="revoke-keys" Sep 17 23:35:28.864681 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.864662 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c79f4562-a752-4e77-a0f4-7658ea1f6835" containerName="istio-proxy" Sep 17 23:35:28.864681 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.864668 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c79f4562-a752-4e77-a0f4-7658ea1f6835" containerName="istio-proxy" Sep 17 23:35:28.864681 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.864676 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64b1a682-e6fa-460e-a462-14827fb346a4" containerName="istio-proxy" Sep 17 23:35:28.864681 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.864681 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="64b1a682-e6fa-460e-a462-14827fb346a4" containerName="istio-proxy" Sep 17 23:35:28.865056 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.864745 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="64b1a682-e6fa-460e-a462-14827fb346a4" containerName="istio-proxy" Sep 17 23:35:28.865056 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.864752 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="33cd46e0-8439-4167-916a-1b5f17676cc0" containerName="revoke-keys" Sep 17 23:35:28.865056 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.864758 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="64b1a682-e6fa-460e-a462-14827fb346a4" containerName="istio-proxy" Sep 17 23:35:28.865056 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.864765 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="04502971-5f33-4f73-b07e-5025585c6be3" containerName="payload-pre-processing" Sep 17 23:35:28.867334 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.867314 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:28.870255 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.870230 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-8f81ee-openshift-default-dockercfg-mxv62\"" Sep 17 23:35:28.882964 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.882927 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm"] Sep 17 23:35:28.998715 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.998679 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-workload-certs\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:28.998894 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.998733 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-data\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:28.998894 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.998811 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istiod-ca-cert\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:28.998894 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.998870 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-podinfo\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:28.999031 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.998924 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-envoy\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:28.999031 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.998958 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-credential-socket\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:28.999031 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.998985 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mqll\" (UniqueName: \"kubernetes.io/projected/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-kube-api-access-6mqll\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:28.999201 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.999048 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-token\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:28.999201 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:28.999120 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-workload-socket\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.101036 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.100982 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-podinfo\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.101334 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.101313 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-envoy\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.102042 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.102003 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-credential-socket\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.102398 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.102347 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mqll\" (UniqueName: \"kubernetes.io/projected/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-kube-api-access-6mqll\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.102566 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.102443 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-token\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.102566 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.102487 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-workload-socket\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.102700 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.102569 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-workload-certs\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.102700 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.102632 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-data\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.102700 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.102680 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istiod-ca-cert\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.103192 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.103148 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-workload-certs\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.103297 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.103211 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-workload-socket\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.103297 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.103252 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-data\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.103460 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.103436 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istiod-ca-cert\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.103551 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.103467 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-credential-socket\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.103969 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.103950 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-envoy\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.104237 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.104219 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-podinfo\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.115821 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.115768 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-token\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.115937 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.115920 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mqll\" (UniqueName: \"kubernetes.io/projected/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-kube-api-access-6mqll\") pod \"e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.178921 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.178892 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:29.476174 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.476143 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm"] Sep 17 23:35:29.479654 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:35:29.479627 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode67e0b2a_0666_4cd7_a0d2_0c845adc9dba.slice/crio-f254a61599106a6499843e0120a52ede36912c46199d74026f214fc5f33db5fb WatchSource:0}: Error finding container f254a61599106a6499843e0120a52ede36912c46199d74026f214fc5f33db5fb: Status 404 returned error can't find the container with id f254a61599106a6499843e0120a52ede36912c46199d74026f214fc5f33db5fb Sep 17 23:35:29.481872 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.481843 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:35:29.481976 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.481905 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:35:29.481976 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:29.481935 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:35:30.062862 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:30.062827 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" event={"ID":"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba","Type":"ContainerStarted","Data":"10b0d6097e6c1666e6e0e896dba279cb94d4e1d5f7c6597001d612b9559a25e6"} Sep 17 23:35:30.063248 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:30.062879 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" event={"ID":"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba","Type":"ContainerStarted","Data":"f254a61599106a6499843e0120a52ede36912c46199d74026f214fc5f33db5fb"} Sep 17 23:35:30.085611 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:30.085563 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" podStartSLOduration=2.085546838 podStartE2EDuration="2.085546838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:35:30.083647493 +0000 UTC m=+1769.017554828" watchObservedRunningTime="2026-09-17 23:35:30.085546838 +0000 UTC m=+1769.019453974" Sep 17 23:35:30.179653 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:30.179620 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:30.184513 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:30.184475 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:31.068755 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:31.068718 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:31.070145 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:31.070115 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:34.464209 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:34.464174 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55"] Sep 17 23:35:34.468101 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:34.468081 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55" Sep 17 23:35:34.470852 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:34.470808 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8f81ee\"" Sep 17 23:35:34.471032 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:34.471013 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8f81ee-dockercfg-ns2mn\"" Sep 17 23:35:34.478608 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:34.478567 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55"] Sep 17 23:35:34.558468 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:34.558427 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24rqc\" (UniqueName: \"kubernetes.io/projected/5ec0ba84-ad9b-4584-af7c-044629bd6876-kube-api-access-24rqc\") pod \"payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55\" (UID: \"5ec0ba84-ad9b-4584-af7c-044629bd6876\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55" Sep 17 23:35:34.558665 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:34.558586 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec0ba84-ad9b-4584-af7c-044629bd6876-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55\" (UID: \"5ec0ba84-ad9b-4584-af7c-044629bd6876\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55" Sep 17 23:35:34.659969 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:34.659924 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec0ba84-ad9b-4584-af7c-044629bd6876-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55\" (UID: \"5ec0ba84-ad9b-4584-af7c-044629bd6876\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55" Sep 17 23:35:34.660172 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:34.659986 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24rqc\" (UniqueName: \"kubernetes.io/projected/5ec0ba84-ad9b-4584-af7c-044629bd6876-kube-api-access-24rqc\") pod \"payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55\" (UID: \"5ec0ba84-ad9b-4584-af7c-044629bd6876\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55" Sep 17 23:35:34.660682 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:34.660661 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec0ba84-ad9b-4584-af7c-044629bd6876-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55\" (UID: \"5ec0ba84-ad9b-4584-af7c-044629bd6876\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55" Sep 17 23:35:34.668147 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:34.668120 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24rqc\" (UniqueName: \"kubernetes.io/projected/5ec0ba84-ad9b-4584-af7c-044629bd6876-kube-api-access-24rqc\") pod \"payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55\" (UID: \"5ec0ba84-ad9b-4584-af7c-044629bd6876\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55" Sep 17 23:35:34.782907 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:34.782870 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55" Sep 17 23:35:35.062563 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:35.062418 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55"] Sep 17 23:35:35.065237 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:35:35.065208 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec0ba84_ad9b_4584_af7c_044629bd6876.slice/crio-7ea012b3ed597012aee79e201dd6c9eb03727c156f4591ec729a9fd605f66241 WatchSource:0}: Error finding container 7ea012b3ed597012aee79e201dd6c9eb03727c156f4591ec729a9fd605f66241: Status 404 returned error can't find the container with id 7ea012b3ed597012aee79e201dd6c9eb03727c156f4591ec729a9fd605f66241 Sep 17 23:35:35.095212 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:35.095175 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55" event={"ID":"5ec0ba84-ad9b-4584-af7c-044629bd6876","Type":"ContainerStarted","Data":"7ea012b3ed597012aee79e201dd6c9eb03727c156f4591ec729a9fd605f66241"} Sep 17 23:35:36.103205 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:36.103166 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55" event={"ID":"5ec0ba84-ad9b-4584-af7c-044629bd6876","Type":"ContainerStarted","Data":"43fe85b7434ed01a7359602f961ab0babbf5d6e47971c1bc216c7f5bbc2c4f70"} Sep 17 23:35:36.103610 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:36.103266 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55" Sep 17 23:35:36.119330 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:36.119280 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55" podStartSLOduration=2.119265626 podStartE2EDuration="2.119265626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:35:36.116992162 +0000 UTC m=+1775.050899297" watchObservedRunningTime="2026-09-17 23:35:36.119265626 +0000 UTC m=+1775.053172761" Sep 17 23:35:47.110730 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:47.110643 2582 patch_prober.go:28] interesting pod/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 17 23:35:47.111090 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:47.110723 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" podUID="e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 17 23:35:47.112208 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:47.112183 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55" Sep 17 23:35:48.181257 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:48.181221 2582 generic.go:356] "Generic (PLEG): container finished" podID="e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" containerID="10b0d6097e6c1666e6e0e896dba279cb94d4e1d5f7c6597001d612b9559a25e6" exitCode=0 Sep 17 23:35:48.181257 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:48.181245 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" event={"ID":"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba","Type":"ContainerDied","Data":"10b0d6097e6c1666e6e0e896dba279cb94d4e1d5f7c6597001d612b9559a25e6"} Sep 17 23:35:48.181716 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:48.181674 2582 scope.go:117] "RemoveContainer" containerID="10b0d6097e6c1666e6e0e896dba279cb94d4e1d5f7c6597001d612b9559a25e6" Sep 17 23:35:48.182840 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:48.182811 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:35:48.182917 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:48.182877 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:35:48.182955 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:48.182914 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:35:49.193022 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:49.192979 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" event={"ID":"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba","Type":"ContainerStarted","Data":"811d657f45eed5ee8ce5c5ea75c6a10af1a0f560470e556a2019c6c3bfe97d38"} Sep 17 23:35:50.179750 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:50.179706 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:50.180078 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:50.179847 2582 patch_prober.go:28] interesting pod/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Sep 17 23:35:50.180078 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:50.179893 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" podUID="e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Sep 17 23:35:51.179399 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:51.179358 2582 patch_prober.go:28] interesting pod/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Sep 17 23:35:51.179917 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:51.179433 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" podUID="e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Sep 17 23:35:52.179971 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:52.179933 2582 patch_prober.go:28] interesting pod/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Sep 17 23:35:52.180605 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:52.180559 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" podUID="e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Sep 17 23:35:53.179986 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:53.179947 2582 patch_prober.go:28] interesting pod/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Sep 17 23:35:53.180587 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:53.180551 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" podUID="e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Sep 17 23:35:54.179627 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:54.179582 2582 patch_prober.go:28] interesting pod/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Sep 17 23:35:54.179800 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:54.179655 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" podUID="e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Sep 17 23:35:55.257270 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:55.257237 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:55.257770 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:55.257750 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:55.258445 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:55.258424 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:35:57.968301 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:57.968264 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w"] Sep 17 23:35:57.972481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:57.972455 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:57.975464 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:57.975438 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-accfb5-openshift-default-dockercfg-zhdfr\"" Sep 17 23:35:57.987012 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:57.986987 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w"] Sep 17 23:35:58.086861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.086299 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-token\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.086861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.086355 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-data\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.086861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.086387 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-envoy\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.086861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.086446 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-workload-socket\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.086861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.086481 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-credential-socket\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.086861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.086537 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istiod-ca-cert\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.086861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.086593 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-workload-certs\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.086861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.086635 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-podinfo\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.086861 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.086687 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt9q9\" (UniqueName: \"kubernetes.io/projected/1a34e881-1ea4-4e54-bd79-95ee49f68ece-kube-api-access-qt9q9\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.188055 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.188021 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-data\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.188055 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.188065 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-envoy\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.188284 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.188202 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-workload-socket\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.188284 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.188241 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-credential-socket\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.188284 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.188280 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istiod-ca-cert\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.188453 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.188310 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-workload-certs\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.188453 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.188357 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-podinfo\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.188453 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.188417 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt9q9\" (UniqueName: \"kubernetes.io/projected/1a34e881-1ea4-4e54-bd79-95ee49f68ece-kube-api-access-qt9q9\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.188645 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.188483 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-data\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.188645 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.188544 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-token\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.188645 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.188578 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-workload-socket\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.188788 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.188722 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-credential-socket\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.188788 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.188765 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-workload-certs\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.189198 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.189178 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istiod-ca-cert\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.190425 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.190403 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-envoy\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.190932 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.190904 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-podinfo\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.196102 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.196081 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-token\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.196336 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.196316 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt9q9\" (UniqueName: \"kubernetes.io/projected/1a34e881-1ea4-4e54-bd79-95ee49f68ece-kube-api-access-qt9q9\") pod \"e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.288612 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.288584 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:58.564930 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.564902 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w"] Sep 17 23:35:58.567148 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:35:58.567118 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a34e881_1ea4_4e54_bd79_95ee49f68ece.slice/crio-42e5d7d5f678c33a0e140ed7f58aced63a3add2e1fcf8cad25f0acca99ff42ba WatchSource:0}: Error finding container 42e5d7d5f678c33a0e140ed7f58aced63a3add2e1fcf8cad25f0acca99ff42ba: Status 404 returned error can't find the container with id 42e5d7d5f678c33a0e140ed7f58aced63a3add2e1fcf8cad25f0acca99ff42ba Sep 17 23:35:58.569295 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.569256 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:35:58.569416 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.569327 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:35:58.569416 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:58.569365 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:35:59.263560 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:59.263521 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" event={"ID":"1a34e881-1ea4-4e54-bd79-95ee49f68ece","Type":"ContainerStarted","Data":"a968f22c8c5a5d71b3af7f340db6092b14cf86c2153d65c54a9c79a54202a0b4"} Sep 17 23:35:59.263923 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:59.263575 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" event={"ID":"1a34e881-1ea4-4e54-bd79-95ee49f68ece","Type":"ContainerStarted","Data":"42e5d7d5f678c33a0e140ed7f58aced63a3add2e1fcf8cad25f0acca99ff42ba"} Sep 17 23:35:59.284107 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:59.284055 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" podStartSLOduration=2.284039069 podStartE2EDuration="2.284039069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:35:59.28164752 +0000 UTC m=+1798.215554670" watchObservedRunningTime="2026-09-17 23:35:59.284039069 +0000 UTC m=+1798.217946276" Sep 17 23:35:59.295251 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:59.295224 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:35:59.295398 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:35:59.295315 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:36:00.269966 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:00.269931 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:36:00.271468 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:00.271442 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:36:03.636128 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:03.636098 2582 scope.go:117] "RemoveContainer" containerID="11e09570053bf1a0cad613d5f30c70184afbad137d779976f8660cd673c48772" Sep 17 23:36:03.645723 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:03.645700 2582 scope.go:117] "RemoveContainer" containerID="62e7a433e0097cc8b60911cea4828737a1ec8912532b4438cfd4750b75a3b2fe" Sep 17 23:36:04.204521 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:04.204465 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj"] Sep 17 23:36:04.210139 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:04.210117 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj" Sep 17 23:36:04.212470 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:04.212450 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-accfb5\"" Sep 17 23:36:04.212580 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:04.212561 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-accfb5-dockercfg-8gvrt\"" Sep 17 23:36:04.218326 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:04.218305 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj"] Sep 17 23:36:04.348471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:04.348439 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86sz5\" (UniqueName: \"kubernetes.io/projected/e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2-kube-api-access-86sz5\") pod \"payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj\" (UID: \"e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj" Sep 17 23:36:04.348650 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:04.348553 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj\" (UID: \"e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj" Sep 17 23:36:04.449577 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:04.449539 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj\" (UID: \"e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj" Sep 17 23:36:04.449730 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:04.449668 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86sz5\" (UniqueName: \"kubernetes.io/projected/e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2-kube-api-access-86sz5\") pod \"payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj\" (UID: \"e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj" Sep 17 23:36:04.450226 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:04.450206 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj\" (UID: \"e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj" Sep 17 23:36:04.457155 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:04.457087 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86sz5\" (UniqueName: \"kubernetes.io/projected/e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2-kube-api-access-86sz5\") pod \"payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj\" (UID: \"e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj" Sep 17 23:36:04.521957 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:04.521923 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj" Sep 17 23:36:04.801155 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:04.801124 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj"] Sep 17 23:36:04.803172 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:36:04.803145 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d8133f_5fe9_4ac1_91bc_cf4b17f2b9f2.slice/crio-b32ab24c6d80083e68865e94151147bddfebe84ee57f36227f1a6ce21772fbe3 WatchSource:0}: Error finding container b32ab24c6d80083e68865e94151147bddfebe84ee57f36227f1a6ce21772fbe3: Status 404 returned error can't find the container with id b32ab24c6d80083e68865e94151147bddfebe84ee57f36227f1a6ce21772fbe3 Sep 17 23:36:05.303300 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:05.303266 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj" event={"ID":"e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2","Type":"ContainerStarted","Data":"818b4ccd7f9af41c0528a07e56d576ed80be7f2507e9e931b8922c96d31a9ad1"} Sep 17 23:36:05.303481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:05.303323 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj" event={"ID":"e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2","Type":"ContainerStarted","Data":"b32ab24c6d80083e68865e94151147bddfebe84ee57f36227f1a6ce21772fbe3"} Sep 17 23:36:05.303481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:05.303421 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj" Sep 17 23:36:05.319575 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:05.319526 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj" podStartSLOduration=1.319511012 podStartE2EDuration="1.319511012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:36:05.317916612 +0000 UTC m=+1804.251823748" watchObservedRunningTime="2026-09-17 23:36:05.319511012 +0000 UTC m=+1804.253418140" Sep 17 23:36:16.312024 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:16.311989 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj" Sep 17 23:36:20.391349 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:20.391312 2582 generic.go:356] "Generic (PLEG): container finished" podID="1a34e881-1ea4-4e54-bd79-95ee49f68ece" containerID="a968f22c8c5a5d71b3af7f340db6092b14cf86c2153d65c54a9c79a54202a0b4" exitCode=0 Sep 17 23:36:20.391796 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:20.391385 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" event={"ID":"1a34e881-1ea4-4e54-bd79-95ee49f68ece","Type":"ContainerDied","Data":"a968f22c8c5a5d71b3af7f340db6092b14cf86c2153d65c54a9c79a54202a0b4"} Sep 17 23:36:20.391796 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:20.391790 2582 scope.go:117] "RemoveContainer" containerID="a968f22c8c5a5d71b3af7f340db6092b14cf86c2153d65c54a9c79a54202a0b4" Sep 17 23:36:20.392940 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:20.392912 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:36:20.393001 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:20.392980 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:36:20.393042 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:20.393015 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:36:21.400526 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:21.400477 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" event={"ID":"1a34e881-1ea4-4e54-bd79-95ee49f68ece","Type":"ContainerStarted","Data":"7e9de78b66e600725f4ff0f56624dd97581db6b49925cd75494b8a1012e412f5"} Sep 17 23:36:22.289299 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:22.289258 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:36:22.289965 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:22.289635 2582 patch_prober.go:28] interesting pod/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Sep 17 23:36:22.289965 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:22.289690 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" podUID="1a34e881-1ea4-4e54-bd79-95ee49f68ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Sep 17 23:36:23.296665 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:23.296624 2582 patch_prober.go:28] interesting pod/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Sep 17 23:36:23.297106 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:23.296698 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" podUID="1a34e881-1ea4-4e54-bd79-95ee49f68ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Sep 17 23:36:24.289644 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:24.289610 2582 patch_prober.go:28] interesting pod/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Sep 17 23:36:24.289930 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:24.289902 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" podUID="1a34e881-1ea4-4e54-bd79-95ee49f68ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Sep 17 23:36:25.294275 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:25.294237 2582 patch_prober.go:28] interesting pod/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Sep 17 23:36:25.296735 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:25.296698 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" podUID="1a34e881-1ea4-4e54-bd79-95ee49f68ece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Sep 17 23:36:26.479197 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:26.479161 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:36:26.479903 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:26.479874 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:36:26.480778 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:26.480757 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:36:31.371899 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:36:31.371869 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-pxpjd_b6287a30-1142-48b6-a7ea-50b426e7b598/payload-processing/0.log" Sep 17 23:37:02.584755 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:02.584677 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-pxpjd_b6287a30-1142-48b6-a7ea-50b426e7b598/payload-processing/0.log" Sep 17 23:37:03.284886 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.284850 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr"] Sep 17 23:37:03.290334 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.290055 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.292654 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.292632 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-e485c4-openshift-default-dockercfg-xxwh6\"" Sep 17 23:37:03.301441 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.301418 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr"] Sep 17 23:37:03.402807 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.402771 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-token\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.403070 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.403046 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-podinfo\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.403200 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.403179 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-envoy\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.403404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.403384 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00baf2f-3963-4c40-b684-6fb1e2565771-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.403527 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.403439 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqxrc\" (UniqueName: \"kubernetes.io/projected/d00baf2f-3963-4c40-b684-6fb1e2565771-kube-api-access-gqxrc\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.403604 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.403545 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-workload-socket\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.403604 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.403578 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-data\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.403710 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.403620 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-credential-socket\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.403710 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.403651 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-workload-certs\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.504909 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.504876 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqxrc\" (UniqueName: \"kubernetes.io/projected/d00baf2f-3963-4c40-b684-6fb1e2565771-kube-api-access-gqxrc\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.505116 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.504954 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-workload-socket\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.505116 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.504986 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-data\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.505116 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.505026 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-credential-socket\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.505116 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.505053 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-workload-certs\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.505116 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.505116 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-token\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.505401 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.505148 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-podinfo\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.505401 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.505195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-envoy\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.505401 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.505242 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00baf2f-3963-4c40-b684-6fb1e2565771-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.505401 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.505388 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-workload-socket\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.505583 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.505429 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-data\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.505583 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.505478 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-credential-socket\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.505774 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.505754 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-workload-certs\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.505998 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.505971 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00baf2f-3963-4c40-b684-6fb1e2565771-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.507580 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.507563 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-envoy\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.507683 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.507668 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-podinfo\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.512198 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.512169 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-token\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.512306 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.512272 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqxrc\" (UniqueName: \"kubernetes.io/projected/d00baf2f-3963-4c40-b684-6fb1e2565771-kube-api-access-gqxrc\") pod \"e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.604426 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.604346 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:03.888575 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.888547 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr"] Sep 17 23:37:03.890194 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:37:03.890161 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00baf2f_3963_4c40_b684_6fb1e2565771.slice/crio-72f770e8a7de5206176dae6283f08a44078bf71674bf011c4f4f61b780e915e6 WatchSource:0}: Error finding container 72f770e8a7de5206176dae6283f08a44078bf71674bf011c4f4f61b780e915e6: Status 404 returned error can't find the container with id 72f770e8a7de5206176dae6283f08a44078bf71674bf011c4f4f61b780e915e6 Sep 17 23:37:03.892521 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.892469 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:37:03.892595 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.892572 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:37:03.892636 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:03.892614 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:37:04.663382 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:04.663344 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" event={"ID":"d00baf2f-3963-4c40-b684-6fb1e2565771","Type":"ContainerStarted","Data":"3f21c6ce74318605e8741c0eac21ed59b73295accef1209230cdc63429dd2368"} Sep 17 23:37:04.663800 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:04.663413 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" event={"ID":"d00baf2f-3963-4c40-b684-6fb1e2565771","Type":"ContainerStarted","Data":"72f770e8a7de5206176dae6283f08a44078bf71674bf011c4f4f61b780e915e6"} Sep 17 23:37:04.681256 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:04.681204 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" podStartSLOduration=1.681186342 podStartE2EDuration="1.681186342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:37:04.680870343 +0000 UTC m=+1863.614777489" watchObservedRunningTime="2026-09-17 23:37:04.681186342 +0000 UTC m=+1863.615093479" Sep 17 23:37:05.604677 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:05.604633 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:05.609905 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:05.609875 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:05.670008 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:05.669973 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:05.671311 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:05.671283 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:26.799968 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:26.799904 2582 generic.go:356] "Generic (PLEG): container finished" podID="d00baf2f-3963-4c40-b684-6fb1e2565771" containerID="3f21c6ce74318605e8741c0eac21ed59b73295accef1209230cdc63429dd2368" exitCode=0 Sep 17 23:37:26.799968 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:26.799965 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" event={"ID":"d00baf2f-3963-4c40-b684-6fb1e2565771","Type":"ContainerDied","Data":"3f21c6ce74318605e8741c0eac21ed59b73295accef1209230cdc63429dd2368"} Sep 17 23:37:26.800377 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:26.800353 2582 scope.go:117] "RemoveContainer" containerID="3f21c6ce74318605e8741c0eac21ed59b73295accef1209230cdc63429dd2368" Sep 17 23:37:26.801547 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:26.801522 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:37:26.801623 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:26.801586 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:37:26.801623 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:26.801619 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 23:37:27.808176 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:27.808143 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" event={"ID":"d00baf2f-3963-4c40-b684-6fb1e2565771","Type":"ContainerStarted","Data":"1ba426535d09ea7888293fcb5dc499403dc1b15fc13b9f88cb9d89b7c437e0ab"} Sep 17 23:37:28.605442 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:28.605399 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:28.605671 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:28.605651 2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Sep 17 23:37:28.605779 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:28.605704 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" podUID="d00baf2f-3963-4c40-b684-6fb1e2565771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Sep 17 23:37:29.605550 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:29.605488 2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Sep 17 23:37:29.606022 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:29.605578 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" podUID="d00baf2f-3963-4c40-b684-6fb1e2565771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Sep 17 23:37:30.605341 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:30.605301 2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Sep 17 23:37:30.605546 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:30.605361 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" podUID="d00baf2f-3963-4c40-b684-6fb1e2565771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Sep 17 23:37:31.605476 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:31.605438 2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Sep 17 23:37:31.605893 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:31.605523 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" podUID="d00baf2f-3963-4c40-b684-6fb1e2565771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Sep 17 23:37:32.766430 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:32.766397 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:32.767143 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:32.766849 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:32.767654 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:32.767627 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:37:37.401378 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:37.401338 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e485c4-288dz"] Sep 17 23:37:37.404233 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:37.404215 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e485c4-288dz" Sep 17 23:37:37.406444 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:37.406423 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:37:37.411535 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:37.411491 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e485c4-288dz"] Sep 17 23:37:37.548555 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:37.548524 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af03d2ff-4347-4b52-8e99-dfe596d82ed7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e485c4-288dz\" (UID: \"af03d2ff-4347-4b52-8e99-dfe596d82ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e485c4-288dz" Sep 17 23:37:37.649786 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:37.649750 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af03d2ff-4347-4b52-8e99-dfe596d82ed7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e485c4-288dz\" (UID: \"af03d2ff-4347-4b52-8e99-dfe596d82ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e485c4-288dz" Sep 17 23:37:37.650319 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:37.650293 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af03d2ff-4347-4b52-8e99-dfe596d82ed7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e485c4-288dz\" (UID: \"af03d2ff-4347-4b52-8e99-dfe596d82ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e485c4-288dz" Sep 17 23:37:37.716350 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:37.716265 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e485c4-288dz" Sep 17 23:37:37.988371 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:37.988344 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e485c4-288dz"] Sep 17 23:37:37.990133 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:37:37.990105 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf03d2ff_4347_4b52_8e99_dfe596d82ed7.slice/crio-2dbcd2f0d181178ce51b9b335dbec006528faf2a9bad67467940189fd5b0b3fb WatchSource:0}: Error finding container 2dbcd2f0d181178ce51b9b335dbec006528faf2a9bad67467940189fd5b0b3fb: Status 404 returned error can't find the container with id 2dbcd2f0d181178ce51b9b335dbec006528faf2a9bad67467940189fd5b0b3fb Sep 17 23:37:38.876963 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:38.876870 2582 generic.go:356] "Generic (PLEG): container finished" podID="af03d2ff-4347-4b52-8e99-dfe596d82ed7" containerID="0e5cdab91c482192db0896c05c3f5b808689e141a640283245ef672f610198a2" exitCode=0 Sep 17 23:37:38.877343 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:38.876957 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e485c4-288dz" event={"ID":"af03d2ff-4347-4b52-8e99-dfe596d82ed7","Type":"ContainerDied","Data":"0e5cdab91c482192db0896c05c3f5b808689e141a640283245ef672f610198a2"} Sep 17 23:37:38.877343 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:38.877013 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e485c4-288dz" event={"ID":"af03d2ff-4347-4b52-8e99-dfe596d82ed7","Type":"ContainerStarted","Data":"2dbcd2f0d181178ce51b9b335dbec006528faf2a9bad67467940189fd5b0b3fb"} Sep 17 23:37:40.019790 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:40.019765 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e485c4-288dz" Sep 17 23:37:40.173058 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:40.172967 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af03d2ff-4347-4b52-8e99-dfe596d82ed7-maas-api-service-ca\") pod \"af03d2ff-4347-4b52-8e99-dfe596d82ed7\" (UID: \"af03d2ff-4347-4b52-8e99-dfe596d82ed7\") " Sep 17 23:37:40.173347 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:40.173324 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af03d2ff-4347-4b52-8e99-dfe596d82ed7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af03d2ff-4347-4b52-8e99-dfe596d82ed7" (UID: "af03d2ff-4347-4b52-8e99-dfe596d82ed7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:37:40.274279 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:40.274245 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af03d2ff-4347-4b52-8e99-dfe596d82ed7-maas-api-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:37:40.890381 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:40.890350 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e485c4-288dz" Sep 17 23:37:40.890381 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:40.890369 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e485c4-288dz" event={"ID":"af03d2ff-4347-4b52-8e99-dfe596d82ed7","Type":"ContainerDied","Data":"2dbcd2f0d181178ce51b9b335dbec006528faf2a9bad67467940189fd5b0b3fb"} Sep 17 23:37:40.890684 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:40.890413 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dbcd2f0d181178ce51b9b335dbec006528faf2a9bad67467940189fd5b0b3fb" Sep 17 23:37:48.681998 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:48.681958 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e485c4-288dz"] Sep 17 23:37:48.685320 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:48.685293 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e485c4-288dz"] Sep 17 23:37:49.295261 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:49.295229 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af03d2ff-4347-4b52-8e99-dfe596d82ed7" path="/var/lib/kubelet/pods/af03d2ff-4347-4b52-8e99-dfe596d82ed7/volumes" Sep 17 23:37:55.099559 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:55.099524 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr"] Sep 17 23:37:55.100039 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:55.099734 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" podUID="d00baf2f-3963-4c40-b684-6fb1e2565771" containerName="istio-proxy" containerID="cri-o://1ba426535d09ea7888293fcb5dc499403dc1b15fc13b9f88cb9d89b7c437e0ab" gracePeriod=30 Sep 17 23:37:58.807959 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:58.807924 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr"] Sep 17 23:37:58.808397 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:58.808367 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af03d2ff-4347-4b52-8e99-dfe596d82ed7" containerName="revoke-keys" Sep 17 23:37:58.808397 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:58.808379 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="af03d2ff-4347-4b52-8e99-dfe596d82ed7" containerName="revoke-keys" Sep 17 23:37:58.808474 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:58.808456 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="af03d2ff-4347-4b52-8e99-dfe596d82ed7" containerName="revoke-keys" Sep 17 23:37:58.811629 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:58.811613 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr" Sep 17 23:37:58.813966 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:58.813947 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:37:58.820724 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:58.819543 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr"] Sep 17 23:37:58.861103 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:58.861065 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffcfc6a7-98d0-4da0-bf03-adae73332da8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr\" (UID: \"ffcfc6a7-98d0-4da0-bf03-adae73332da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr" Sep 17 23:37:58.962541 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:58.962477 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffcfc6a7-98d0-4da0-bf03-adae73332da8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr\" (UID: \"ffcfc6a7-98d0-4da0-bf03-adae73332da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr" Sep 17 23:37:58.963159 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:58.963130 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffcfc6a7-98d0-4da0-bf03-adae73332da8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr\" (UID: \"ffcfc6a7-98d0-4da0-bf03-adae73332da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr" Sep 17 23:37:59.124642 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:59.124559 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr" Sep 17 23:37:59.410911 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:37:59.410871 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr"] Sep 17 23:37:59.413308 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:37:59.413277 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcfc6a7_98d0_4da0_bf03_adae73332da8.slice/crio-1d797559c350fa833ee7bcbb348f0efbb0ea32b9939bdb6b919639b8fbec3b6f WatchSource:0}: Error finding container 1d797559c350fa833ee7bcbb348f0efbb0ea32b9939bdb6b919639b8fbec3b6f: Status 404 returned error can't find the container with id 1d797559c350fa833ee7bcbb348f0efbb0ea32b9939bdb6b919639b8fbec3b6f Sep 17 23:38:00.002637 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.002603 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr" event={"ID":"ffcfc6a7-98d0-4da0-bf03-adae73332da8","Type":"ContainerStarted","Data":"a7067b19e39734d0c677f605c808bb1703a1865ecb7c7493122b5877b615b3c2"} Sep 17 23:38:00.003048 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.002658 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr" event={"ID":"ffcfc6a7-98d0-4da0-bf03-adae73332da8","Type":"ContainerStarted","Data":"1d797559c350fa833ee7bcbb348f0efbb0ea32b9939bdb6b919639b8fbec3b6f"} Sep 17 23:38:00.020310 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.020252 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr" podStartSLOduration=2.020234096 podStartE2EDuration="2.020234096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:38:00.018960659 +0000 UTC m=+1918.952867795" watchObservedRunningTime="2026-09-17 23:38:00.020234096 +0000 UTC m=+1918.954141231" Sep 17 23:38:00.355936 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.355909 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:38:00.377207 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.377166 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-workload-socket\") pod \"d00baf2f-3963-4c40-b684-6fb1e2565771\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " Sep 17 23:38:00.377411 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.377218 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-credential-socket\") pod \"d00baf2f-3963-4c40-b684-6fb1e2565771\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " Sep 17 23:38:00.377411 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.377288 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqxrc\" (UniqueName: \"kubernetes.io/projected/d00baf2f-3963-4c40-b684-6fb1e2565771-kube-api-access-gqxrc\") pod \"d00baf2f-3963-4c40-b684-6fb1e2565771\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " Sep 17 23:38:00.377411 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.377318 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00baf2f-3963-4c40-b684-6fb1e2565771-istiod-ca-cert\") pod \"d00baf2f-3963-4c40-b684-6fb1e2565771\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " Sep 17 23:38:00.377411 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.377348 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-workload-certs\") pod \"d00baf2f-3963-4c40-b684-6fb1e2565771\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " Sep 17 23:38:00.377690 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.377439 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-envoy\") pod \"d00baf2f-3963-4c40-b684-6fb1e2565771\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " Sep 17 23:38:00.377690 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.377539 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-token\") pod \"d00baf2f-3963-4c40-b684-6fb1e2565771\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " Sep 17 23:38:00.377690 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.377577 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-data\") pod \"d00baf2f-3963-4c40-b684-6fb1e2565771\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " Sep 17 23:38:00.377690 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.377611 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-podinfo\") pod \"d00baf2f-3963-4c40-b684-6fb1e2565771\" (UID: \"d00baf2f-3963-4c40-b684-6fb1e2565771\") " Sep 17 23:38:00.378382 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.378013 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d00baf2f-3963-4c40-b684-6fb1e2565771" (UID: "d00baf2f-3963-4c40-b684-6fb1e2565771"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:38:00.378523 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.378384 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00baf2f-3963-4c40-b684-6fb1e2565771-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d00baf2f-3963-4c40-b684-6fb1e2565771" (UID: "d00baf2f-3963-4c40-b684-6fb1e2565771"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:38:00.378523 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.378453 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d00baf2f-3963-4c40-b684-6fb1e2565771" (UID: "d00baf2f-3963-4c40-b684-6fb1e2565771"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:38:00.378943 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.378815 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-data" (OuterVolumeSpecName: "istio-data") pod "d00baf2f-3963-4c40-b684-6fb1e2565771" (UID: "d00baf2f-3963-4c40-b684-6fb1e2565771"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:38:00.379326 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.379279 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d00baf2f-3963-4c40-b684-6fb1e2565771" (UID: "d00baf2f-3963-4c40-b684-6fb1e2565771"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:38:00.381185 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.381137 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d00baf2f-3963-4c40-b684-6fb1e2565771" (UID: "d00baf2f-3963-4c40-b684-6fb1e2565771"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:38:00.381288 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.381245 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00baf2f-3963-4c40-b684-6fb1e2565771-kube-api-access-gqxrc" (OuterVolumeSpecName: "kube-api-access-gqxrc") pod "d00baf2f-3963-4c40-b684-6fb1e2565771" (UID: "d00baf2f-3963-4c40-b684-6fb1e2565771"). InnerVolumeSpecName "kube-api-access-gqxrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:38:00.381351 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.381334 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-token" (OuterVolumeSpecName: "istio-token") pod "d00baf2f-3963-4c40-b684-6fb1e2565771" (UID: "d00baf2f-3963-4c40-b684-6fb1e2565771"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:38:00.381957 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.381930 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d00baf2f-3963-4c40-b684-6fb1e2565771" (UID: "d00baf2f-3963-4c40-b684-6fb1e2565771"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:38:00.479533 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.479480 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-envoy\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:38:00.479533 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.479526 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:38:00.479533 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.479536 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-data\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:38:00.479533 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.479543 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00baf2f-3963-4c40-b684-6fb1e2565771-istio-podinfo\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:38:00.479800 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.479552 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-workload-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:38:00.479800 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.479560 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-credential-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:38:00.479800 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.479568 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqxrc\" (UniqueName: \"kubernetes.io/projected/d00baf2f-3963-4c40-b684-6fb1e2565771-kube-api-access-gqxrc\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:38:00.479800 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.479576 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00baf2f-3963-4c40-b684-6fb1e2565771-istiod-ca-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:38:00.479800 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:00.479584 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00baf2f-3963-4c40-b684-6fb1e2565771-workload-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:38:01.010046 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:01.010011 2582 generic.go:356] "Generic (PLEG): container finished" podID="ffcfc6a7-98d0-4da0-bf03-adae73332da8" containerID="a7067b19e39734d0c677f605c808bb1703a1865ecb7c7493122b5877b615b3c2" exitCode=0 Sep 17 23:38:01.010489 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:01.010099 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr" event={"ID":"ffcfc6a7-98d0-4da0-bf03-adae73332da8","Type":"ContainerDied","Data":"a7067b19e39734d0c677f605c808bb1703a1865ecb7c7493122b5877b615b3c2"} Sep 17 23:38:01.011855 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:01.011834 2582 generic.go:356] "Generic (PLEG): container finished" podID="d00baf2f-3963-4c40-b684-6fb1e2565771" containerID="1ba426535d09ea7888293fcb5dc499403dc1b15fc13b9f88cb9d89b7c437e0ab" exitCode=0 Sep 17 23:38:01.011976 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:01.011874 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" Sep 17 23:38:01.011976 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:01.011915 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" event={"ID":"d00baf2f-3963-4c40-b684-6fb1e2565771","Type":"ContainerDied","Data":"1ba426535d09ea7888293fcb5dc499403dc1b15fc13b9f88cb9d89b7c437e0ab"} Sep 17 23:38:01.011976 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:01.011958 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr" event={"ID":"d00baf2f-3963-4c40-b684-6fb1e2565771","Type":"ContainerDied","Data":"72f770e8a7de5206176dae6283f08a44078bf71674bf011c4f4f61b780e915e6"} Sep 17 23:38:01.012091 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:01.011978 2582 scope.go:117] "RemoveContainer" containerID="1ba426535d09ea7888293fcb5dc499403dc1b15fc13b9f88cb9d89b7c437e0ab" Sep 17 23:38:01.021721 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:01.021705 2582 scope.go:117] "RemoveContainer" containerID="3f21c6ce74318605e8741c0eac21ed59b73295accef1209230cdc63429dd2368" Sep 17 23:38:01.031168 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:01.031144 2582 scope.go:117] "RemoveContainer" containerID="1ba426535d09ea7888293fcb5dc499403dc1b15fc13b9f88cb9d89b7c437e0ab" Sep 17 23:38:01.031523 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:38:01.031475 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba426535d09ea7888293fcb5dc499403dc1b15fc13b9f88cb9d89b7c437e0ab\": container with ID starting with 1ba426535d09ea7888293fcb5dc499403dc1b15fc13b9f88cb9d89b7c437e0ab not found: ID does not exist" containerID="1ba426535d09ea7888293fcb5dc499403dc1b15fc13b9f88cb9d89b7c437e0ab" Sep 17 23:38:01.031632 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:01.031532 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba426535d09ea7888293fcb5dc499403dc1b15fc13b9f88cb9d89b7c437e0ab"} err="failed to get container status \"1ba426535d09ea7888293fcb5dc499403dc1b15fc13b9f88cb9d89b7c437e0ab\": rpc error: code = NotFound desc = could not find container \"1ba426535d09ea7888293fcb5dc499403dc1b15fc13b9f88cb9d89b7c437e0ab\": container with ID starting with 1ba426535d09ea7888293fcb5dc499403dc1b15fc13b9f88cb9d89b7c437e0ab not found: ID does not exist" Sep 17 23:38:01.031632 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:01.031553 2582 scope.go:117] "RemoveContainer" containerID="3f21c6ce74318605e8741c0eac21ed59b73295accef1209230cdc63429dd2368" Sep 17 23:38:01.031845 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:38:01.031819 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f21c6ce74318605e8741c0eac21ed59b73295accef1209230cdc63429dd2368\": container with ID starting with 3f21c6ce74318605e8741c0eac21ed59b73295accef1209230cdc63429dd2368 not found: ID does not exist" containerID="3f21c6ce74318605e8741c0eac21ed59b73295accef1209230cdc63429dd2368" Sep 17 23:38:01.031914 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:01.031852 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f21c6ce74318605e8741c0eac21ed59b73295accef1209230cdc63429dd2368"} err="failed to get container status \"3f21c6ce74318605e8741c0eac21ed59b73295accef1209230cdc63429dd2368\": rpc error: code = NotFound desc = could not find container \"3f21c6ce74318605e8741c0eac21ed59b73295accef1209230cdc63429dd2368\": container with ID starting with 3f21c6ce74318605e8741c0eac21ed59b73295accef1209230cdc63429dd2368 not found: ID does not exist" Sep 17 23:38:01.040531 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:01.040485 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr"] Sep 17 23:38:01.045193 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:01.045170 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e485c4-openshift-default-8d8b4ccc4-9dvqr"] Sep 17 23:38:01.294779 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:01.294702 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d00baf2f-3963-4c40-b684-6fb1e2565771" path="/var/lib/kubelet/pods/d00baf2f-3963-4c40-b684-6fb1e2565771/volumes" Sep 17 23:38:02.162873 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:02.162849 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr" Sep 17 23:38:02.196903 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:02.196868 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffcfc6a7-98d0-4da0-bf03-adae73332da8-maas-api-service-ca\") pod \"ffcfc6a7-98d0-4da0-bf03-adae73332da8\" (UID: \"ffcfc6a7-98d0-4da0-bf03-adae73332da8\") " Sep 17 23:38:02.197166 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:02.197139 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffcfc6a7-98d0-4da0-bf03-adae73332da8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ffcfc6a7-98d0-4da0-bf03-adae73332da8" (UID: "ffcfc6a7-98d0-4da0-bf03-adae73332da8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:38:02.197316 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:02.197302 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffcfc6a7-98d0-4da0-bf03-adae73332da8-maas-api-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:38:03.025194 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:03.025166 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr" Sep 17 23:38:03.025398 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:03.025192 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr" event={"ID":"ffcfc6a7-98d0-4da0-bf03-adae73332da8","Type":"ContainerDied","Data":"1d797559c350fa833ee7bcbb348f0efbb0ea32b9939bdb6b919639b8fbec3b6f"} Sep 17 23:38:03.025398 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:03.025236 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d797559c350fa833ee7bcbb348f0efbb0ea32b9939bdb6b919639b8fbec3b6f" Sep 17 23:38:14.740891 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:14.740859 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55"] Sep 17 23:38:14.741358 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:14.741080 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55" podUID="5ec0ba84-ad9b-4584-af7c-044629bd6876" containerName="payload-pre-processing" containerID="cri-o://43fe85b7434ed01a7359602f961ab0babbf5d6e47971c1bc216c7f5bbc2c4f70" gracePeriod=30 Sep 17 23:38:14.904554 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:14.904481 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr"] Sep 17 23:38:14.907602 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:14.907572 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8f81ee-phldr"] Sep 17 23:38:15.010111 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:15.010083 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55" Sep 17 23:38:15.099787 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:15.099753 2582 generic.go:356] "Generic (PLEG): container finished" podID="5ec0ba84-ad9b-4584-af7c-044629bd6876" containerID="43fe85b7434ed01a7359602f961ab0babbf5d6e47971c1bc216c7f5bbc2c4f70" exitCode=0 Sep 17 23:38:15.099975 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:15.099795 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55" event={"ID":"5ec0ba84-ad9b-4584-af7c-044629bd6876","Type":"ContainerDied","Data":"43fe85b7434ed01a7359602f961ab0babbf5d6e47971c1bc216c7f5bbc2c4f70"} Sep 17 23:38:15.099975 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:15.099807 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55" Sep 17 23:38:15.099975 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:15.099838 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55" event={"ID":"5ec0ba84-ad9b-4584-af7c-044629bd6876","Type":"ContainerDied","Data":"7ea012b3ed597012aee79e201dd6c9eb03727c156f4591ec729a9fd605f66241"} Sep 17 23:38:15.099975 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:15.099867 2582 scope.go:117] "RemoveContainer" containerID="43fe85b7434ed01a7359602f961ab0babbf5d6e47971c1bc216c7f5bbc2c4f70" Sep 17 23:38:15.112678 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:15.112640 2582 scope.go:117] "RemoveContainer" containerID="43fe85b7434ed01a7359602f961ab0babbf5d6e47971c1bc216c7f5bbc2c4f70" Sep 17 23:38:15.112971 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:38:15.112946 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43fe85b7434ed01a7359602f961ab0babbf5d6e47971c1bc216c7f5bbc2c4f70\": container with ID starting with 43fe85b7434ed01a7359602f961ab0babbf5d6e47971c1bc216c7f5bbc2c4f70 not found: ID does not exist" containerID="43fe85b7434ed01a7359602f961ab0babbf5d6e47971c1bc216c7f5bbc2c4f70" Sep 17 23:38:15.113087 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:15.112976 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43fe85b7434ed01a7359602f961ab0babbf5d6e47971c1bc216c7f5bbc2c4f70"} err="failed to get container status \"43fe85b7434ed01a7359602f961ab0babbf5d6e47971c1bc216c7f5bbc2c4f70\": rpc error: code = NotFound desc = could not find container \"43fe85b7434ed01a7359602f961ab0babbf5d6e47971c1bc216c7f5bbc2c4f70\": container with ID starting with 43fe85b7434ed01a7359602f961ab0babbf5d6e47971c1bc216c7f5bbc2c4f70 not found: ID does not exist" Sep 17 23:38:15.125480 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:15.125450 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec0ba84-ad9b-4584-af7c-044629bd6876-plugins-config-volume\") pod \"5ec0ba84-ad9b-4584-af7c-044629bd6876\" (UID: \"5ec0ba84-ad9b-4584-af7c-044629bd6876\") " Sep 17 23:38:15.125635 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:15.125579 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24rqc\" (UniqueName: \"kubernetes.io/projected/5ec0ba84-ad9b-4584-af7c-044629bd6876-kube-api-access-24rqc\") pod \"5ec0ba84-ad9b-4584-af7c-044629bd6876\" (UID: \"5ec0ba84-ad9b-4584-af7c-044629bd6876\") " Sep 17 23:38:15.125840 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:15.125817 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec0ba84-ad9b-4584-af7c-044629bd6876-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ec0ba84-ad9b-4584-af7c-044629bd6876" (UID: "5ec0ba84-ad9b-4584-af7c-044629bd6876"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:38:15.125931 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:15.125913 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ec0ba84-ad9b-4584-af7c-044629bd6876-plugins-config-volume\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:38:15.127732 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:15.127709 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec0ba84-ad9b-4584-af7c-044629bd6876-kube-api-access-24rqc" (OuterVolumeSpecName: "kube-api-access-24rqc") pod "5ec0ba84-ad9b-4584-af7c-044629bd6876" (UID: "5ec0ba84-ad9b-4584-af7c-044629bd6876"). InnerVolumeSpecName "kube-api-access-24rqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:38:15.227276 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:15.227237 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24rqc\" (UniqueName: \"kubernetes.io/projected/5ec0ba84-ad9b-4584-af7c-044629bd6876-kube-api-access-24rqc\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:38:15.294533 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:15.294446 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffcfc6a7-98d0-4da0-bf03-adae73332da8" path="/var/lib/kubelet/pods/ffcfc6a7-98d0-4da0-bf03-adae73332da8/volumes" Sep 17 23:38:15.416511 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:15.416476 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55"] Sep 17 23:38:15.421594 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:15.421564 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f81ee-5758f7c88b-62k55"] Sep 17 23:38:17.293754 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:38:17.293716 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ec0ba84-ad9b-4584-af7c-044629bd6876" path="/var/lib/kubelet/pods/5ec0ba84-ad9b-4584-af7c-044629bd6876/volumes" Sep 17 23:39:03.063740 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.063677 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm"] Sep 17 23:39:03.064312 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.064026 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" podUID="e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" containerName="istio-proxy" containerID="cri-o://811d657f45eed5ee8ce5c5ea75c6a10af1a0f560470e556a2019c6c3bfe97d38" gracePeriod=30 Sep 17 23:39:03.433393 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.433320 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6"] Sep 17 23:39:03.433769 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.433756 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d00baf2f-3963-4c40-b684-6fb1e2565771" containerName="istio-proxy" Sep 17 23:39:03.433817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.433770 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d00baf2f-3963-4c40-b684-6fb1e2565771" containerName="istio-proxy" Sep 17 23:39:03.433817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.433777 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffcfc6a7-98d0-4da0-bf03-adae73332da8" containerName="revoke-keys" Sep 17 23:39:03.433817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.433782 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffcfc6a7-98d0-4da0-bf03-adae73332da8" containerName="revoke-keys" Sep 17 23:39:03.433817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.433792 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ec0ba84-ad9b-4584-af7c-044629bd6876" containerName="payload-pre-processing" Sep 17 23:39:03.433817 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.433798 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ec0ba84-ad9b-4584-af7c-044629bd6876" containerName="payload-pre-processing" Sep 17 23:39:03.433972 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.433868 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffcfc6a7-98d0-4da0-bf03-adae73332da8" containerName="revoke-keys" Sep 17 23:39:03.433972 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.433877 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ec0ba84-ad9b-4584-af7c-044629bd6876" containerName="payload-pre-processing" Sep 17 23:39:03.433972 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.433883 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d00baf2f-3963-4c40-b684-6fb1e2565771" containerName="istio-proxy" Sep 17 23:39:03.437155 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.437138 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6" Sep 17 23:39:03.439637 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.439606 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:39:03.443473 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.443449 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6"] Sep 17 23:39:03.563251 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.563215 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0deb2f22-c55a-41e9-82a0-dd211c97888a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6\" (UID: \"0deb2f22-c55a-41e9-82a0-dd211c97888a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6" Sep 17 23:39:03.664538 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.664490 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0deb2f22-c55a-41e9-82a0-dd211c97888a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6\" (UID: \"0deb2f22-c55a-41e9-82a0-dd211c97888a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6" Sep 17 23:39:03.665097 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.665080 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0deb2f22-c55a-41e9-82a0-dd211c97888a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6\" (UID: \"0deb2f22-c55a-41e9-82a0-dd211c97888a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6" Sep 17 23:39:03.708948 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.708879 2582 scope.go:117] "RemoveContainer" containerID="4cdecc6def40059031b39c4e7d7092bd7bc7cfc6ac9f775c80a8dec39675a2f0" Sep 17 23:39:03.750334 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:03.750310 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6" Sep 17 23:39:04.033222 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:04.033197 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6"] Sep 17 23:39:04.035150 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:39:04.035122 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0deb2f22_c55a_41e9_82a0_dd211c97888a.slice/crio-bc19bc661a097774ae277eab74fa6ae98385e2b983b891513ce9a259a80f7d10 WatchSource:0}: Error finding container bc19bc661a097774ae277eab74fa6ae98385e2b983b891513ce9a259a80f7d10: Status 404 returned error can't find the container with id bc19bc661a097774ae277eab74fa6ae98385e2b983b891513ce9a259a80f7d10 Sep 17 23:39:04.388093 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:04.388006 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6" event={"ID":"0deb2f22-c55a-41e9-82a0-dd211c97888a","Type":"ContainerStarted","Data":"af62723302db9b0a2cb9cd59cbcd9e4e0ad4ccffe91560595fc87fa439339b29"} Sep 17 23:39:04.388093 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:04.388084 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6" event={"ID":"0deb2f22-c55a-41e9-82a0-dd211c97888a","Type":"ContainerStarted","Data":"bc19bc661a097774ae277eab74fa6ae98385e2b983b891513ce9a259a80f7d10"} Sep 17 23:39:04.404052 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:04.403998 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6" podStartSLOduration=1.403981927 podStartE2EDuration="1.403981927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:39:04.401384963 +0000 UTC m=+1983.335292097" watchObservedRunningTime="2026-09-17 23:39:04.403981927 +0000 UTC m=+1983.337889063" Sep 17 23:39:05.395678 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:05.395646 2582 generic.go:356] "Generic (PLEG): container finished" podID="0deb2f22-c55a-41e9-82a0-dd211c97888a" containerID="af62723302db9b0a2cb9cd59cbcd9e4e0ad4ccffe91560595fc87fa439339b29" exitCode=0 Sep 17 23:39:05.396078 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:05.395739 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6" event={"ID":"0deb2f22-c55a-41e9-82a0-dd211c97888a","Type":"ContainerDied","Data":"af62723302db9b0a2cb9cd59cbcd9e4e0ad4ccffe91560595fc87fa439339b29"} Sep 17 23:39:06.540783 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:06.540761 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6" Sep 17 23:39:06.691967 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:06.691900 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0deb2f22-c55a-41e9-82a0-dd211c97888a-maas-api-service-ca\") pod \"0deb2f22-c55a-41e9-82a0-dd211c97888a\" (UID: \"0deb2f22-c55a-41e9-82a0-dd211c97888a\") " Sep 17 23:39:06.692237 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:06.692213 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0deb2f22-c55a-41e9-82a0-dd211c97888a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0deb2f22-c55a-41e9-82a0-dd211c97888a" (UID: "0deb2f22-c55a-41e9-82a0-dd211c97888a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:39:06.793662 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:06.793622 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0deb2f22-c55a-41e9-82a0-dd211c97888a-maas-api-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:07.407961 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:07.407935 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6" Sep 17 23:39:07.408137 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:07.407975 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6" event={"ID":"0deb2f22-c55a-41e9-82a0-dd211c97888a","Type":"ContainerDied","Data":"bc19bc661a097774ae277eab74fa6ae98385e2b983b891513ce9a259a80f7d10"} Sep 17 23:39:07.408137 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:07.408021 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc19bc661a097774ae277eab74fa6ae98385e2b983b891513ce9a259a80f7d10" Sep 17 23:39:08.322859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.322834 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:39:08.408110 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.408030 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-envoy\") pod \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " Sep 17 23:39:08.408110 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.408064 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-credential-socket\") pod \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " Sep 17 23:39:08.408110 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.408091 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-podinfo\") pod \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " Sep 17 23:39:08.408110 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.408108 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-data\") pod \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " Sep 17 23:39:08.408436 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.408131 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-token\") pod \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " Sep 17 23:39:08.408436 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.408189 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istiod-ca-cert\") pod \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " Sep 17 23:39:08.408436 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.408226 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-workload-socket\") pod \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " Sep 17 23:39:08.408436 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.408346 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-workload-certs\") pod \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " Sep 17 23:39:08.408436 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.408380 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mqll\" (UniqueName: \"kubernetes.io/projected/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-kube-api-access-6mqll\") pod \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\" (UID: \"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba\") " Sep 17 23:39:08.408436 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.408401 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" (UID: "e67e0b2a-0666-4cd7-a0d2-0c845adc9dba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:39:08.408789 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.408416 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-data" (OuterVolumeSpecName: "istio-data") pod "e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" (UID: "e67e0b2a-0666-4cd7-a0d2-0c845adc9dba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:39:08.408789 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.408494 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" (UID: "e67e0b2a-0666-4cd7-a0d2-0c845adc9dba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:39:08.408789 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.408684 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" (UID: "e67e0b2a-0666-4cd7-a0d2-0c845adc9dba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:39:08.408789 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.408768 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" (UID: "e67e0b2a-0666-4cd7-a0d2-0c845adc9dba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:39:08.408986 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.408852 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-credential-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:08.408986 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.408870 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-data\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:08.408986 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.408883 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istiod-ca-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:08.408986 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.408895 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-workload-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:08.408986 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.408907 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-workload-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:08.410786 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.410758 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" (UID: "e67e0b2a-0666-4cd7-a0d2-0c845adc9dba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:39:08.410911 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.410852 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-token" (OuterVolumeSpecName: "istio-token") pod "e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" (UID: "e67e0b2a-0666-4cd7-a0d2-0c845adc9dba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:39:08.410963 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.410925 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-kube-api-access-6mqll" (OuterVolumeSpecName: "kube-api-access-6mqll") pod "e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" (UID: "e67e0b2a-0666-4cd7-a0d2-0c845adc9dba"). InnerVolumeSpecName "kube-api-access-6mqll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:39:08.410963 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.410925 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" (UID: "e67e0b2a-0666-4cd7-a0d2-0c845adc9dba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:39:08.416454 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.416424 2582 generic.go:356] "Generic (PLEG): container finished" podID="e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" containerID="811d657f45eed5ee8ce5c5ea75c6a10af1a0f560470e556a2019c6c3bfe97d38" exitCode=0 Sep 17 23:39:08.416618 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.416477 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" Sep 17 23:39:08.416618 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.416562 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" event={"ID":"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba","Type":"ContainerDied","Data":"811d657f45eed5ee8ce5c5ea75c6a10af1a0f560470e556a2019c6c3bfe97d38"} Sep 17 23:39:08.416710 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.416621 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm" event={"ID":"e67e0b2a-0666-4cd7-a0d2-0c845adc9dba","Type":"ContainerDied","Data":"f254a61599106a6499843e0120a52ede36912c46199d74026f214fc5f33db5fb"} Sep 17 23:39:08.416710 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.416652 2582 scope.go:117] "RemoveContainer" containerID="811d657f45eed5ee8ce5c5ea75c6a10af1a0f560470e556a2019c6c3bfe97d38" Sep 17 23:39:08.433922 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.433905 2582 scope.go:117] "RemoveContainer" containerID="10b0d6097e6c1666e6e0e896dba279cb94d4e1d5f7c6597001d612b9559a25e6" Sep 17 23:39:08.442209 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.442191 2582 scope.go:117] "RemoveContainer" containerID="811d657f45eed5ee8ce5c5ea75c6a10af1a0f560470e556a2019c6c3bfe97d38" Sep 17 23:39:08.442469 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:39:08.442450 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"811d657f45eed5ee8ce5c5ea75c6a10af1a0f560470e556a2019c6c3bfe97d38\": container with ID starting with 811d657f45eed5ee8ce5c5ea75c6a10af1a0f560470e556a2019c6c3bfe97d38 not found: ID does not exist" containerID="811d657f45eed5ee8ce5c5ea75c6a10af1a0f560470e556a2019c6c3bfe97d38" Sep 17 23:39:08.442569 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.442477 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"811d657f45eed5ee8ce5c5ea75c6a10af1a0f560470e556a2019c6c3bfe97d38"} err="failed to get container status \"811d657f45eed5ee8ce5c5ea75c6a10af1a0f560470e556a2019c6c3bfe97d38\": rpc error: code = NotFound desc = could not find container \"811d657f45eed5ee8ce5c5ea75c6a10af1a0f560470e556a2019c6c3bfe97d38\": container with ID starting with 811d657f45eed5ee8ce5c5ea75c6a10af1a0f560470e556a2019c6c3bfe97d38 not found: ID does not exist" Sep 17 23:39:08.442569 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.442494 2582 scope.go:117] "RemoveContainer" containerID="10b0d6097e6c1666e6e0e896dba279cb94d4e1d5f7c6597001d612b9559a25e6" Sep 17 23:39:08.442765 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:39:08.442749 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b0d6097e6c1666e6e0e896dba279cb94d4e1d5f7c6597001d612b9559a25e6\": container with ID starting with 10b0d6097e6c1666e6e0e896dba279cb94d4e1d5f7c6597001d612b9559a25e6 not found: ID does not exist" containerID="10b0d6097e6c1666e6e0e896dba279cb94d4e1d5f7c6597001d612b9559a25e6" Sep 17 23:39:08.442824 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.442769 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b0d6097e6c1666e6e0e896dba279cb94d4e1d5f7c6597001d612b9559a25e6"} err="failed to get container status \"10b0d6097e6c1666e6e0e896dba279cb94d4e1d5f7c6597001d612b9559a25e6\": rpc error: code = NotFound desc = could not find container \"10b0d6097e6c1666e6e0e896dba279cb94d4e1d5f7c6597001d612b9559a25e6\": container with ID starting with 10b0d6097e6c1666e6e0e896dba279cb94d4e1d5f7c6597001d612b9559a25e6 not found: ID does not exist" Sep 17 23:39:08.443798 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.443777 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm"] Sep 17 23:39:08.447167 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.447147 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-8f81ee-openshift-default-68c4d6d54d-dglsm"] Sep 17 23:39:08.509886 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.509866 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mqll\" (UniqueName: \"kubernetes.io/projected/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-kube-api-access-6mqll\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:08.509886 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.509886 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-envoy\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:08.510011 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.509898 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-podinfo\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:08.510011 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:08.509905 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba-istio-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:09.293899 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:09.293870 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" path="/var/lib/kubelet/pods/e67e0b2a-0666-4cd7-a0d2-0c845adc9dba/volumes" Sep 17 23:39:14.244397 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:14.244358 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj"] Sep 17 23:39:14.244942 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:14.244630 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj" podUID="e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2" containerName="payload-pre-processing" containerID="cri-o://818b4ccd7f9af41c0528a07e56d576ed80be7f2507e9e931b8922c96d31a9ad1" gracePeriod=30 Sep 17 23:39:14.405368 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:14.405334 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6"] Sep 17 23:39:14.408789 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:14.408760 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-accfb5-v8gm6"] Sep 17 23:39:14.452881 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:14.452848 2582 generic.go:356] "Generic (PLEG): container finished" podID="e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2" containerID="818b4ccd7f9af41c0528a07e56d576ed80be7f2507e9e931b8922c96d31a9ad1" exitCode=0 Sep 17 23:39:14.453048 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:14.452884 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj" event={"ID":"e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2","Type":"ContainerDied","Data":"818b4ccd7f9af41c0528a07e56d576ed80be7f2507e9e931b8922c96d31a9ad1"} Sep 17 23:39:14.502886 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:14.502814 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj" Sep 17 23:39:14.664351 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:14.664318 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2-plugins-config-volume\") pod \"e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2\" (UID: \"e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2\") " Sep 17 23:39:14.664574 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:14.664360 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86sz5\" (UniqueName: \"kubernetes.io/projected/e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2-kube-api-access-86sz5\") pod \"e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2\" (UID: \"e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2\") " Sep 17 23:39:14.664747 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:14.664722 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2" (UID: "e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:39:14.666466 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:14.666446 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2-kube-api-access-86sz5" (OuterVolumeSpecName: "kube-api-access-86sz5") pod "e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2" (UID: "e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2"). InnerVolumeSpecName "kube-api-access-86sz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:39:14.765367 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:14.765287 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2-plugins-config-volume\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:14.765367 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:14.765314 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86sz5\" (UniqueName: \"kubernetes.io/projected/e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2-kube-api-access-86sz5\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:15.294173 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:15.294140 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0deb2f22-c55a-41e9-82a0-dd211c97888a" path="/var/lib/kubelet/pods/0deb2f22-c55a-41e9-82a0-dd211c97888a/volumes" Sep 17 23:39:15.466882 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:15.466852 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj" Sep 17 23:39:15.466882 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:15.466867 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj" event={"ID":"e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2","Type":"ContainerDied","Data":"b32ab24c6d80083e68865e94151147bddfebe84ee57f36227f1a6ce21772fbe3"} Sep 17 23:39:15.467119 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:15.466943 2582 scope.go:117] "RemoveContainer" containerID="818b4ccd7f9af41c0528a07e56d576ed80be7f2507e9e931b8922c96d31a9ad1" Sep 17 23:39:15.485078 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:15.485052 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj"] Sep 17 23:39:15.491434 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:15.491410 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-accfb5-5fbd7d546-s5phj"] Sep 17 23:39:17.293481 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:17.293450 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2" path="/var/lib/kubelet/pods/e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2/volumes" Sep 17 23:39:20.772934 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:20.772895 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w"] Sep 17 23:39:20.773409 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:20.773188 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" podUID="1a34e881-1ea4-4e54-bd79-95ee49f68ece" containerName="istio-proxy" containerID="cri-o://7e9de78b66e600725f4ff0f56624dd97581db6b49925cd75494b8a1012e412f5" gracePeriod=30 Sep 17 23:39:21.156592 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.156492 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz"] Sep 17 23:39:21.156938 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.156925 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d00baf2f-3963-4c40-b684-6fb1e2565771" containerName="istio-proxy" Sep 17 23:39:21.156989 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.156939 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d00baf2f-3963-4c40-b684-6fb1e2565771" containerName="istio-proxy" Sep 17 23:39:21.156989 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.156950 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" containerName="istio-proxy" Sep 17 23:39:21.156989 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.156955 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" containerName="istio-proxy" Sep 17 23:39:21.156989 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.156964 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2" containerName="payload-pre-processing" Sep 17 23:39:21.156989 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.156969 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2" containerName="payload-pre-processing" Sep 17 23:39:21.156989 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.156976 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0deb2f22-c55a-41e9-82a0-dd211c97888a" containerName="revoke-keys" Sep 17 23:39:21.156989 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.156982 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0deb2f22-c55a-41e9-82a0-dd211c97888a" containerName="revoke-keys" Sep 17 23:39:21.156989 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.156989 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" containerName="istio-proxy" Sep 17 23:39:21.156989 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.156994 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" containerName="istio-proxy" Sep 17 23:39:21.157292 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.157059 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" containerName="istio-proxy" Sep 17 23:39:21.157292 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.157066 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2d8133f-5fe9-4ac1-91bc-cf4b17f2b9f2" containerName="payload-pre-processing" Sep 17 23:39:21.157292 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.157075 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0deb2f22-c55a-41e9-82a0-dd211c97888a" containerName="revoke-keys" Sep 17 23:39:21.157292 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.157085 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d00baf2f-3963-4c40-b684-6fb1e2565771" containerName="istio-proxy" Sep 17 23:39:21.161534 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.161493 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz" Sep 17 23:39:21.163954 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.163930 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:39:21.166986 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.166954 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz"] Sep 17 23:39:21.323870 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.323838 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89ae238-38af-487c-856b-14003ba0c259-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz\" (UID: \"e89ae238-38af-487c-856b-14003ba0c259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz" Sep 17 23:39:21.425441 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.425360 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89ae238-38af-487c-856b-14003ba0c259-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz\" (UID: \"e89ae238-38af-487c-856b-14003ba0c259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz" Sep 17 23:39:21.426019 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.426002 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89ae238-38af-487c-856b-14003ba0c259-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz\" (UID: \"e89ae238-38af-487c-856b-14003ba0c259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz" Sep 17 23:39:21.480827 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.480780 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz" Sep 17 23:39:21.757492 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:21.757466 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz"] Sep 17 23:39:21.759024 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:39:21.759000 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89ae238_38af_487c_856b_14003ba0c259.slice/crio-7b19e8ebeaf485d41e19b7e17442760dbaa245850d943b97190d8b38dd56fb2f WatchSource:0}: Error finding container 7b19e8ebeaf485d41e19b7e17442760dbaa245850d943b97190d8b38dd56fb2f: Status 404 returned error can't find the container with id 7b19e8ebeaf485d41e19b7e17442760dbaa245850d943b97190d8b38dd56fb2f Sep 17 23:39:22.514167 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:22.514132 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz" event={"ID":"e89ae238-38af-487c-856b-14003ba0c259","Type":"ContainerStarted","Data":"5766cd2b021260a3a3a1eb52a509140cd43eb8fd53d4305fdb50e765baabefd0"} Sep 17 23:39:22.514650 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:22.514184 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz" event={"ID":"e89ae238-38af-487c-856b-14003ba0c259","Type":"ContainerStarted","Data":"7b19e8ebeaf485d41e19b7e17442760dbaa245850d943b97190d8b38dd56fb2f"} Sep 17 23:39:22.528654 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:22.528596 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz" podStartSLOduration=1.5285772130000002 podStartE2EDuration="1.528577213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:39:22.527468694 +0000 UTC m=+2001.461375828" watchObservedRunningTime="2026-09-17 23:39:22.528577213 +0000 UTC m=+2001.462484349" Sep 17 23:39:23.521035 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:23.521002 2582 generic.go:356] "Generic (PLEG): container finished" podID="e89ae238-38af-487c-856b-14003ba0c259" containerID="5766cd2b021260a3a3a1eb52a509140cd43eb8fd53d4305fdb50e765baabefd0" exitCode=0 Sep 17 23:39:23.521427 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:23.521127 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz" event={"ID":"e89ae238-38af-487c-856b-14003ba0c259","Type":"ContainerDied","Data":"5766cd2b021260a3a3a1eb52a509140cd43eb8fd53d4305fdb50e765baabefd0"} Sep 17 23:39:24.663445 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:24.663422 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz" Sep 17 23:39:24.859522 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:24.859432 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89ae238-38af-487c-856b-14003ba0c259-maas-api-service-ca\") pod \"e89ae238-38af-487c-856b-14003ba0c259\" (UID: \"e89ae238-38af-487c-856b-14003ba0c259\") " Sep 17 23:39:24.859819 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:24.859793 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89ae238-38af-487c-856b-14003ba0c259-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e89ae238-38af-487c-856b-14003ba0c259" (UID: "e89ae238-38af-487c-856b-14003ba0c259"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:39:24.960999 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:24.960974 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89ae238-38af-487c-856b-14003ba0c259-maas-api-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:25.533185 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:25.533156 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz" Sep 17 23:39:25.533323 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:25.533187 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz" event={"ID":"e89ae238-38af-487c-856b-14003ba0c259","Type":"ContainerDied","Data":"7b19e8ebeaf485d41e19b7e17442760dbaa245850d943b97190d8b38dd56fb2f"} Sep 17 23:39:25.533323 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:25.533230 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b19e8ebeaf485d41e19b7e17442760dbaa245850d943b97190d8b38dd56fb2f" Sep 17 23:39:26.030253 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.030229 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:39:26.072468 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.072402 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-envoy\") pod \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " Sep 17 23:39:26.072468 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.072444 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qt9q9\" (UniqueName: \"kubernetes.io/projected/1a34e881-1ea4-4e54-bd79-95ee49f68ece-kube-api-access-qt9q9\") pod \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " Sep 17 23:39:26.072647 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.072540 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-workload-certs\") pod \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " Sep 17 23:39:26.072647 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.072566 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-data\") pod \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " Sep 17 23:39:26.072647 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.072611 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-podinfo\") pod \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " Sep 17 23:39:26.072647 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.072637 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istiod-ca-cert\") pod \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " Sep 17 23:39:26.072853 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.072693 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-workload-socket\") pod \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " Sep 17 23:39:26.072853 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.072734 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-token\") pod \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " Sep 17 23:39:26.072853 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.072768 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-credential-socket\") pod \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\" (UID: \"1a34e881-1ea4-4e54-bd79-95ee49f68ece\") " Sep 17 23:39:26.073013 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.072860 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a34e881-1ea4-4e54-bd79-95ee49f68ece" (UID: "1a34e881-1ea4-4e54-bd79-95ee49f68ece"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:39:26.073091 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.073066 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a34e881-1ea4-4e54-bd79-95ee49f68ece" (UID: "1a34e881-1ea4-4e54-bd79-95ee49f68ece"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:39:26.073165 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.073131 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a34e881-1ea4-4e54-bd79-95ee49f68ece" (UID: "1a34e881-1ea4-4e54-bd79-95ee49f68ece"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:39:26.073271 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.073247 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a34e881-1ea4-4e54-bd79-95ee49f68ece" (UID: "1a34e881-1ea4-4e54-bd79-95ee49f68ece"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:39:26.073366 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.073345 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a34e881-1ea4-4e54-bd79-95ee49f68ece" (UID: "1a34e881-1ea4-4e54-bd79-95ee49f68ece"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:39:26.073530 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.073476 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istiod-ca-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:26.073530 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.073516 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-workload-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:26.073674 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.073531 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-credential-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:26.073674 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.073545 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-workload-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:26.073674 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.073558 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-data\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:26.075109 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.075081 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a34e881-1ea4-4e54-bd79-95ee49f68ece" (UID: "1a34e881-1ea4-4e54-bd79-95ee49f68ece"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:39:26.075265 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.075111 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a34e881-1ea4-4e54-bd79-95ee49f68ece" (UID: "1a34e881-1ea4-4e54-bd79-95ee49f68ece"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:39:26.075400 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.075371 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a34e881-1ea4-4e54-bd79-95ee49f68ece-kube-api-access-qt9q9" (OuterVolumeSpecName: "kube-api-access-qt9q9") pod "1a34e881-1ea4-4e54-bd79-95ee49f68ece" (UID: "1a34e881-1ea4-4e54-bd79-95ee49f68ece"). InnerVolumeSpecName "kube-api-access-qt9q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:39:26.075570 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.075549 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a34e881-1ea4-4e54-bd79-95ee49f68ece" (UID: "1a34e881-1ea4-4e54-bd79-95ee49f68ece"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:39:26.174305 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.174283 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-podinfo\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:26.174305 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.174304 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:26.174446 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.174312 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a34e881-1ea4-4e54-bd79-95ee49f68ece-istio-envoy\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:26.174446 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.174321 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt9q9\" (UniqueName: \"kubernetes.io/projected/1a34e881-1ea4-4e54-bd79-95ee49f68ece-kube-api-access-qt9q9\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:26.542091 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.542055 2582 generic.go:356] "Generic (PLEG): container finished" podID="1a34e881-1ea4-4e54-bd79-95ee49f68ece" containerID="7e9de78b66e600725f4ff0f56624dd97581db6b49925cd75494b8a1012e412f5" exitCode=0 Sep 17 23:39:26.542268 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.542095 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" event={"ID":"1a34e881-1ea4-4e54-bd79-95ee49f68ece","Type":"ContainerDied","Data":"7e9de78b66e600725f4ff0f56624dd97581db6b49925cd75494b8a1012e412f5"} Sep 17 23:39:26.542268 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.542111 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" Sep 17 23:39:26.542268 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.542150 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w" event={"ID":"1a34e881-1ea4-4e54-bd79-95ee49f68ece","Type":"ContainerDied","Data":"42e5d7d5f678c33a0e140ed7f58aced63a3add2e1fcf8cad25f0acca99ff42ba"} Sep 17 23:39:26.542268 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.542171 2582 scope.go:117] "RemoveContainer" containerID="7e9de78b66e600725f4ff0f56624dd97581db6b49925cd75494b8a1012e412f5" Sep 17 23:39:26.552210 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.552196 2582 scope.go:117] "RemoveContainer" containerID="a968f22c8c5a5d71b3af7f340db6092b14cf86c2153d65c54a9c79a54202a0b4" Sep 17 23:39:26.560067 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.560048 2582 scope.go:117] "RemoveContainer" containerID="7e9de78b66e600725f4ff0f56624dd97581db6b49925cd75494b8a1012e412f5" Sep 17 23:39:26.560326 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:39:26.560308 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e9de78b66e600725f4ff0f56624dd97581db6b49925cd75494b8a1012e412f5\": container with ID starting with 7e9de78b66e600725f4ff0f56624dd97581db6b49925cd75494b8a1012e412f5 not found: ID does not exist" containerID="7e9de78b66e600725f4ff0f56624dd97581db6b49925cd75494b8a1012e412f5" Sep 17 23:39:26.560393 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.560332 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e9de78b66e600725f4ff0f56624dd97581db6b49925cd75494b8a1012e412f5"} err="failed to get container status \"7e9de78b66e600725f4ff0f56624dd97581db6b49925cd75494b8a1012e412f5\": rpc error: code = NotFound desc = could not find container \"7e9de78b66e600725f4ff0f56624dd97581db6b49925cd75494b8a1012e412f5\": container with ID starting with 7e9de78b66e600725f4ff0f56624dd97581db6b49925cd75494b8a1012e412f5 not found: ID does not exist" Sep 17 23:39:26.560393 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.560349 2582 scope.go:117] "RemoveContainer" containerID="a968f22c8c5a5d71b3af7f340db6092b14cf86c2153d65c54a9c79a54202a0b4" Sep 17 23:39:26.560620 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:39:26.560602 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a968f22c8c5a5d71b3af7f340db6092b14cf86c2153d65c54a9c79a54202a0b4\": container with ID starting with a968f22c8c5a5d71b3af7f340db6092b14cf86c2153d65c54a9c79a54202a0b4 not found: ID does not exist" containerID="a968f22c8c5a5d71b3af7f340db6092b14cf86c2153d65c54a9c79a54202a0b4" Sep 17 23:39:26.560665 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.560626 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a968f22c8c5a5d71b3af7f340db6092b14cf86c2153d65c54a9c79a54202a0b4"} err="failed to get container status \"a968f22c8c5a5d71b3af7f340db6092b14cf86c2153d65c54a9c79a54202a0b4\": rpc error: code = NotFound desc = could not find container \"a968f22c8c5a5d71b3af7f340db6092b14cf86c2153d65c54a9c79a54202a0b4\": container with ID starting with a968f22c8c5a5d71b3af7f340db6092b14cf86c2153d65c54a9c79a54202a0b4 not found: ID does not exist" Sep 17 23:39:26.564517 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.564486 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w"] Sep 17 23:39:26.568644 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:26.568621 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-accfb5-openshift-default-8f7559dbd-z2w5w"] Sep 17 23:39:27.293720 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:27.293687 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a34e881-1ea4-4e54-bd79-95ee49f68ece" path="/var/lib/kubelet/pods/1a34e881-1ea4-4e54-bd79-95ee49f68ece/volumes" Sep 17 23:39:31.602653 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:31.602619 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d90df4-29828130-v78x7"] Sep 17 23:39:31.605894 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:31.605869 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d90df4-29828130-v78x7"] Sep 17 23:39:31.793993 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:31.793954 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz"] Sep 17 23:39:31.796269 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:31.796242 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d90df4-l8jvz"] Sep 17 23:39:33.295149 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:33.295118 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc9318ac-fd7c-41cc-a573-a91a09360db2" path="/var/lib/kubelet/pods/cc9318ac-fd7c-41cc-a573-a91a09360db2/volumes" Sep 17 23:39:33.295620 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:33.295427 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e89ae238-38af-487c-856b-14003ba0c259" path="/var/lib/kubelet/pods/e89ae238-38af-487c-856b-14003ba0c259/volumes" Sep 17 23:39:38.276249 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.276212 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls"] Sep 17 23:39:38.276831 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.276530 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" podUID="a2e7b030-1b71-4ce5-8232-1845609394a2" containerName="istio-proxy" containerID="cri-o://b9d41583181aa3f257b87bf2bca32464d6f466062fed9a6495d89ebb4b489817" gracePeriod=30 Sep 17 23:39:38.646244 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.646172 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf"] Sep 17 23:39:38.646690 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.646674 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89ae238-38af-487c-856b-14003ba0c259" containerName="revoke-keys" Sep 17 23:39:38.646745 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.646692 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89ae238-38af-487c-856b-14003ba0c259" containerName="revoke-keys" Sep 17 23:39:38.646745 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.646710 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a34e881-1ea4-4e54-bd79-95ee49f68ece" containerName="istio-proxy" Sep 17 23:39:38.646745 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.646717 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a34e881-1ea4-4e54-bd79-95ee49f68ece" containerName="istio-proxy" Sep 17 23:39:38.646745 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.646728 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a34e881-1ea4-4e54-bd79-95ee49f68ece" containerName="istio-proxy" Sep 17 23:39:38.646745 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.646735 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a34e881-1ea4-4e54-bd79-95ee49f68ece" containerName="istio-proxy" Sep 17 23:39:38.646898 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.646826 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e89ae238-38af-487c-856b-14003ba0c259" containerName="revoke-keys" Sep 17 23:39:38.646898 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.646835 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e67e0b2a-0666-4cd7-a0d2-0c845adc9dba" containerName="istio-proxy" Sep 17 23:39:38.646898 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.646843 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a34e881-1ea4-4e54-bd79-95ee49f68ece" containerName="istio-proxy" Sep 17 23:39:38.646898 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.646849 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a34e881-1ea4-4e54-bd79-95ee49f68ece" containerName="istio-proxy" Sep 17 23:39:38.651272 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.651254 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf" Sep 17 23:39:38.653588 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.653571 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:39:38.657711 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.657687 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf"] Sep 17 23:39:38.687079 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.687049 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e55b8e5-8a26-43c2-8de9-b783996a50bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf\" (UID: \"4e55b8e5-8a26-43c2-8de9-b783996a50bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf" Sep 17 23:39:38.788276 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.788236 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e55b8e5-8a26-43c2-8de9-b783996a50bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf\" (UID: \"4e55b8e5-8a26-43c2-8de9-b783996a50bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf" Sep 17 23:39:38.788857 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.788840 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e55b8e5-8a26-43c2-8de9-b783996a50bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf\" (UID: \"4e55b8e5-8a26-43c2-8de9-b783996a50bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf" Sep 17 23:39:38.964319 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:38.964243 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf" Sep 17 23:39:39.236954 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:39.236928 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf"] Sep 17 23:39:39.239572 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:39:39.239543 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e55b8e5_8a26_43c2_8de9_b783996a50bf.slice/crio-4af41c4a923afd50d74ec5f65784326485b95445a1d54920345f78396c3e4126 WatchSource:0}: Error finding container 4af41c4a923afd50d74ec5f65784326485b95445a1d54920345f78396c3e4126: Status 404 returned error can't find the container with id 4af41c4a923afd50d74ec5f65784326485b95445a1d54920345f78396c3e4126 Sep 17 23:39:39.637060 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:39.637026 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf" event={"ID":"4e55b8e5-8a26-43c2-8de9-b783996a50bf","Type":"ContainerStarted","Data":"951e4e2300700142899e9a8e769e9b16490adeddfc2ce5b95d6d0df13cf78ea0"} Sep 17 23:39:39.637454 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:39.637080 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf" event={"ID":"4e55b8e5-8a26-43c2-8de9-b783996a50bf","Type":"ContainerStarted","Data":"4af41c4a923afd50d74ec5f65784326485b95445a1d54920345f78396c3e4126"} Sep 17 23:39:39.652611 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:39.652565 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf" podStartSLOduration=1.6525494200000002 podStartE2EDuration="1.65254942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:39:39.650288167 +0000 UTC m=+2018.584195304" watchObservedRunningTime="2026-09-17 23:39:39.65254942 +0000 UTC m=+2018.586456555" Sep 17 23:39:40.644423 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:40.644390 2582 generic.go:356] "Generic (PLEG): container finished" podID="4e55b8e5-8a26-43c2-8de9-b783996a50bf" containerID="951e4e2300700142899e9a8e769e9b16490adeddfc2ce5b95d6d0df13cf78ea0" exitCode=0 Sep 17 23:39:40.644828 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:40.644479 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf" event={"ID":"4e55b8e5-8a26-43c2-8de9-b783996a50bf","Type":"ContainerDied","Data":"951e4e2300700142899e9a8e769e9b16490adeddfc2ce5b95d6d0df13cf78ea0"} Sep 17 23:39:41.782879 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:41.782852 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf" Sep 17 23:39:41.814119 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:41.814095 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e55b8e5-8a26-43c2-8de9-b783996a50bf-maas-api-service-ca\") pod \"4e55b8e5-8a26-43c2-8de9-b783996a50bf\" (UID: \"4e55b8e5-8a26-43c2-8de9-b783996a50bf\") " Sep 17 23:39:41.814395 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:41.814375 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e55b8e5-8a26-43c2-8de9-b783996a50bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e55b8e5-8a26-43c2-8de9-b783996a50bf" (UID: "4e55b8e5-8a26-43c2-8de9-b783996a50bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:39:41.814523 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:41.814486 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e55b8e5-8a26-43c2-8de9-b783996a50bf-maas-api-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:42.658621 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:42.658590 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf" Sep 17 23:39:42.658806 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:42.658585 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf" event={"ID":"4e55b8e5-8a26-43c2-8de9-b783996a50bf","Type":"ContainerDied","Data":"4af41c4a923afd50d74ec5f65784326485b95445a1d54920345f78396c3e4126"} Sep 17 23:39:42.658806 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:42.658740 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4af41c4a923afd50d74ec5f65784326485b95445a1d54920345f78396c3e4126" Sep 17 23:39:43.519805 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.519780 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:39:43.630613 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.630536 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-token\") pod \"a2e7b030-1b71-4ce5-8232-1845609394a2\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " Sep 17 23:39:43.630613 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.630577 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-data\") pod \"a2e7b030-1b71-4ce5-8232-1845609394a2\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " Sep 17 23:39:43.630826 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.630631 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e7b030-1b71-4ce5-8232-1845609394a2-istiod-ca-cert\") pod \"a2e7b030-1b71-4ce5-8232-1845609394a2\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " Sep 17 23:39:43.630826 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.630663 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-credential-socket\") pod \"a2e7b030-1b71-4ce5-8232-1845609394a2\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " Sep 17 23:39:43.630826 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.630722 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfr7x\" (UniqueName: \"kubernetes.io/projected/a2e7b030-1b71-4ce5-8232-1845609394a2-kube-api-access-zfr7x\") pod \"a2e7b030-1b71-4ce5-8232-1845609394a2\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " Sep 17 23:39:43.630826 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.630752 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-workload-socket\") pod \"a2e7b030-1b71-4ce5-8232-1845609394a2\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " Sep 17 23:39:43.630826 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.630782 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-envoy\") pod \"a2e7b030-1b71-4ce5-8232-1845609394a2\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " Sep 17 23:39:43.630826 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.630806 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-workload-certs\") pod \"a2e7b030-1b71-4ce5-8232-1845609394a2\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " Sep 17 23:39:43.631135 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.630870 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-podinfo\") pod \"a2e7b030-1b71-4ce5-8232-1845609394a2\" (UID: \"a2e7b030-1b71-4ce5-8232-1845609394a2\") " Sep 17 23:39:43.631135 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.630974 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-data" (OuterVolumeSpecName: "istio-data") pod "a2e7b030-1b71-4ce5-8232-1845609394a2" (UID: "a2e7b030-1b71-4ce5-8232-1845609394a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:39:43.631135 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.631000 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a2e7b030-1b71-4ce5-8232-1845609394a2" (UID: "a2e7b030-1b71-4ce5-8232-1845609394a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:39:43.631135 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.631035 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a2e7b030-1b71-4ce5-8232-1845609394a2" (UID: "a2e7b030-1b71-4ce5-8232-1845609394a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:39:43.631135 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.631043 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e7b030-1b71-4ce5-8232-1845609394a2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a2e7b030-1b71-4ce5-8232-1845609394a2" (UID: "a2e7b030-1b71-4ce5-8232-1845609394a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:39:43.631404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.631224 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-data\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:43.631404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.631243 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e7b030-1b71-4ce5-8232-1845609394a2-istiod-ca-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:43.631404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.631257 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-credential-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:43.631404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.631271 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-workload-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:43.631404 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.631287 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a2e7b030-1b71-4ce5-8232-1845609394a2" (UID: "a2e7b030-1b71-4ce5-8232-1845609394a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:39:43.632815 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.632794 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-token" (OuterVolumeSpecName: "istio-token") pod "a2e7b030-1b71-4ce5-8232-1845609394a2" (UID: "a2e7b030-1b71-4ce5-8232-1845609394a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:39:43.632941 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.632922 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e7b030-1b71-4ce5-8232-1845609394a2-kube-api-access-zfr7x" (OuterVolumeSpecName: "kube-api-access-zfr7x") pod "a2e7b030-1b71-4ce5-8232-1845609394a2" (UID: "a2e7b030-1b71-4ce5-8232-1845609394a2"). InnerVolumeSpecName "kube-api-access-zfr7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:39:43.633002 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.632983 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a2e7b030-1b71-4ce5-8232-1845609394a2" (UID: "a2e7b030-1b71-4ce5-8232-1845609394a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:39:43.633041 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.632992 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a2e7b030-1b71-4ce5-8232-1845609394a2" (UID: "a2e7b030-1b71-4ce5-8232-1845609394a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:39:43.665940 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.665911 2582 generic.go:356] "Generic (PLEG): container finished" podID="a2e7b030-1b71-4ce5-8232-1845609394a2" containerID="b9d41583181aa3f257b87bf2bca32464d6f466062fed9a6495d89ebb4b489817" exitCode=0 Sep 17 23:39:43.666116 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.665961 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" event={"ID":"a2e7b030-1b71-4ce5-8232-1845609394a2","Type":"ContainerDied","Data":"b9d41583181aa3f257b87bf2bca32464d6f466062fed9a6495d89ebb4b489817"} Sep 17 23:39:43.666116 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.665965 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" Sep 17 23:39:43.666116 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.666000 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls" event={"ID":"a2e7b030-1b71-4ce5-8232-1845609394a2","Type":"ContainerDied","Data":"9d662e7a969596ff4f89acc9486daaad5e6052479422f7699f7fd3f9a9f2a450"} Sep 17 23:39:43.666116 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.666021 2582 scope.go:117] "RemoveContainer" containerID="b9d41583181aa3f257b87bf2bca32464d6f466062fed9a6495d89ebb4b489817" Sep 17 23:39:43.675196 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.675181 2582 scope.go:117] "RemoveContainer" containerID="5a33108fc17c20bdc3c9c9d52142f27180752a6622513a47f4ae3d79a3bb4519" Sep 17 23:39:43.683237 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.683222 2582 scope.go:117] "RemoveContainer" containerID="b9d41583181aa3f257b87bf2bca32464d6f466062fed9a6495d89ebb4b489817" Sep 17 23:39:43.683485 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:39:43.683469 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d41583181aa3f257b87bf2bca32464d6f466062fed9a6495d89ebb4b489817\": container with ID starting with b9d41583181aa3f257b87bf2bca32464d6f466062fed9a6495d89ebb4b489817 not found: ID does not exist" containerID="b9d41583181aa3f257b87bf2bca32464d6f466062fed9a6495d89ebb4b489817" Sep 17 23:39:43.683540 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.683491 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d41583181aa3f257b87bf2bca32464d6f466062fed9a6495d89ebb4b489817"} err="failed to get container status \"b9d41583181aa3f257b87bf2bca32464d6f466062fed9a6495d89ebb4b489817\": rpc error: code = NotFound desc = could not find container \"b9d41583181aa3f257b87bf2bca32464d6f466062fed9a6495d89ebb4b489817\": container with ID starting with b9d41583181aa3f257b87bf2bca32464d6f466062fed9a6495d89ebb4b489817 not found: ID does not exist" Sep 17 23:39:43.683540 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.683528 2582 scope.go:117] "RemoveContainer" containerID="5a33108fc17c20bdc3c9c9d52142f27180752a6622513a47f4ae3d79a3bb4519" Sep 17 23:39:43.683751 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:39:43.683737 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a33108fc17c20bdc3c9c9d52142f27180752a6622513a47f4ae3d79a3bb4519\": container with ID starting with 5a33108fc17c20bdc3c9c9d52142f27180752a6622513a47f4ae3d79a3bb4519 not found: ID does not exist" containerID="5a33108fc17c20bdc3c9c9d52142f27180752a6622513a47f4ae3d79a3bb4519" Sep 17 23:39:43.683799 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.683754 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a33108fc17c20bdc3c9c9d52142f27180752a6622513a47f4ae3d79a3bb4519"} err="failed to get container status \"5a33108fc17c20bdc3c9c9d52142f27180752a6622513a47f4ae3d79a3bb4519\": rpc error: code = NotFound desc = could not find container \"5a33108fc17c20bdc3c9c9d52142f27180752a6622513a47f4ae3d79a3bb4519\": container with ID starting with 5a33108fc17c20bdc3c9c9d52142f27180752a6622513a47f4ae3d79a3bb4519 not found: ID does not exist" Sep 17 23:39:43.689007 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.688985 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls"] Sep 17 23:39:43.692299 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.692278 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d90df4-openshift-default-7f8cdc4b8b-jq9ls"] Sep 17 23:39:43.731899 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.731877 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfr7x\" (UniqueName: \"kubernetes.io/projected/a2e7b030-1b71-4ce5-8232-1845609394a2-kube-api-access-zfr7x\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:43.731899 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.731897 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-envoy\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:43.732023 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.731906 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e7b030-1b71-4ce5-8232-1845609394a2-workload-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:43.732023 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.731914 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-podinfo\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:43.732023 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:43.731922 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e7b030-1b71-4ce5-8232-1845609394a2-istio-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:45.293881 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:45.293852 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2e7b030-1b71-4ce5-8232-1845609394a2" path="/var/lib/kubelet/pods/a2e7b030-1b71-4ce5-8232-1845609394a2/volumes" Sep 17 23:39:49.420186 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:49.420146 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf"] Sep 17 23:39:49.422313 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:49.422282 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d92920-qwdbf"] Sep 17 23:39:51.294026 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:51.293991 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e55b8e5-8a26-43c2-8de9-b783996a50bf" path="/var/lib/kubelet/pods/4e55b8e5-8a26-43c2-8de9-b783996a50bf/volumes" Sep 17 23:39:56.193262 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:56.193221 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7"] Sep 17 23:39:56.193688 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:56.193558 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" podUID="dc656650-7bb4-46ee-9b96-55cc6cc7f124" containerName="istio-proxy" containerID="cri-o://3565dbe8e458195802c7950ad8e7b266f720da80760f50e56ac9e4d5301b7466" gracePeriod=30 Sep 17 23:39:58.971219 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:58.971180 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7cd595f459-2wdxz"] Sep 17 23:39:58.971725 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:58.971385 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7cd595f459-2wdxz" podUID="7ed809b5-b370-4deb-8b01-69513dabaab3" containerName="manager" containerID="cri-o://3b122221a8a1ebfb195396a4e36cec0e8edd9c836396140ff07d52c69ee9b9c9" gracePeriod=30 Sep 17 23:39:59.222719 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.222656 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd595f459-2wdxz" Sep 17 23:39:59.371201 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.371174 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed809b5-b370-4deb-8b01-69513dabaab3-metrics-certs\") pod \"7ed809b5-b370-4deb-8b01-69513dabaab3\" (UID: \"7ed809b5-b370-4deb-8b01-69513dabaab3\") " Sep 17 23:39:59.371379 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.371249 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkd8f\" (UniqueName: \"kubernetes.io/projected/7ed809b5-b370-4deb-8b01-69513dabaab3-kube-api-access-gkd8f\") pod \"7ed809b5-b370-4deb-8b01-69513dabaab3\" (UID: \"7ed809b5-b370-4deb-8b01-69513dabaab3\") " Sep 17 23:39:59.371379 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.371341 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ed809b5-b370-4deb-8b01-69513dabaab3-webhook-cert\") pod \"7ed809b5-b370-4deb-8b01-69513dabaab3\" (UID: \"7ed809b5-b370-4deb-8b01-69513dabaab3\") " Sep 17 23:39:59.373253 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.373221 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed809b5-b370-4deb-8b01-69513dabaab3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7ed809b5-b370-4deb-8b01-69513dabaab3" (UID: "7ed809b5-b370-4deb-8b01-69513dabaab3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:39:59.373368 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.373278 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed809b5-b370-4deb-8b01-69513dabaab3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7ed809b5-b370-4deb-8b01-69513dabaab3" (UID: "7ed809b5-b370-4deb-8b01-69513dabaab3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:39:59.373368 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.373350 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed809b5-b370-4deb-8b01-69513dabaab3-kube-api-access-gkd8f" (OuterVolumeSpecName: "kube-api-access-gkd8f") pod "7ed809b5-b370-4deb-8b01-69513dabaab3" (UID: "7ed809b5-b370-4deb-8b01-69513dabaab3"). InnerVolumeSpecName "kube-api-access-gkd8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:39:59.472038 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.472012 2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ed809b5-b370-4deb-8b01-69513dabaab3-webhook-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:59.472038 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.472032 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed809b5-b370-4deb-8b01-69513dabaab3-metrics-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:59.472038 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.472041 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkd8f\" (UniqueName: \"kubernetes.io/projected/7ed809b5-b370-4deb-8b01-69513dabaab3-kube-api-access-gkd8f\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:39:59.763823 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.763787 2582 generic.go:356] "Generic (PLEG): container finished" podID="7ed809b5-b370-4deb-8b01-69513dabaab3" containerID="3b122221a8a1ebfb195396a4e36cec0e8edd9c836396140ff07d52c69ee9b9c9" exitCode=0 Sep 17 23:39:59.764007 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.763821 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd595f459-2wdxz" event={"ID":"7ed809b5-b370-4deb-8b01-69513dabaab3","Type":"ContainerDied","Data":"3b122221a8a1ebfb195396a4e36cec0e8edd9c836396140ff07d52c69ee9b9c9"} Sep 17 23:39:59.764007 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.763834 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd595f459-2wdxz" Sep 17 23:39:59.764007 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.763875 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd595f459-2wdxz" event={"ID":"7ed809b5-b370-4deb-8b01-69513dabaab3","Type":"ContainerDied","Data":"18ad73c6464dada3d4d3a0c2852ae5a8f2c7261f959544bc13ddff518119bc20"} Sep 17 23:39:59.764007 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.763901 2582 scope.go:117] "RemoveContainer" containerID="3b122221a8a1ebfb195396a4e36cec0e8edd9c836396140ff07d52c69ee9b9c9" Sep 17 23:39:59.775089 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.775074 2582 scope.go:117] "RemoveContainer" containerID="3b122221a8a1ebfb195396a4e36cec0e8edd9c836396140ff07d52c69ee9b9c9" Sep 17 23:39:59.775373 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:39:59.775354 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b122221a8a1ebfb195396a4e36cec0e8edd9c836396140ff07d52c69ee9b9c9\": container with ID starting with 3b122221a8a1ebfb195396a4e36cec0e8edd9c836396140ff07d52c69ee9b9c9 not found: ID does not exist" containerID="3b122221a8a1ebfb195396a4e36cec0e8edd9c836396140ff07d52c69ee9b9c9" Sep 17 23:39:59.775447 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.775383 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b122221a8a1ebfb195396a4e36cec0e8edd9c836396140ff07d52c69ee9b9c9"} err="failed to get container status \"3b122221a8a1ebfb195396a4e36cec0e8edd9c836396140ff07d52c69ee9b9c9\": rpc error: code = NotFound desc = could not find container \"3b122221a8a1ebfb195396a4e36cec0e8edd9c836396140ff07d52c69ee9b9c9\": container with ID starting with 3b122221a8a1ebfb195396a4e36cec0e8edd9c836396140ff07d52c69ee9b9c9 not found: ID does not exist" Sep 17 23:39:59.784494 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.784473 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7cd595f459-2wdxz"] Sep 17 23:39:59.788121 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.788093 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7cd595f459-2wdxz"] Sep 17 23:39:59.895135 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.895105 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 23:39:59.895238 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:39:59.895172 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" podUID="dc656650-7bb4-46ee-9b96-55cc6cc7f124" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 23:40:01.294163 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.294136 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ed809b5-b370-4deb-8b01-69513dabaab3" path="/var/lib/kubelet/pods/7ed809b5-b370-4deb-8b01-69513dabaab3/volumes" Sep 17 23:40:01.444615 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.444591 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:40:01.592595 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.592558 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-workload-certs\") pod \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " Sep 17 23:40:01.592786 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.592653 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-workload-socket\") pod \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " Sep 17 23:40:01.592786 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.592694 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-credential-socket\") pod \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " Sep 17 23:40:01.592786 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.592732 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istiod-ca-cert\") pod \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " Sep 17 23:40:01.592786 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.592771 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-envoy\") pod \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " Sep 17 23:40:01.593004 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.592799 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-data\") pod \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " Sep 17 23:40:01.593004 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.592855 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-token\") pod \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " Sep 17 23:40:01.593004 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.592893 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-podinfo\") pod \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " Sep 17 23:40:01.593004 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.592906 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dc656650-7bb4-46ee-9b96-55cc6cc7f124" (UID: "dc656650-7bb4-46ee-9b96-55cc6cc7f124"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:40:01.593004 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.592920 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whcmw\" (UniqueName: \"kubernetes.io/projected/dc656650-7bb4-46ee-9b96-55cc6cc7f124-kube-api-access-whcmw\") pod \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\" (UID: \"dc656650-7bb4-46ee-9b96-55cc6cc7f124\") " Sep 17 23:40:01.593257 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.593023 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dc656650-7bb4-46ee-9b96-55cc6cc7f124" (UID: "dc656650-7bb4-46ee-9b96-55cc6cc7f124"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:40:01.593257 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.593069 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dc656650-7bb4-46ee-9b96-55cc6cc7f124" (UID: "dc656650-7bb4-46ee-9b96-55cc6cc7f124"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:40:01.593358 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.593295 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-workload-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:40:01.593358 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.593315 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-workload-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:40:01.593358 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.593329 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-credential-socket\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:40:01.593596 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.593577 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-data" (OuterVolumeSpecName: "istio-data") pod "dc656650-7bb4-46ee-9b96-55cc6cc7f124" (UID: "dc656650-7bb4-46ee-9b96-55cc6cc7f124"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:40:01.593719 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.593691 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dc656650-7bb4-46ee-9b96-55cc6cc7f124" (UID: "dc656650-7bb4-46ee-9b96-55cc6cc7f124"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:40:01.596185 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.596156 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-token" (OuterVolumeSpecName: "istio-token") pod "dc656650-7bb4-46ee-9b96-55cc6cc7f124" (UID: "dc656650-7bb4-46ee-9b96-55cc6cc7f124"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:40:01.596185 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.596168 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc656650-7bb4-46ee-9b96-55cc6cc7f124-kube-api-access-whcmw" (OuterVolumeSpecName: "kube-api-access-whcmw") pod "dc656650-7bb4-46ee-9b96-55cc6cc7f124" (UID: "dc656650-7bb4-46ee-9b96-55cc6cc7f124"). InnerVolumeSpecName "kube-api-access-whcmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:40:01.596353 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.596260 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dc656650-7bb4-46ee-9b96-55cc6cc7f124" (UID: "dc656650-7bb4-46ee-9b96-55cc6cc7f124"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 23:40:01.596408 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.596386 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dc656650-7bb4-46ee-9b96-55cc6cc7f124" (UID: "dc656650-7bb4-46ee-9b96-55cc6cc7f124"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:40:01.694835 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.694802 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istiod-ca-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:40:01.694835 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.694837 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-envoy\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:40:01.695006 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.694851 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-data\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:40:01.695006 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.694864 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-token\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:40:01.695006 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.694878 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc656650-7bb4-46ee-9b96-55cc6cc7f124-istio-podinfo\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:40:01.695006 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.694891 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whcmw\" (UniqueName: \"kubernetes.io/projected/dc656650-7bb4-46ee-9b96-55cc6cc7f124-kube-api-access-whcmw\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:40:01.783244 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.783206 2582 generic.go:356] "Generic (PLEG): container finished" podID="dc656650-7bb4-46ee-9b96-55cc6cc7f124" containerID="3565dbe8e458195802c7950ad8e7b266f720da80760f50e56ac9e4d5301b7466" exitCode=0 Sep 17 23:40:01.783427 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.783300 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" event={"ID":"dc656650-7bb4-46ee-9b96-55cc6cc7f124","Type":"ContainerDied","Data":"3565dbe8e458195802c7950ad8e7b266f720da80760f50e56ac9e4d5301b7466"} Sep 17 23:40:01.783427 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.783323 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" Sep 17 23:40:01.783427 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.783359 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7" event={"ID":"dc656650-7bb4-46ee-9b96-55cc6cc7f124","Type":"ContainerDied","Data":"7ff0e53b1968b9c5ff6002912ef015c6dfc13ed59c60ced154ebf536c43b75a9"} Sep 17 23:40:01.783427 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.783392 2582 scope.go:117] "RemoveContainer" containerID="3565dbe8e458195802c7950ad8e7b266f720da80760f50e56ac9e4d5301b7466" Sep 17 23:40:01.794381 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.794366 2582 scope.go:117] "RemoveContainer" containerID="20dff8cb474387c803aacce3199ac60c83df7f55994d47755d6d778e61bf3e39" Sep 17 23:40:01.803197 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.803173 2582 scope.go:117] "RemoveContainer" containerID="3565dbe8e458195802c7950ad8e7b266f720da80760f50e56ac9e4d5301b7466" Sep 17 23:40:01.803461 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:40:01.803443 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3565dbe8e458195802c7950ad8e7b266f720da80760f50e56ac9e4d5301b7466\": container with ID starting with 3565dbe8e458195802c7950ad8e7b266f720da80760f50e56ac9e4d5301b7466 not found: ID does not exist" containerID="3565dbe8e458195802c7950ad8e7b266f720da80760f50e56ac9e4d5301b7466" Sep 17 23:40:01.803515 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.803471 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3565dbe8e458195802c7950ad8e7b266f720da80760f50e56ac9e4d5301b7466"} err="failed to get container status \"3565dbe8e458195802c7950ad8e7b266f720da80760f50e56ac9e4d5301b7466\": rpc error: code = NotFound desc = could not find container \"3565dbe8e458195802c7950ad8e7b266f720da80760f50e56ac9e4d5301b7466\": container with ID starting with 3565dbe8e458195802c7950ad8e7b266f720da80760f50e56ac9e4d5301b7466 not found: ID does not exist" Sep 17 23:40:01.803515 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.803487 2582 scope.go:117] "RemoveContainer" containerID="20dff8cb474387c803aacce3199ac60c83df7f55994d47755d6d778e61bf3e39" Sep 17 23:40:01.803773 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:40:01.803754 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20dff8cb474387c803aacce3199ac60c83df7f55994d47755d6d778e61bf3e39\": container with ID starting with 20dff8cb474387c803aacce3199ac60c83df7f55994d47755d6d778e61bf3e39 not found: ID does not exist" containerID="20dff8cb474387c803aacce3199ac60c83df7f55994d47755d6d778e61bf3e39" Sep 17 23:40:01.803812 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.803778 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20dff8cb474387c803aacce3199ac60c83df7f55994d47755d6d778e61bf3e39"} err="failed to get container status \"20dff8cb474387c803aacce3199ac60c83df7f55994d47755d6d778e61bf3e39\": rpc error: code = NotFound desc = could not find container \"20dff8cb474387c803aacce3199ac60c83df7f55994d47755d6d778e61bf3e39\": container with ID starting with 20dff8cb474387c803aacce3199ac60c83df7f55994d47755d6d778e61bf3e39 not found: ID does not exist" Sep 17 23:40:01.809252 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.809216 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7"] Sep 17 23:40:01.811826 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:01.811805 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d92920-openshift-default-58b85f54c8-fdjd7"] Sep 17 23:40:03.293847 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:03.293819 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc656650-7bb4-46ee-9b96-55cc6cc7f124" path="/var/lib/kubelet/pods/dc656650-7bb4-46ee-9b96-55cc6cc7f124/volumes" Sep 17 23:40:03.736771 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:03.736745 2582 scope.go:117] "RemoveContainer" containerID="4e0510d469b57870ceae88b5c8efca3d537cbe299a4ac24c824f5facc3f3f6cd" Sep 17 23:40:03.745325 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:03.745301 2582 scope.go:117] "RemoveContainer" containerID="7f60e4b95230295425dd5eda8374a29a302c9ff228afe0e0091556a955f58e7f" Sep 17 23:40:03.752927 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:03.752910 2582 scope.go:117] "RemoveContainer" containerID="6fbd1ed76b03dfe2cb2df21675a2de8840668d7e1cb3b850863f5e16fd6f3f5d" Sep 17 23:40:03.760070 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:40:03.760052 2582 scope.go:117] "RemoveContainer" containerID="c0545a4429d0c0a621b447c6e4651461cb862f85609d111a5f4667afc65c730e" Sep 17 23:41:00.598787 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.598751 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7cd595f459-5vmsz"] Sep 17 23:41:00.599300 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.599157 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ed809b5-b370-4deb-8b01-69513dabaab3" containerName="manager" Sep 17 23:41:00.599300 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.599168 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ed809b5-b370-4deb-8b01-69513dabaab3" containerName="manager" Sep 17 23:41:00.599300 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.599181 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e55b8e5-8a26-43c2-8de9-b783996a50bf" containerName="revoke-keys" Sep 17 23:41:00.599300 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.599186 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e55b8e5-8a26-43c2-8de9-b783996a50bf" containerName="revoke-keys" Sep 17 23:41:00.599300 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.599192 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc656650-7bb4-46ee-9b96-55cc6cc7f124" containerName="istio-proxy" Sep 17 23:41:00.599300 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.599197 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc656650-7bb4-46ee-9b96-55cc6cc7f124" containerName="istio-proxy" Sep 17 23:41:00.599300 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.599205 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc656650-7bb4-46ee-9b96-55cc6cc7f124" containerName="istio-proxy" Sep 17 23:41:00.599300 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.599210 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc656650-7bb4-46ee-9b96-55cc6cc7f124" containerName="istio-proxy" Sep 17 23:41:00.599300 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.599220 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2e7b030-1b71-4ce5-8232-1845609394a2" containerName="istio-proxy" Sep 17 23:41:00.599300 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.599226 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2e7b030-1b71-4ce5-8232-1845609394a2" containerName="istio-proxy" Sep 17 23:41:00.599300 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.599288 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2e7b030-1b71-4ce5-8232-1845609394a2" containerName="istio-proxy" Sep 17 23:41:00.599300 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.599295 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc656650-7bb4-46ee-9b96-55cc6cc7f124" containerName="istio-proxy" Sep 17 23:41:00.599300 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.599301 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e55b8e5-8a26-43c2-8de9-b783996a50bf" containerName="revoke-keys" Sep 17 23:41:00.599300 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.599309 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ed809b5-b370-4deb-8b01-69513dabaab3" containerName="manager" Sep 17 23:41:00.600020 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.599316 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc656650-7bb4-46ee-9b96-55cc6cc7f124" containerName="istio-proxy" Sep 17 23:41:00.602620 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.602601 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd595f459-5vmsz" Sep 17 23:41:00.605745 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.605709 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 17 23:41:00.605863 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.605730 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 17 23:41:00.605863 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.605761 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2mntz\"" Sep 17 23:41:00.605863 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.605784 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 17 23:41:00.612291 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.612267 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7cd595f459-5vmsz"] Sep 17 23:41:00.690096 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.690067 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3250289b-4542-4d15-abbf-4a0a7296329b-webhook-cert\") pod \"maas-controller-7cd595f459-5vmsz\" (UID: \"3250289b-4542-4d15-abbf-4a0a7296329b\") " pod="opendatahub/maas-controller-7cd595f459-5vmsz" Sep 17 23:41:00.690243 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.690101 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3250289b-4542-4d15-abbf-4a0a7296329b-metrics-certs\") pod \"maas-controller-7cd595f459-5vmsz\" (UID: \"3250289b-4542-4d15-abbf-4a0a7296329b\") " pod="opendatahub/maas-controller-7cd595f459-5vmsz" Sep 17 23:41:00.690243 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.690144 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p6fc\" (UniqueName: \"kubernetes.io/projected/3250289b-4542-4d15-abbf-4a0a7296329b-kube-api-access-8p6fc\") pod \"maas-controller-7cd595f459-5vmsz\" (UID: \"3250289b-4542-4d15-abbf-4a0a7296329b\") " pod="opendatahub/maas-controller-7cd595f459-5vmsz" Sep 17 23:41:00.791274 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.791245 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3250289b-4542-4d15-abbf-4a0a7296329b-webhook-cert\") pod \"maas-controller-7cd595f459-5vmsz\" (UID: \"3250289b-4542-4d15-abbf-4a0a7296329b\") " pod="opendatahub/maas-controller-7cd595f459-5vmsz" Sep 17 23:41:00.791413 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.791281 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3250289b-4542-4d15-abbf-4a0a7296329b-metrics-certs\") pod \"maas-controller-7cd595f459-5vmsz\" (UID: \"3250289b-4542-4d15-abbf-4a0a7296329b\") " pod="opendatahub/maas-controller-7cd595f459-5vmsz" Sep 17 23:41:00.791413 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.791339 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p6fc\" (UniqueName: \"kubernetes.io/projected/3250289b-4542-4d15-abbf-4a0a7296329b-kube-api-access-8p6fc\") pod \"maas-controller-7cd595f459-5vmsz\" (UID: \"3250289b-4542-4d15-abbf-4a0a7296329b\") " pod="opendatahub/maas-controller-7cd595f459-5vmsz" Sep 17 23:41:00.793708 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.793685 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3250289b-4542-4d15-abbf-4a0a7296329b-metrics-certs\") pod \"maas-controller-7cd595f459-5vmsz\" (UID: \"3250289b-4542-4d15-abbf-4a0a7296329b\") " pod="opendatahub/maas-controller-7cd595f459-5vmsz" Sep 17 23:41:00.793799 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.793737 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3250289b-4542-4d15-abbf-4a0a7296329b-webhook-cert\") pod \"maas-controller-7cd595f459-5vmsz\" (UID: \"3250289b-4542-4d15-abbf-4a0a7296329b\") " pod="opendatahub/maas-controller-7cd595f459-5vmsz" Sep 17 23:41:00.798669 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.798643 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p6fc\" (UniqueName: \"kubernetes.io/projected/3250289b-4542-4d15-abbf-4a0a7296329b-kube-api-access-8p6fc\") pod \"maas-controller-7cd595f459-5vmsz\" (UID: \"3250289b-4542-4d15-abbf-4a0a7296329b\") " pod="opendatahub/maas-controller-7cd595f459-5vmsz" Sep 17 23:41:00.914259 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:00.914189 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd595f459-5vmsz" Sep 17 23:41:01.184285 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:01.184256 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7cd595f459-5vmsz"] Sep 17 23:41:01.186091 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:41:01.186066 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3250289b_4542_4d15_abbf_4a0a7296329b.slice/crio-a40a2555026b88b0eab0c9f19f0135db159579b0be185724af81381c0f8e127f WatchSource:0}: Error finding container a40a2555026b88b0eab0c9f19f0135db159579b0be185724af81381c0f8e127f: Status 404 returned error can't find the container with id a40a2555026b88b0eab0c9f19f0135db159579b0be185724af81381c0f8e127f Sep 17 23:41:01.187400 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:01.187379 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 23:41:02.099843 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:02.099806 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd595f459-5vmsz" event={"ID":"3250289b-4542-4d15-abbf-4a0a7296329b","Type":"ContainerStarted","Data":"f4cbcc49fc4bf58faa644dd9c1be33a2517464ef285f690d7b59899ca4e90f75"} Sep 17 23:41:02.100228 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:02.099861 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd595f459-5vmsz" event={"ID":"3250289b-4542-4d15-abbf-4a0a7296329b","Type":"ContainerStarted","Data":"a40a2555026b88b0eab0c9f19f0135db159579b0be185724af81381c0f8e127f"} Sep 17 23:41:02.100228 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:02.099907 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7cd595f459-5vmsz" Sep 17 23:41:02.116643 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:02.116590 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7cd595f459-5vmsz" podStartSLOduration=1.7141641189999999 podStartE2EDuration="2.116574165s" totalImagesPullingTime="402.410046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:41:02.114470383 +0000 UTC m=+2101.048377518" watchObservedRunningTime="2026-09-17 23:41:02.116574165 +0000 UTC m=+2101.050481327" Sep 17 23:41:13.107358 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:13.107326 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7cd595f459-5vmsz" Sep 17 23:41:22.473471 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:22.473437 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7cd595f459-5vmsz"] Sep 17 23:41:22.473898 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:22.473664 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7cd595f459-5vmsz" podUID="3250289b-4542-4d15-abbf-4a0a7296329b" containerName="manager" containerID="cri-o://f4cbcc49fc4bf58faa644dd9c1be33a2517464ef285f690d7b59899ca4e90f75" gracePeriod=30 Sep 17 23:41:22.721720 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:22.721697 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd595f459-5vmsz" Sep 17 23:41:22.780720 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:22.780622 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3250289b-4542-4d15-abbf-4a0a7296329b-webhook-cert\") pod \"3250289b-4542-4d15-abbf-4a0a7296329b\" (UID: \"3250289b-4542-4d15-abbf-4a0a7296329b\") " Sep 17 23:41:22.780720 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:22.780674 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3250289b-4542-4d15-abbf-4a0a7296329b-metrics-certs\") pod \"3250289b-4542-4d15-abbf-4a0a7296329b\" (UID: \"3250289b-4542-4d15-abbf-4a0a7296329b\") " Sep 17 23:41:22.780945 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:22.780738 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p6fc\" (UniqueName: \"kubernetes.io/projected/3250289b-4542-4d15-abbf-4a0a7296329b-kube-api-access-8p6fc\") pod \"3250289b-4542-4d15-abbf-4a0a7296329b\" (UID: \"3250289b-4542-4d15-abbf-4a0a7296329b\") " Sep 17 23:41:22.782964 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:22.782938 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3250289b-4542-4d15-abbf-4a0a7296329b-kube-api-access-8p6fc" (OuterVolumeSpecName: "kube-api-access-8p6fc") pod "3250289b-4542-4d15-abbf-4a0a7296329b" (UID: "3250289b-4542-4d15-abbf-4a0a7296329b"). InnerVolumeSpecName "kube-api-access-8p6fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:41:22.783208 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:22.783190 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3250289b-4542-4d15-abbf-4a0a7296329b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3250289b-4542-4d15-abbf-4a0a7296329b" (UID: "3250289b-4542-4d15-abbf-4a0a7296329b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:41:22.783208 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:22.783202 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3250289b-4542-4d15-abbf-4a0a7296329b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3250289b-4542-4d15-abbf-4a0a7296329b" (UID: "3250289b-4542-4d15-abbf-4a0a7296329b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:41:22.881982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:22.881951 2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3250289b-4542-4d15-abbf-4a0a7296329b-webhook-cert\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:41:22.881982 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:22.881979 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3250289b-4542-4d15-abbf-4a0a7296329b-metrics-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:41:22.882195 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:22.881991 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p6fc\" (UniqueName: \"kubernetes.io/projected/3250289b-4542-4d15-abbf-4a0a7296329b-kube-api-access-8p6fc\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:41:23.218427 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:23.218326 2582 generic.go:356] "Generic (PLEG): container finished" podID="3250289b-4542-4d15-abbf-4a0a7296329b" containerID="f4cbcc49fc4bf58faa644dd9c1be33a2517464ef285f690d7b59899ca4e90f75" exitCode=0 Sep 17 23:41:23.218427 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:23.218383 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd595f459-5vmsz" Sep 17 23:41:23.218703 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:23.218429 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd595f459-5vmsz" event={"ID":"3250289b-4542-4d15-abbf-4a0a7296329b","Type":"ContainerDied","Data":"f4cbcc49fc4bf58faa644dd9c1be33a2517464ef285f690d7b59899ca4e90f75"} Sep 17 23:41:23.218703 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:23.218544 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd595f459-5vmsz" event={"ID":"3250289b-4542-4d15-abbf-4a0a7296329b","Type":"ContainerDied","Data":"a40a2555026b88b0eab0c9f19f0135db159579b0be185724af81381c0f8e127f"} Sep 17 23:41:23.218703 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:23.218579 2582 scope.go:117] "RemoveContainer" containerID="f4cbcc49fc4bf58faa644dd9c1be33a2517464ef285f690d7b59899ca4e90f75" Sep 17 23:41:23.228435 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:23.228417 2582 scope.go:117] "RemoveContainer" containerID="f4cbcc49fc4bf58faa644dd9c1be33a2517464ef285f690d7b59899ca4e90f75" Sep 17 23:41:23.228742 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:41:23.228718 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4cbcc49fc4bf58faa644dd9c1be33a2517464ef285f690d7b59899ca4e90f75\": container with ID starting with f4cbcc49fc4bf58faa644dd9c1be33a2517464ef285f690d7b59899ca4e90f75 not found: ID does not exist" containerID="f4cbcc49fc4bf58faa644dd9c1be33a2517464ef285f690d7b59899ca4e90f75" Sep 17 23:41:23.228844 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:23.228749 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4cbcc49fc4bf58faa644dd9c1be33a2517464ef285f690d7b59899ca4e90f75"} err="failed to get container status \"f4cbcc49fc4bf58faa644dd9c1be33a2517464ef285f690d7b59899ca4e90f75\": rpc error: code = NotFound desc = could not find container \"f4cbcc49fc4bf58faa644dd9c1be33a2517464ef285f690d7b59899ca4e90f75\": container with ID starting with f4cbcc49fc4bf58faa644dd9c1be33a2517464ef285f690d7b59899ca4e90f75 not found: ID does not exist" Sep 17 23:41:23.239638 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:23.239613 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7cd595f459-5vmsz"] Sep 17 23:41:23.241450 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:23.241429 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7cd595f459-5vmsz"] Sep 17 23:41:23.294412 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:23.294380 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3250289b-4542-4d15-abbf-4a0a7296329b" path="/var/lib/kubelet/pods/3250289b-4542-4d15-abbf-4a0a7296329b/volumes" Sep 17 23:41:24.030877 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.030843 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7cd595f459-qk6cw"] Sep 17 23:41:24.031248 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.031234 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2e7b030-1b71-4ce5-8232-1845609394a2" containerName="istio-proxy" Sep 17 23:41:24.031248 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.031245 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2e7b030-1b71-4ce5-8232-1845609394a2" containerName="istio-proxy" Sep 17 23:41:24.031319 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.031259 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3250289b-4542-4d15-abbf-4a0a7296329b" containerName="manager" Sep 17 23:41:24.031319 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.031264 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3250289b-4542-4d15-abbf-4a0a7296329b" containerName="manager" Sep 17 23:41:24.031319 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.031317 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3250289b-4542-4d15-abbf-4a0a7296329b" containerName="manager" Sep 17 23:41:24.031416 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.031327 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2e7b030-1b71-4ce5-8232-1845609394a2" containerName="istio-proxy" Sep 17 23:41:24.035890 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.035869 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd595f459-qk6cw" Sep 17 23:41:24.039091 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.039069 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 17 23:41:24.039091 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.039079 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2mntz\"" Sep 17 23:41:24.039248 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.039131 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 17 23:41:24.039248 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.039133 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 17 23:41:24.046592 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.046570 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7cd595f459-qk6cw"] Sep 17 23:41:24.092882 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.092847 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67fb385a-b198-4de6-968b-9e9996d869f6-webhook-cert\") pod \"maas-controller-7cd595f459-qk6cw\" (UID: \"67fb385a-b198-4de6-968b-9e9996d869f6\") " pod="opendatahub/maas-controller-7cd595f459-qk6cw" Sep 17 23:41:24.093030 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.092903 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67fb385a-b198-4de6-968b-9e9996d869f6-metrics-certs\") pod \"maas-controller-7cd595f459-qk6cw\" (UID: \"67fb385a-b198-4de6-968b-9e9996d869f6\") " pod="opendatahub/maas-controller-7cd595f459-qk6cw" Sep 17 23:41:24.093030 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.093020 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdblf\" (UniqueName: \"kubernetes.io/projected/67fb385a-b198-4de6-968b-9e9996d869f6-kube-api-access-kdblf\") pod \"maas-controller-7cd595f459-qk6cw\" (UID: \"67fb385a-b198-4de6-968b-9e9996d869f6\") " pod="opendatahub/maas-controller-7cd595f459-qk6cw" Sep 17 23:41:24.194265 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.194237 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdblf\" (UniqueName: \"kubernetes.io/projected/67fb385a-b198-4de6-968b-9e9996d869f6-kube-api-access-kdblf\") pod \"maas-controller-7cd595f459-qk6cw\" (UID: \"67fb385a-b198-4de6-968b-9e9996d869f6\") " pod="opendatahub/maas-controller-7cd595f459-qk6cw" Sep 17 23:41:24.194398 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.194278 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67fb385a-b198-4de6-968b-9e9996d869f6-webhook-cert\") pod \"maas-controller-7cd595f459-qk6cw\" (UID: \"67fb385a-b198-4de6-968b-9e9996d869f6\") " pod="opendatahub/maas-controller-7cd595f459-qk6cw" Sep 17 23:41:24.194398 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.194327 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67fb385a-b198-4de6-968b-9e9996d869f6-metrics-certs\") pod \"maas-controller-7cd595f459-qk6cw\" (UID: \"67fb385a-b198-4de6-968b-9e9996d869f6\") " pod="opendatahub/maas-controller-7cd595f459-qk6cw" Sep 17 23:41:24.196732 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.196708 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67fb385a-b198-4de6-968b-9e9996d869f6-metrics-certs\") pod \"maas-controller-7cd595f459-qk6cw\" (UID: \"67fb385a-b198-4de6-968b-9e9996d869f6\") " pod="opendatahub/maas-controller-7cd595f459-qk6cw" Sep 17 23:41:24.196834 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.196777 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67fb385a-b198-4de6-968b-9e9996d869f6-webhook-cert\") pod \"maas-controller-7cd595f459-qk6cw\" (UID: \"67fb385a-b198-4de6-968b-9e9996d869f6\") " pod="opendatahub/maas-controller-7cd595f459-qk6cw" Sep 17 23:41:24.201605 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.201584 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdblf\" (UniqueName: \"kubernetes.io/projected/67fb385a-b198-4de6-968b-9e9996d869f6-kube-api-access-kdblf\") pod \"maas-controller-7cd595f459-qk6cw\" (UID: \"67fb385a-b198-4de6-968b-9e9996d869f6\") " pod="opendatahub/maas-controller-7cd595f459-qk6cw" Sep 17 23:41:24.347057 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.346984 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd595f459-qk6cw" Sep 17 23:41:24.618515 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:24.618317 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7cd595f459-qk6cw"] Sep 17 23:41:24.621430 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:41:24.621405 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fb385a_b198_4de6_968b_9e9996d869f6.slice/crio-320cfbabdcef6128f253ff38c478fcd855dcafce539b7f6d5efd7851653a300d WatchSource:0}: Error finding container 320cfbabdcef6128f253ff38c478fcd855dcafce539b7f6d5efd7851653a300d: Status 404 returned error can't find the container with id 320cfbabdcef6128f253ff38c478fcd855dcafce539b7f6d5efd7851653a300d Sep 17 23:41:25.232136 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:25.232099 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd595f459-qk6cw" event={"ID":"67fb385a-b198-4de6-968b-9e9996d869f6","Type":"ContainerStarted","Data":"38b77527585dad9089ca9ac2bac52d5116e8309992b2f5622ed88ebc2a3bd45d"} Sep 17 23:41:25.232537 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:25.232163 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd595f459-qk6cw" event={"ID":"67fb385a-b198-4de6-968b-9e9996d869f6","Type":"ContainerStarted","Data":"320cfbabdcef6128f253ff38c478fcd855dcafce539b7f6d5efd7851653a300d"} Sep 17 23:41:25.232537 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:25.232272 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7cd595f459-qk6cw" Sep 17 23:41:25.249871 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:25.249823 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7cd595f459-qk6cw" podStartSLOduration=0.938063345 podStartE2EDuration="1.2498081s" totalImagesPullingTime="311.744755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:41:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:41:25.247272621 +0000 UTC m=+2124.181179757" watchObservedRunningTime="2026-09-17 23:41:25.2498081 +0000 UTC m=+2124.183715234" Sep 17 23:41:36.238831 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:41:36.238793 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7cd595f459-qk6cw" Sep 17 23:42:51.794169 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:42:51.794125 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg"] Sep 17 23:42:51.796799 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:42:51.794396 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg" podUID="55bc62a6-86f7-46e3-8208-03132e1d11ac" containerName="manager" containerID="cri-o://db0078821fabacdbf3d1c99ef48d52af57c2440a6cfa9f7630a0ae100e025520" gracePeriod=10 Sep 17 23:42:52.700641 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:42:52.700612 2582 generic.go:356] "Generic (PLEG): container finished" podID="55bc62a6-86f7-46e3-8208-03132e1d11ac" containerID="db0078821fabacdbf3d1c99ef48d52af57c2440a6cfa9f7630a0ae100e025520" exitCode=0 Sep 17 23:42:52.700785 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:42:52.700680 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg" event={"ID":"55bc62a6-86f7-46e3-8208-03132e1d11ac","Type":"ContainerDied","Data":"db0078821fabacdbf3d1c99ef48d52af57c2440a6cfa9f7630a0ae100e025520"} Sep 17 23:42:52.715350 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:42:52.715327 2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55bc62a6_86f7_46e3_8208_03132e1d11ac.slice/crio-conmon-db0078821fabacdbf3d1c99ef48d52af57c2440a6cfa9f7630a0ae100e025520.scope\": RecentStats: unable to find data in memory cache]" Sep 17 23:42:53.245476 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:42:53.245452 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg" Sep 17 23:42:53.277093 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:42:53.277063 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55bc62a6-86f7-46e3-8208-03132e1d11ac-extensions-socket-volume\") pod \"55bc62a6-86f7-46e3-8208-03132e1d11ac\" (UID: \"55bc62a6-86f7-46e3-8208-03132e1d11ac\") " Sep 17 23:42:53.277279 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:42:53.277169 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nrwl\" (UniqueName: \"kubernetes.io/projected/55bc62a6-86f7-46e3-8208-03132e1d11ac-kube-api-access-6nrwl\") pod \"55bc62a6-86f7-46e3-8208-03132e1d11ac\" (UID: \"55bc62a6-86f7-46e3-8208-03132e1d11ac\") " Sep 17 23:42:53.277549 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:42:53.277515 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55bc62a6-86f7-46e3-8208-03132e1d11ac-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "55bc62a6-86f7-46e3-8208-03132e1d11ac" (UID: "55bc62a6-86f7-46e3-8208-03132e1d11ac"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 23:42:53.279262 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:42:53.279234 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55bc62a6-86f7-46e3-8208-03132e1d11ac-kube-api-access-6nrwl" (OuterVolumeSpecName: "kube-api-access-6nrwl") pod "55bc62a6-86f7-46e3-8208-03132e1d11ac" (UID: "55bc62a6-86f7-46e3-8208-03132e1d11ac"). InnerVolumeSpecName "kube-api-access-6nrwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:42:53.378493 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:42:53.378405 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nrwl\" (UniqueName: \"kubernetes.io/projected/55bc62a6-86f7-46e3-8208-03132e1d11ac-kube-api-access-6nrwl\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:42:53.378493 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:42:53.378436 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55bc62a6-86f7-46e3-8208-03132e1d11ac-extensions-socket-volume\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:42:53.707369 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:42:53.707292 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg" Sep 17 23:42:53.707369 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:42:53.707326 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg" event={"ID":"55bc62a6-86f7-46e3-8208-03132e1d11ac","Type":"ContainerDied","Data":"7259dd5405c75803da19c7649728e375f91f8f0e10427ad6859124cd3390bfdb"} Sep 17 23:42:53.707585 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:42:53.707390 2582 scope.go:117] "RemoveContainer" containerID="db0078821fabacdbf3d1c99ef48d52af57c2440a6cfa9f7630a0ae100e025520" Sep 17 23:42:53.724342 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:42:53.724314 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg"] Sep 17 23:42:53.726266 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:42:53.726246 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kd6gg"] Sep 17 23:42:55.294310 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:42:55.294280 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55bc62a6-86f7-46e3-8208-03132e1d11ac" path="/var/lib/kubelet/pods/55bc62a6-86f7-46e3-8208-03132e1d11ac/volumes" Sep 17 23:43:52.490935 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:43:52.490899 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xp2rh"] Sep 17 23:43:52.491696 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:43:52.491465 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55bc62a6-86f7-46e3-8208-03132e1d11ac" containerName="manager" Sep 17 23:43:52.491696 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:43:52.491486 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="55bc62a6-86f7-46e3-8208-03132e1d11ac" containerName="manager" Sep 17 23:43:52.491696 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:43:52.491647 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="55bc62a6-86f7-46e3-8208-03132e1d11ac" containerName="manager" Sep 17 23:43:52.495993 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:43:52.495968 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xp2rh" Sep 17 23:43:52.499126 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:43:52.499097 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hgccr\"" Sep 17 23:43:52.508452 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:43:52.508428 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xp2rh"] Sep 17 23:43:52.574955 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:43:52.574930 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67jfd\" (UniqueName: \"kubernetes.io/projected/0ed809f6-cca9-4ce3-8deb-de6c514b51d0-kube-api-access-67jfd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xp2rh\" (UID: \"0ed809f6-cca9-4ce3-8deb-de6c514b51d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xp2rh" Sep 17 23:43:52.575091 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:43:52.574973 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ed809f6-cca9-4ce3-8deb-de6c514b51d0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xp2rh\" (UID: \"0ed809f6-cca9-4ce3-8deb-de6c514b51d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xp2rh" Sep 17 23:43:52.675399 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:43:52.675369 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67jfd\" (UniqueName: \"kubernetes.io/projected/0ed809f6-cca9-4ce3-8deb-de6c514b51d0-kube-api-access-67jfd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xp2rh\" (UID: \"0ed809f6-cca9-4ce3-8deb-de6c514b51d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xp2rh" Sep 17 23:43:52.675521 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:43:52.675419 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ed809f6-cca9-4ce3-8deb-de6c514b51d0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xp2rh\" (UID: \"0ed809f6-cca9-4ce3-8deb-de6c514b51d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xp2rh" Sep 17 23:43:52.675781 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:43:52.675765 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ed809f6-cca9-4ce3-8deb-de6c514b51d0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xp2rh\" (UID: \"0ed809f6-cca9-4ce3-8deb-de6c514b51d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xp2rh" Sep 17 23:43:52.687042 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:43:52.687013 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67jfd\" (UniqueName: \"kubernetes.io/projected/0ed809f6-cca9-4ce3-8deb-de6c514b51d0-kube-api-access-67jfd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xp2rh\" (UID: \"0ed809f6-cca9-4ce3-8deb-de6c514b51d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xp2rh" Sep 17 23:43:52.810137 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:43:52.810074 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xp2rh" Sep 17 23:43:53.112558 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:43:53.112529 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xp2rh"] Sep 17 23:43:53.114193 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:43:53.114166 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed809f6_cca9_4ce3_8deb_de6c514b51d0.slice/crio-dc50846fba1835308fdd4af18c8bdd38f603013e696c898b248ded9352877368 WatchSource:0}: Error finding container dc50846fba1835308fdd4af18c8bdd38f603013e696c898b248ded9352877368: Status 404 returned error can't find the container with id dc50846fba1835308fdd4af18c8bdd38f603013e696c898b248ded9352877368 Sep 17 23:43:54.029561 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:43:54.029530 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xp2rh" event={"ID":"0ed809f6-cca9-4ce3-8deb-de6c514b51d0","Type":"ContainerStarted","Data":"b6b238c3a3eb85554c1c7ab0c5fa132af92fc7620c28884d5daf40eb72e22648"} Sep 17 23:43:54.029955 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:43:54.029580 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xp2rh" event={"ID":"0ed809f6-cca9-4ce3-8deb-de6c514b51d0","Type":"ContainerStarted","Data":"dc50846fba1835308fdd4af18c8bdd38f603013e696c898b248ded9352877368"} Sep 17 23:43:54.029955 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:43:54.029620 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xp2rh" Sep 17 23:43:54.049241 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:43:54.049162 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xp2rh" podStartSLOduration=2.049135421 podStartE2EDuration="2.049135421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:43:54.047098347 +0000 UTC m=+2272.981005483" watchObservedRunningTime="2026-09-17 23:43:54.049135421 +0000 UTC m=+2272.983042558" Sep 17 23:44:03.852209 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:44:03.852174 2582 scope.go:117] "RemoveContainer" containerID="a7067b19e39734d0c677f605c808bb1703a1865ecb7c7493122b5877b615b3c2" Sep 17 23:44:03.860865 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:44:03.860847 2582 scope.go:117] "RemoveContainer" containerID="0e5cdab91c482192db0896c05c3f5b808689e141a640283245ef672f610198a2" Sep 17 23:44:05.037304 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:44:05.037272 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xp2rh" Sep 17 23:45:00.133006 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:00.132968 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828145-v7nbz"] Sep 17 23:45:00.136649 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:00.136630 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828145-v7nbz" Sep 17 23:45:00.138947 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:00.138920 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rdxvj\"" Sep 17 23:45:00.142998 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:00.142972 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828145-v7nbz"] Sep 17 23:45:00.149830 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:00.149807 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828145-v7nbz" Sep 17 23:45:00.428149 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:00.428077 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828145-v7nbz"] Sep 17 23:45:00.431402 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:45:00.431370 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0873c6dd_aaf2_4e6f_9607_c02d4da976c9.slice/crio-d06fc4f55e63160edb118195127f62ccaf2040d88de7e1766a4f586b44caea4f WatchSource:0}: Error finding container d06fc4f55e63160edb118195127f62ccaf2040d88de7e1766a4f586b44caea4f: Status 404 returned error can't find the container with id d06fc4f55e63160edb118195127f62ccaf2040d88de7e1766a4f586b44caea4f Sep 17 23:45:01.387556 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:01.387523 2582 generic.go:356] "Generic (PLEG): container finished" podID="0873c6dd-aaf2-4e6f-9607-c02d4da976c9" containerID="cf612a8389df60041ebc36f8543f8b3b51d808d0d6323f89f39fcf13ef46f516" exitCode=0 Sep 17 23:45:01.387942 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:01.387599 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828145-v7nbz" event={"ID":"0873c6dd-aaf2-4e6f-9607-c02d4da976c9","Type":"ContainerDied","Data":"cf612a8389df60041ebc36f8543f8b3b51d808d0d6323f89f39fcf13ef46f516"} Sep 17 23:45:01.387942 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:01.387653 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828145-v7nbz" event={"ID":"0873c6dd-aaf2-4e6f-9607-c02d4da976c9","Type":"ContainerStarted","Data":"d06fc4f55e63160edb118195127f62ccaf2040d88de7e1766a4f586b44caea4f"} Sep 17 23:45:02.525394 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:02.525367 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828145-v7nbz" Sep 17 23:45:03.399493 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:03.399460 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828145-v7nbz" event={"ID":"0873c6dd-aaf2-4e6f-9607-c02d4da976c9","Type":"ContainerDied","Data":"d06fc4f55e63160edb118195127f62ccaf2040d88de7e1766a4f586b44caea4f"} Sep 17 23:45:03.399493 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:03.399482 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828145-v7nbz" Sep 17 23:45:03.399759 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:03.399530 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d06fc4f55e63160edb118195127f62ccaf2040d88de7e1766a4f586b44caea4f" Sep 17 23:45:40.555950 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.555913 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2"] Sep 17 23:45:40.556512 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.556434 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0873c6dd-aaf2-4e6f-9607-c02d4da976c9" containerName="cleanup" Sep 17 23:45:40.556512 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.556448 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0873c6dd-aaf2-4e6f-9607-c02d4da976c9" containerName="cleanup" Sep 17 23:45:40.556609 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.556534 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0873c6dd-aaf2-4e6f-9607-c02d4da976c9" containerName="cleanup" Sep 17 23:45:40.559902 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.559887 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" Sep 17 23:45:40.563257 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.563210 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 23:45:40.563257 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.563255 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-skqc5\"" Sep 17 23:45:40.563492 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.563220 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 17 23:45:40.569621 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.569600 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2"] Sep 17 23:45:40.655551 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.655494 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx4nb\" (UniqueName: \"kubernetes.io/projected/de7e6f48-e58c-4301-a7c9-099efec3db2a-kube-api-access-rx4nb\") pod \"maas-api-6678c8db5f-p4xq2\" (UID: \"de7e6f48-e58c-4301-a7c9-099efec3db2a\") " pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" Sep 17 23:45:40.655711 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.655622 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de7e6f48-e58c-4301-a7c9-099efec3db2a-metrics-certs\") pod \"maas-api-6678c8db5f-p4xq2\" (UID: \"de7e6f48-e58c-4301-a7c9-099efec3db2a\") " pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" Sep 17 23:45:40.655711 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.655664 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7e6f48-e58c-4301-a7c9-099efec3db2a-openshift-service-ca\") pod \"maas-api-6678c8db5f-p4xq2\" (UID: \"de7e6f48-e58c-4301-a7c9-099efec3db2a\") " pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" Sep 17 23:45:40.655711 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.655705 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de7e6f48-e58c-4301-a7c9-099efec3db2a-maas-api-tls\") pod \"maas-api-6678c8db5f-p4xq2\" (UID: \"de7e6f48-e58c-4301-a7c9-099efec3db2a\") " pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" Sep 17 23:45:40.756733 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.756692 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de7e6f48-e58c-4301-a7c9-099efec3db2a-metrics-certs\") pod \"maas-api-6678c8db5f-p4xq2\" (UID: \"de7e6f48-e58c-4301-a7c9-099efec3db2a\") " pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" Sep 17 23:45:40.756910 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.756761 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7e6f48-e58c-4301-a7c9-099efec3db2a-openshift-service-ca\") pod \"maas-api-6678c8db5f-p4xq2\" (UID: \"de7e6f48-e58c-4301-a7c9-099efec3db2a\") " pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" Sep 17 23:45:40.756910 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.756803 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de7e6f48-e58c-4301-a7c9-099efec3db2a-maas-api-tls\") pod \"maas-api-6678c8db5f-p4xq2\" (UID: \"de7e6f48-e58c-4301-a7c9-099efec3db2a\") " pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" Sep 17 23:45:40.756910 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.756894 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx4nb\" (UniqueName: \"kubernetes.io/projected/de7e6f48-e58c-4301-a7c9-099efec3db2a-kube-api-access-rx4nb\") pod \"maas-api-6678c8db5f-p4xq2\" (UID: \"de7e6f48-e58c-4301-a7c9-099efec3db2a\") " pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" Sep 17 23:45:40.757729 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.757696 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7e6f48-e58c-4301-a7c9-099efec3db2a-openshift-service-ca\") pod \"maas-api-6678c8db5f-p4xq2\" (UID: \"de7e6f48-e58c-4301-a7c9-099efec3db2a\") " pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" Sep 17 23:45:40.759682 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.759653 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de7e6f48-e58c-4301-a7c9-099efec3db2a-maas-api-tls\") pod \"maas-api-6678c8db5f-p4xq2\" (UID: \"de7e6f48-e58c-4301-a7c9-099efec3db2a\") " pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" Sep 17 23:45:40.760063 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.760041 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de7e6f48-e58c-4301-a7c9-099efec3db2a-metrics-certs\") pod \"maas-api-6678c8db5f-p4xq2\" (UID: \"de7e6f48-e58c-4301-a7c9-099efec3db2a\") " pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" Sep 17 23:45:40.765424 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.765396 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx4nb\" (UniqueName: \"kubernetes.io/projected/de7e6f48-e58c-4301-a7c9-099efec3db2a-kube-api-access-rx4nb\") pod \"maas-api-6678c8db5f-p4xq2\" (UID: \"de7e6f48-e58c-4301-a7c9-099efec3db2a\") " pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" Sep 17 23:45:40.872736 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:40.872616 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" Sep 17 23:45:41.152339 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:41.152312 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2"] Sep 17 23:45:41.154190 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:45:41.154166 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7e6f48_e58c_4301_a7c9_099efec3db2a.slice/crio-e2999416edc02f87e72f7e67ef060c7f509c0fa714c60c355b46dba9841365cc WatchSource:0}: Error finding container e2999416edc02f87e72f7e67ef060c7f509c0fa714c60c355b46dba9841365cc: Status 404 returned error can't find the container with id e2999416edc02f87e72f7e67ef060c7f509c0fa714c60c355b46dba9841365cc Sep 17 23:45:41.602444 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:41.602413 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" event={"ID":"de7e6f48-e58c-4301-a7c9-099efec3db2a","Type":"ContainerStarted","Data":"e2999416edc02f87e72f7e67ef060c7f509c0fa714c60c355b46dba9841365cc"} Sep 17 23:45:43.615807 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:43.615766 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" event={"ID":"de7e6f48-e58c-4301-a7c9-099efec3db2a","Type":"ContainerStarted","Data":"9528721fc73088317f8ea318a8c785c87b73ad0a9375052d2199b938f0113ee9"} Sep 17 23:45:43.616253 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:43.615826 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" Sep 17 23:45:43.632226 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:43.632181 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" podStartSLOduration=2.066565649 podStartE2EDuration="3.632166182s" totalImagesPullingTime="1.565600533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:45:43.63083709 +0000 UTC m=+2382.564744239" watchObservedRunningTime="2026-09-17 23:45:43.632166182 +0000 UTC m=+2382.566073319" Sep 17 23:45:48.376786 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.376752 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2"] Sep 17 23:45:48.377257 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.377020 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" podUID="de7e6f48-e58c-4301-a7c9-099efec3db2a" containerName="maas-api" containerID="cri-o://9528721fc73088317f8ea318a8c785c87b73ad0a9375052d2199b938f0113ee9" gracePeriod=30 Sep 17 23:45:48.382988 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.382967 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" Sep 17 23:45:48.411887 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.411853 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59cb547587-vkpcr"] Sep 17 23:45:48.417543 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.417154 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59cb547587-vkpcr" Sep 17 23:45:48.427856 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.427831 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59cb547587-vkpcr"] Sep 17 23:45:48.528984 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.528948 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9bj7\" (UniqueName: \"kubernetes.io/projected/61bf0b6b-92bc-4d21-8bc9-754069feca4f-kube-api-access-r9bj7\") pod \"maas-api-59cb547587-vkpcr\" (UID: \"61bf0b6b-92bc-4d21-8bc9-754069feca4f\") " pod="odh-ai-gateway-infra/maas-api-59cb547587-vkpcr" Sep 17 23:45:48.529155 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.529009 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61bf0b6b-92bc-4d21-8bc9-754069feca4f-maas-api-tls\") pod \"maas-api-59cb547587-vkpcr\" (UID: \"61bf0b6b-92bc-4d21-8bc9-754069feca4f\") " pod="odh-ai-gateway-infra/maas-api-59cb547587-vkpcr" Sep 17 23:45:48.529155 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.529038 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61bf0b6b-92bc-4d21-8bc9-754069feca4f-openshift-service-ca\") pod \"maas-api-59cb547587-vkpcr\" (UID: \"61bf0b6b-92bc-4d21-8bc9-754069feca4f\") " pod="odh-ai-gateway-infra/maas-api-59cb547587-vkpcr" Sep 17 23:45:48.529155 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.529084 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61bf0b6b-92bc-4d21-8bc9-754069feca4f-metrics-certs\") pod \"maas-api-59cb547587-vkpcr\" (UID: \"61bf0b6b-92bc-4d21-8bc9-754069feca4f\") " pod="odh-ai-gateway-infra/maas-api-59cb547587-vkpcr" Sep 17 23:45:48.629701 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.629637 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" Sep 17 23:45:48.629872 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.629843 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61bf0b6b-92bc-4d21-8bc9-754069feca4f-maas-api-tls\") pod \"maas-api-59cb547587-vkpcr\" (UID: \"61bf0b6b-92bc-4d21-8bc9-754069feca4f\") " pod="odh-ai-gateway-infra/maas-api-59cb547587-vkpcr" Sep 17 23:45:48.629950 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.629892 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61bf0b6b-92bc-4d21-8bc9-754069feca4f-openshift-service-ca\") pod \"maas-api-59cb547587-vkpcr\" (UID: \"61bf0b6b-92bc-4d21-8bc9-754069feca4f\") " pod="odh-ai-gateway-infra/maas-api-59cb547587-vkpcr" Sep 17 23:45:48.629950 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.629933 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61bf0b6b-92bc-4d21-8bc9-754069feca4f-metrics-certs\") pod \"maas-api-59cb547587-vkpcr\" (UID: \"61bf0b6b-92bc-4d21-8bc9-754069feca4f\") " pod="odh-ai-gateway-infra/maas-api-59cb547587-vkpcr" Sep 17 23:45:48.630078 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.630066 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9bj7\" (UniqueName: \"kubernetes.io/projected/61bf0b6b-92bc-4d21-8bc9-754069feca4f-kube-api-access-r9bj7\") pod \"maas-api-59cb547587-vkpcr\" (UID: \"61bf0b6b-92bc-4d21-8bc9-754069feca4f\") " pod="odh-ai-gateway-infra/maas-api-59cb547587-vkpcr" Sep 17 23:45:48.630556 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.630491 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61bf0b6b-92bc-4d21-8bc9-754069feca4f-openshift-service-ca\") pod \"maas-api-59cb547587-vkpcr\" (UID: \"61bf0b6b-92bc-4d21-8bc9-754069feca4f\") " pod="odh-ai-gateway-infra/maas-api-59cb547587-vkpcr" Sep 17 23:45:48.632270 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.632239 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61bf0b6b-92bc-4d21-8bc9-754069feca4f-maas-api-tls\") pod \"maas-api-59cb547587-vkpcr\" (UID: \"61bf0b6b-92bc-4d21-8bc9-754069feca4f\") " pod="odh-ai-gateway-infra/maas-api-59cb547587-vkpcr" Sep 17 23:45:48.632371 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.632309 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61bf0b6b-92bc-4d21-8bc9-754069feca4f-metrics-certs\") pod \"maas-api-59cb547587-vkpcr\" (UID: \"61bf0b6b-92bc-4d21-8bc9-754069feca4f\") " pod="odh-ai-gateway-infra/maas-api-59cb547587-vkpcr" Sep 17 23:45:48.637484 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.637466 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9bj7\" (UniqueName: \"kubernetes.io/projected/61bf0b6b-92bc-4d21-8bc9-754069feca4f-kube-api-access-r9bj7\") pod \"maas-api-59cb547587-vkpcr\" (UID: \"61bf0b6b-92bc-4d21-8bc9-754069feca4f\") " pod="odh-ai-gateway-infra/maas-api-59cb547587-vkpcr" Sep 17 23:45:48.646415 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.646383 2582 generic.go:356] "Generic (PLEG): container finished" podID="de7e6f48-e58c-4301-a7c9-099efec3db2a" containerID="9528721fc73088317f8ea318a8c785c87b73ad0a9375052d2199b938f0113ee9" exitCode=0 Sep 17 23:45:48.647181 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.646486 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" event={"ID":"de7e6f48-e58c-4301-a7c9-099efec3db2a","Type":"ContainerDied","Data":"9528721fc73088317f8ea318a8c785c87b73ad0a9375052d2199b938f0113ee9"} Sep 17 23:45:48.647181 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.646574 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" event={"ID":"de7e6f48-e58c-4301-a7c9-099efec3db2a","Type":"ContainerDied","Data":"e2999416edc02f87e72f7e67ef060c7f509c0fa714c60c355b46dba9841365cc"} Sep 17 23:45:48.647181 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.646622 2582 scope.go:117] "RemoveContainer" containerID="9528721fc73088317f8ea318a8c785c87b73ad0a9375052d2199b938f0113ee9" Sep 17 23:45:48.647181 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.646786 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2" Sep 17 23:45:48.657330 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.657310 2582 scope.go:117] "RemoveContainer" containerID="9528721fc73088317f8ea318a8c785c87b73ad0a9375052d2199b938f0113ee9" Sep 17 23:45:48.657748 ip-10-0-143-82 kubenswrapper[2582]: E0917 23:45:48.657715 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9528721fc73088317f8ea318a8c785c87b73ad0a9375052d2199b938f0113ee9\": container with ID starting with 9528721fc73088317f8ea318a8c785c87b73ad0a9375052d2199b938f0113ee9 not found: ID does not exist" containerID="9528721fc73088317f8ea318a8c785c87b73ad0a9375052d2199b938f0113ee9" Sep 17 23:45:48.657748 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.657742 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9528721fc73088317f8ea318a8c785c87b73ad0a9375052d2199b938f0113ee9"} err="failed to get container status \"9528721fc73088317f8ea318a8c785c87b73ad0a9375052d2199b938f0113ee9\": rpc error: code = NotFound desc = could not find container \"9528721fc73088317f8ea318a8c785c87b73ad0a9375052d2199b938f0113ee9\": container with ID starting with 9528721fc73088317f8ea318a8c785c87b73ad0a9375052d2199b938f0113ee9 not found: ID does not exist" Sep 17 23:45:48.730560 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.730532 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de7e6f48-e58c-4301-a7c9-099efec3db2a-metrics-certs\") pod \"de7e6f48-e58c-4301-a7c9-099efec3db2a\" (UID: \"de7e6f48-e58c-4301-a7c9-099efec3db2a\") " Sep 17 23:45:48.730794 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.730590 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rx4nb\" (UniqueName: \"kubernetes.io/projected/de7e6f48-e58c-4301-a7c9-099efec3db2a-kube-api-access-rx4nb\") pod \"de7e6f48-e58c-4301-a7c9-099efec3db2a\" (UID: \"de7e6f48-e58c-4301-a7c9-099efec3db2a\") " Sep 17 23:45:48.730794 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.730626 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7e6f48-e58c-4301-a7c9-099efec3db2a-openshift-service-ca\") pod \"de7e6f48-e58c-4301-a7c9-099efec3db2a\" (UID: \"de7e6f48-e58c-4301-a7c9-099efec3db2a\") " Sep 17 23:45:48.730794 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.730685 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de7e6f48-e58c-4301-a7c9-099efec3db2a-maas-api-tls\") pod \"de7e6f48-e58c-4301-a7c9-099efec3db2a\" (UID: \"de7e6f48-e58c-4301-a7c9-099efec3db2a\") " Sep 17 23:45:48.731081 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.731059 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7e6f48-e58c-4301-a7c9-099efec3db2a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "de7e6f48-e58c-4301-a7c9-099efec3db2a" (UID: "de7e6f48-e58c-4301-a7c9-099efec3db2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 23:45:48.732916 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.732883 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de7e6f48-e58c-4301-a7c9-099efec3db2a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "de7e6f48-e58c-4301-a7c9-099efec3db2a" (UID: "de7e6f48-e58c-4301-a7c9-099efec3db2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:45:48.732916 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.732902 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de7e6f48-e58c-4301-a7c9-099efec3db2a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "de7e6f48-e58c-4301-a7c9-099efec3db2a" (UID: "de7e6f48-e58c-4301-a7c9-099efec3db2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 23:45:48.733090 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.733005 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7e6f48-e58c-4301-a7c9-099efec3db2a-kube-api-access-rx4nb" (OuterVolumeSpecName: "kube-api-access-rx4nb") pod "de7e6f48-e58c-4301-a7c9-099efec3db2a" (UID: "de7e6f48-e58c-4301-a7c9-099efec3db2a"). InnerVolumeSpecName "kube-api-access-rx4nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 23:45:48.757931 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.757913 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59cb547587-vkpcr" Sep 17 23:45:48.831703 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.831673 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de7e6f48-e58c-4301-a7c9-099efec3db2a-metrics-certs\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:45:48.831836 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.831706 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx4nb\" (UniqueName: \"kubernetes.io/projected/de7e6f48-e58c-4301-a7c9-099efec3db2a-kube-api-access-rx4nb\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:45:48.831836 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.831720 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7e6f48-e58c-4301-a7c9-099efec3db2a-openshift-service-ca\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:45:48.831836 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.831736 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de7e6f48-e58c-4301-a7c9-099efec3db2a-maas-api-tls\") on node \"ip-10-0-143-82.ec2.internal\" DevicePath \"\"" Sep 17 23:45:48.968031 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.968001 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2"] Sep 17 23:45:48.976241 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:48.976205 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6678c8db5f-p4xq2"] Sep 17 23:45:49.038862 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:49.038840 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59cb547587-vkpcr"] Sep 17 23:45:49.040637 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:45:49.040607 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61bf0b6b_92bc_4d21_8bc9_754069feca4f.slice/crio-0c3aee8ec08f869beda1d9c524d5d8c54524d6c2164a0824e5cf19b3c1f57703 WatchSource:0}: Error finding container 0c3aee8ec08f869beda1d9c524d5d8c54524d6c2164a0824e5cf19b3c1f57703: Status 404 returned error can't find the container with id 0c3aee8ec08f869beda1d9c524d5d8c54524d6c2164a0824e5cf19b3c1f57703 Sep 17 23:45:49.293209 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:49.293180 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de7e6f48-e58c-4301-a7c9-099efec3db2a" path="/var/lib/kubelet/pods/de7e6f48-e58c-4301-a7c9-099efec3db2a/volumes" Sep 17 23:45:49.654064 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:49.653980 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59cb547587-vkpcr" event={"ID":"61bf0b6b-92bc-4d21-8bc9-754069feca4f","Type":"ContainerStarted","Data":"f2538517109d2a863bfa0a1648a22c6351d1efea9c5f0110f9a8cc12d74f67ed"} Sep 17 23:45:49.654064 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:49.654030 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59cb547587-vkpcr" event={"ID":"61bf0b6b-92bc-4d21-8bc9-754069feca4f","Type":"ContainerStarted","Data":"0c3aee8ec08f869beda1d9c524d5d8c54524d6c2164a0824e5cf19b3c1f57703"} Sep 17 23:45:49.654064 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:49.654058 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59cb547587-vkpcr" Sep 17 23:45:49.672212 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:49.672160 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59cb547587-vkpcr" podStartSLOduration=1.362248087 podStartE2EDuration="1.672140041s" totalImagesPullingTime="309.891954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:45:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:45:49.668992779 +0000 UTC m=+2388.602899915" watchObservedRunningTime="2026-09-17 23:45:49.672140041 +0000 UTC m=+2388.606047177" Sep 17 23:45:55.664788 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:45:55.664761 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59cb547587-vkpcr" Sep 17 23:46:03.896668 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:03.896631 2582 scope.go:117] "RemoveContainer" containerID="af62723302db9b0a2cb9cd59cbcd9e4e0ad4ccffe91560595fc87fa439339b29" Sep 17 23:46:03.907070 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:03.907049 2582 scope.go:117] "RemoveContainer" containerID="951e4e2300700142899e9a8e769e9b16490adeddfc2ce5b95d6d0df13cf78ea0" Sep 17 23:46:03.916201 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:03.916179 2582 scope.go:117] "RemoveContainer" containerID="5766cd2b021260a3a3a1eb52a509140cd43eb8fd53d4305fdb50e765baabefd0" Sep 17 23:46:10.561625 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:10.561593 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-76b64f6977-5hc4l_cc430d25-8f1f-4a68-a409-91c5768c9746/authorino/0.log" Sep 17 23:46:16.236095 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:16.236022 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9gdrq_dad3084b-e040-483e-9c23-171a0573347c/manager/0.log" Sep 17 23:46:16.359214 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:16.359185 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-lw5wt_dd805404-17a7-41b6-9972-b3570a138562/manager/0.log" Sep 17 23:46:16.476089 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:16.476059 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7cd595f459-qk6cw_67fb385a-b198-4de6-968b-9e9996d869f6/manager/0.log" Sep 17 23:46:16.596559 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:16.596439 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-k7l6b_7c004f0a-2484-466f-8a7d-1f34096f8080/server/0.log" Sep 17 23:46:17.111022 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:17.110984 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-747fdf4bf5-xl5hw_60f5593a-58fb-4ff3-9df6-7b9a43eb49e2/manager/0.log" Sep 17 23:46:17.116152 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:17.116134 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-747fdf4bf5-xl5hw_60f5593a-58fb-4ff3-9df6-7b9a43eb49e2/copy-manifests/0.log" Sep 17 23:46:17.584298 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:17.584273 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-59cb547587-vkpcr_61bf0b6b-92bc-4d21-8bc9-754069feca4f/maas-api/0.log" Sep 17 23:46:17.696853 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:17.696821 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29828130-k86l8_59306d16-3b7d-44be-8a9a-360276bdedb0/cleanup/0.log" Sep 17 23:46:17.812847 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:17.812819 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29828145-v7nbz_0873c6dd-aaf2-4e6f-9607-c02d4da976c9/cleanup/0.log" Sep 17 23:46:17.929707 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:17.929625 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-z5h2z_402d4843-f631-4d80-b8ec-c5999f751f53/postgres/0.log" Sep 17 23:46:18.716739 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:18.716709 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-76b64f6977-5hc4l_cc430d25-8f1f-4a68-a409-91c5768c9746/authorino/0.log" Sep 17 23:46:18.834301 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:18.834277 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-ztvbq_241122b8-292b-4710-96f6-8d0419e36c88/manager/0.log" Sep 17 23:46:18.948168 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:18.948143 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-pqwsr_853502d1-76ef-4606-a89d-57cde294e9fc/manager/0.log" Sep 17 23:46:19.186754 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:19.186730 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-xp2rh_0ed809f6-cca9-4ce3-8deb-de6c514b51d0/manager/0.log" Sep 17 23:46:19.442788 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:19.442727 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-96vl8_00282dde-9ace-4275-bc1e-08edf677ef20/manager/0.log" Sep 17 23:46:20.150049 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:20.150028 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d4c67b97b-lcbjm_f3db4a6f-b531-4ebe-bcd6-f56239784525/kube-auth-proxy/0.log" Sep 17 23:46:20.267241 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:20.267209 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-b66bc_7b3c4835-32b2-46a3-8d2d-f64071d509fd/istio-proxy/1.log" Sep 17 23:46:20.508879 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:20.508851 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-pxpjd_b6287a30-1142-48b6-a7ea-50b426e7b598/payload-processing/0.log" Sep 17 23:46:21.107120 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:21.107090 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-kkfxc_c3a82f97-c23a-4088-8baa-a95e6ef32ecb/main/0.log" Sep 17 23:46:27.825131 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:27.825096 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kg5sm/must-gather-z8s26"] Sep 17 23:46:27.825527 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:27.825490 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de7e6f48-e58c-4301-a7c9-099efec3db2a" containerName="maas-api" Sep 17 23:46:27.825527 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:27.825525 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="de7e6f48-e58c-4301-a7c9-099efec3db2a" containerName="maas-api" Sep 17 23:46:27.825617 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:27.825597 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="de7e6f48-e58c-4301-a7c9-099efec3db2a" containerName="maas-api" Sep 17 23:46:27.831031 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:27.831012 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kg5sm/must-gather-z8s26" Sep 17 23:46:27.833217 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:27.833196 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kg5sm\"/\"openshift-service-ca.crt\"" Sep 17 23:46:27.834122 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:27.834100 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kg5sm\"/\"default-dockercfg-hwp27\"" Sep 17 23:46:27.834444 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:27.834413 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kg5sm\"/\"kube-root-ca.crt\"" Sep 17 23:46:27.837534 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:27.837509 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kg5sm/must-gather-z8s26"] Sep 17 23:46:27.951340 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:27.951310 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f8e2da3-0566-43e3-a36c-81c4d500ab4e-must-gather-output\") pod \"must-gather-z8s26\" (UID: \"0f8e2da3-0566-43e3-a36c-81c4d500ab4e\") " pod="openshift-must-gather-kg5sm/must-gather-z8s26" Sep 17 23:46:27.951522 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:27.951359 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbkmf\" (UniqueName: \"kubernetes.io/projected/0f8e2da3-0566-43e3-a36c-81c4d500ab4e-kube-api-access-vbkmf\") pod \"must-gather-z8s26\" (UID: \"0f8e2da3-0566-43e3-a36c-81c4d500ab4e\") " pod="openshift-must-gather-kg5sm/must-gather-z8s26" Sep 17 23:46:28.052598 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:28.052569 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f8e2da3-0566-43e3-a36c-81c4d500ab4e-must-gather-output\") pod \"must-gather-z8s26\" (UID: \"0f8e2da3-0566-43e3-a36c-81c4d500ab4e\") " pod="openshift-must-gather-kg5sm/must-gather-z8s26" Sep 17 23:46:28.052732 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:28.052616 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbkmf\" (UniqueName: \"kubernetes.io/projected/0f8e2da3-0566-43e3-a36c-81c4d500ab4e-kube-api-access-vbkmf\") pod \"must-gather-z8s26\" (UID: \"0f8e2da3-0566-43e3-a36c-81c4d500ab4e\") " pod="openshift-must-gather-kg5sm/must-gather-z8s26" Sep 17 23:46:28.052891 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:28.052872 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f8e2da3-0566-43e3-a36c-81c4d500ab4e-must-gather-output\") pod \"must-gather-z8s26\" (UID: \"0f8e2da3-0566-43e3-a36c-81c4d500ab4e\") " pod="openshift-must-gather-kg5sm/must-gather-z8s26" Sep 17 23:46:28.060266 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:28.060240 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbkmf\" (UniqueName: \"kubernetes.io/projected/0f8e2da3-0566-43e3-a36c-81c4d500ab4e-kube-api-access-vbkmf\") pod \"must-gather-z8s26\" (UID: \"0f8e2da3-0566-43e3-a36c-81c4d500ab4e\") " pod="openshift-must-gather-kg5sm/must-gather-z8s26" Sep 17 23:46:28.141638 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:28.141583 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kg5sm/must-gather-z8s26" Sep 17 23:46:28.421603 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:28.421524 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kg5sm/must-gather-z8s26"] Sep 17 23:46:28.425933 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:46:28.425908 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f8e2da3_0566_43e3_a36c_81c4d500ab4e.slice/crio-62c900eb02e091a00e5e3061412a1a0ea2a61586a56da6767a6ba05d7dbf7ede WatchSource:0}: Error finding container 62c900eb02e091a00e5e3061412a1a0ea2a61586a56da6767a6ba05d7dbf7ede: Status 404 returned error can't find the container with id 62c900eb02e091a00e5e3061412a1a0ea2a61586a56da6767a6ba05d7dbf7ede Sep 17 23:46:28.427859 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:28.427844 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 23:46:28.855726 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:28.855690 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kg5sm/must-gather-z8s26" event={"ID":"0f8e2da3-0566-43e3-a36c-81c4d500ab4e","Type":"ContainerStarted","Data":"62c900eb02e091a00e5e3061412a1a0ea2a61586a56da6767a6ba05d7dbf7ede"} Sep 17 23:46:29.865655 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:29.865610 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kg5sm/must-gather-z8s26" event={"ID":"0f8e2da3-0566-43e3-a36c-81c4d500ab4e","Type":"ContainerStarted","Data":"613f0cfcd1dcfc8343418097478375ae0e911de3bd1ff17e5d37f958e80b5636"} Sep 17 23:46:29.866115 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:29.865671 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kg5sm/must-gather-z8s26" event={"ID":"0f8e2da3-0566-43e3-a36c-81c4d500ab4e","Type":"ContainerStarted","Data":"f9537c84683351a1a9883c8bc71eaf5cee80576a88b4e8c261283b6faf4e1140"} Sep 17 23:46:29.883255 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:29.883195 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kg5sm/must-gather-z8s26" podStartSLOduration=2.092982875 podStartE2EDuration="2.883175457s" totalImagesPullingTime="790.192582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:46:29.88036697 +0000 UTC m=+2428.814274106" watchObservedRunningTime="2026-09-17 23:46:29.883175457 +0000 UTC m=+2428.817082594" Sep 17 23:46:30.793239 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:30.793212 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-drknm_66bdcd7b-3b69-4bdd-9579-9e035e9d19b0/global-pull-secret-syncer/0.log" Sep 17 23:46:30.895281 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:30.895248 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-svq9k_d4518bc1-148d-4005-a3ec-2b36297214fa/konnectivity-agent/0.log" Sep 17 23:46:30.961145 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:30.961117 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-82.ec2.internal_9b0067a4916205e46911412e53db8082/haproxy/0.log" Sep 17 23:46:35.378945 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:35.378405 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-76b64f6977-5hc4l_cc430d25-8f1f-4a68-a409-91c5768c9746/authorino/0.log" Sep 17 23:46:35.427326 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:35.427291 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-ztvbq_241122b8-292b-4710-96f6-8d0419e36c88/manager/0.log" Sep 17 23:46:35.455923 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:35.455867 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-pqwsr_853502d1-76ef-4606-a89d-57cde294e9fc/manager/0.log" Sep 17 23:46:35.518788 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:35.518696 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-xp2rh_0ed809f6-cca9-4ce3-8deb-de6c514b51d0/manager/0.log" Sep 17 23:46:35.612181 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:35.612145 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-96vl8_00282dde-9ace-4275-bc1e-08edf677ef20/manager/0.log" Sep 17 23:46:37.292231 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:37.292172 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-t9g4f_e6c946e8-c084-434e-b49d-7ab905a4be36/kube-state-metrics/0.log" Sep 17 23:46:37.312712 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:37.312652 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-t9g4f_e6c946e8-c084-434e-b49d-7ab905a4be36/kube-rbac-proxy-main/0.log" Sep 17 23:46:37.336665 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:37.336630 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-t9g4f_e6c946e8-c084-434e-b49d-7ab905a4be36/kube-rbac-proxy-self/0.log" Sep 17 23:46:37.363439 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:37.363411 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7b56dd7c9d-4rmh7_1ee2d368-c317-40c4-a001-46103a67a613/metrics-server/0.log" Sep 17 23:46:37.633077 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:37.632994 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-trds7_7971320f-7694-45b5-8fe4-964eb59d081e/node-exporter/0.log" Sep 17 23:46:37.655490 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:37.655462 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-trds7_7971320f-7694-45b5-8fe4-964eb59d081e/kube-rbac-proxy/0.log" Sep 17 23:46:37.685838 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:37.685782 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-trds7_7971320f-7694-45b5-8fe4-964eb59d081e/init-textfile/0.log" Sep 17 23:46:38.116831 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:38.116802 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7858b594b4-bqbbp_da235c63-916e-40bd-ab57-ffc8d26f12f9/thanos-query/0.log" Sep 17 23:46:38.138414 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:38.138367 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7858b594b4-bqbbp_da235c63-916e-40bd-ab57-ffc8d26f12f9/kube-rbac-proxy-web/0.log" Sep 17 23:46:38.163823 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:38.163778 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7858b594b4-bqbbp_da235c63-916e-40bd-ab57-ffc8d26f12f9/kube-rbac-proxy/0.log" Sep 17 23:46:38.186727 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:38.186698 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7858b594b4-bqbbp_da235c63-916e-40bd-ab57-ffc8d26f12f9/prom-label-proxy/0.log" Sep 17 23:46:38.208036 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:38.208006 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7858b594b4-bqbbp_da235c63-916e-40bd-ab57-ffc8d26f12f9/kube-rbac-proxy-rules/0.log" Sep 17 23:46:38.231406 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:38.231374 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7858b594b4-bqbbp_da235c63-916e-40bd-ab57-ffc8d26f12f9/kube-rbac-proxy-metrics/0.log" Sep 17 23:46:40.621875 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.621840 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7"] Sep 17 23:46:40.632821 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.632793 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7"] Sep 17 23:46:40.633039 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.632889 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" Sep 17 23:46:40.705094 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.705040 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spmlb\" (UniqueName: \"kubernetes.io/projected/290a2885-848a-4db6-ae92-fcc3e27cff51-kube-api-access-spmlb\") pod \"perf-node-gather-daemonset-m9wv7\" (UID: \"290a2885-848a-4db6-ae92-fcc3e27cff51\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" Sep 17 23:46:40.705417 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.705393 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/290a2885-848a-4db6-ae92-fcc3e27cff51-lib-modules\") pod \"perf-node-gather-daemonset-m9wv7\" (UID: \"290a2885-848a-4db6-ae92-fcc3e27cff51\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" Sep 17 23:46:40.705636 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.705617 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/290a2885-848a-4db6-ae92-fcc3e27cff51-proc\") pod \"perf-node-gather-daemonset-m9wv7\" (UID: \"290a2885-848a-4db6-ae92-fcc3e27cff51\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" Sep 17 23:46:40.705781 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.705766 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/290a2885-848a-4db6-ae92-fcc3e27cff51-podres\") pod \"perf-node-gather-daemonset-m9wv7\" (UID: \"290a2885-848a-4db6-ae92-fcc3e27cff51\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" Sep 17 23:46:40.705903 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.705889 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/290a2885-848a-4db6-ae92-fcc3e27cff51-sys\") pod \"perf-node-gather-daemonset-m9wv7\" (UID: \"290a2885-848a-4db6-ae92-fcc3e27cff51\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" Sep 17 23:46:40.730766 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.730737 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6bdf65cc46-s9l7f_64798c7a-5f3a-44f1-8aec-d4c676708015/console/0.log" Sep 17 23:46:40.765525 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.765467 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-gjxw7_19bd3f81-d2c5-4cfd-bb84-c3608eaaa14b/download-server/0.log" Sep 17 23:46:40.806673 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.806630 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/290a2885-848a-4db6-ae92-fcc3e27cff51-proc\") pod \"perf-node-gather-daemonset-m9wv7\" (UID: \"290a2885-848a-4db6-ae92-fcc3e27cff51\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" Sep 17 23:46:40.806865 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.806741 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/290a2885-848a-4db6-ae92-fcc3e27cff51-proc\") pod \"perf-node-gather-daemonset-m9wv7\" (UID: \"290a2885-848a-4db6-ae92-fcc3e27cff51\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" Sep 17 23:46:40.806865 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.806743 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/290a2885-848a-4db6-ae92-fcc3e27cff51-podres\") pod \"perf-node-gather-daemonset-m9wv7\" (UID: \"290a2885-848a-4db6-ae92-fcc3e27cff51\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" Sep 17 23:46:40.806865 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.806819 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/290a2885-848a-4db6-ae92-fcc3e27cff51-sys\") pod \"perf-node-gather-daemonset-m9wv7\" (UID: \"290a2885-848a-4db6-ae92-fcc3e27cff51\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" Sep 17 23:46:40.806865 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.806857 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/290a2885-848a-4db6-ae92-fcc3e27cff51-podres\") pod \"perf-node-gather-daemonset-m9wv7\" (UID: \"290a2885-848a-4db6-ae92-fcc3e27cff51\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" Sep 17 23:46:40.807086 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.806915 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/290a2885-848a-4db6-ae92-fcc3e27cff51-sys\") pod \"perf-node-gather-daemonset-m9wv7\" (UID: \"290a2885-848a-4db6-ae92-fcc3e27cff51\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" Sep 17 23:46:40.807086 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.806924 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spmlb\" (UniqueName: \"kubernetes.io/projected/290a2885-848a-4db6-ae92-fcc3e27cff51-kube-api-access-spmlb\") pod \"perf-node-gather-daemonset-m9wv7\" (UID: \"290a2885-848a-4db6-ae92-fcc3e27cff51\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" Sep 17 23:46:40.807086 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.806966 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/290a2885-848a-4db6-ae92-fcc3e27cff51-lib-modules\") pod \"perf-node-gather-daemonset-m9wv7\" (UID: \"290a2885-848a-4db6-ae92-fcc3e27cff51\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" Sep 17 23:46:40.807261 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.807130 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/290a2885-848a-4db6-ae92-fcc3e27cff51-lib-modules\") pod \"perf-node-gather-daemonset-m9wv7\" (UID: \"290a2885-848a-4db6-ae92-fcc3e27cff51\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" Sep 17 23:46:40.815517 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.815475 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spmlb\" (UniqueName: \"kubernetes.io/projected/290a2885-848a-4db6-ae92-fcc3e27cff51-kube-api-access-spmlb\") pod \"perf-node-gather-daemonset-m9wv7\" (UID: \"290a2885-848a-4db6-ae92-fcc3e27cff51\") " pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" Sep 17 23:46:40.950684 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:40.950589 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" Sep 17 23:46:41.240736 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:41.238664 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7"] Sep 17 23:46:41.241820 ip-10-0-143-82 kubenswrapper[2582]: W0917 23:46:41.241788 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod290a2885_848a_4db6_ae92_fcc3e27cff51.slice/crio-d2bfb186f861988c15e8f99b8f53c27f82b1adcc06091e88e68347d90cb2f082 WatchSource:0}: Error finding container d2bfb186f861988c15e8f99b8f53c27f82b1adcc06091e88e68347d90cb2f082: Status 404 returned error can't find the container with id d2bfb186f861988c15e8f99b8f53c27f82b1adcc06091e88e68347d90cb2f082 Sep 17 23:46:41.437740 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:41.437711 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-jwjpz_c7efbb43-0384-4fed-ac1c-4f8664a12e72/volume-data-source-validator/0.log" Sep 17 23:46:41.960700 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:41.960665 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" event={"ID":"290a2885-848a-4db6-ae92-fcc3e27cff51","Type":"ContainerStarted","Data":"0a9880d3847096b909431db3d0354b0c37808d84138066895f771ab8a630fbed"} Sep 17 23:46:41.961192 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:41.960720 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" event={"ID":"290a2885-848a-4db6-ae92-fcc3e27cff51","Type":"ContainerStarted","Data":"d2bfb186f861988c15e8f99b8f53c27f82b1adcc06091e88e68347d90cb2f082"} Sep 17 23:46:41.961192 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:41.960771 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" Sep 17 23:46:41.976904 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:41.976845 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" podStartSLOduration=1.976825582 podStartE2EDuration="1.976825582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:46:41.974406468 +0000 UTC m=+2440.908313604" watchObservedRunningTime="2026-09-17 23:46:41.976825582 +0000 UTC m=+2440.910732718" Sep 17 23:46:42.533120 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:42.533094 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pkh9l_5cd06380-e744-4e37-a5f9-3cac49aee5d4/dns/0.log" Sep 17 23:46:42.552902 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:42.552876 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pkh9l_5cd06380-e744-4e37-a5f9-3cac49aee5d4/kube-rbac-proxy/0.log" Sep 17 23:46:42.639155 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:42.639127 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k2bg2_bbda2aea-1bb6-4a24-a74f-fc8d79b07586/dns-node-resolver/0.log" Sep 17 23:46:43.193214 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:43.193180 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-cb7955849-x2x4q_ec6eaa22-50b5-4788-a5e8-b8e783034759/registry/0.log" Sep 17 23:46:43.264081 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:43.264048 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-trs6m_bdbecc0a-d250-434e-8478-2f6f57c094d3/node-ca/0.log" Sep 17 23:46:44.562829 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:44.562806 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d4c67b97b-lcbjm_f3db4a6f-b531-4ebe-bcd6-f56239784525/kube-auth-proxy/0.log" Sep 17 23:46:44.590074 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:44.590044 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-b66bc_7b3c4835-32b2-46a3-8d2d-f64071d509fd/istio-proxy/0.log" Sep 17 23:46:44.593456 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:44.593417 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-b66bc_7b3c4835-32b2-46a3-8d2d-f64071d509fd/istio-proxy/1.log" Sep 17 23:46:44.657272 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:44.657240 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-pxpjd_b6287a30-1142-48b6-a7ea-50b426e7b598/payload-processing/0.log" Sep 17 23:46:45.507233 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:45.507205 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kk5m4_48bdac6b-06a2-4d13-8a48-c0d5b5af03b5/serve-healthcheck-canary/0.log" Sep 17 23:46:46.000820 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:46.000796 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4nj87_2f26392a-99d2-4431-928c-30bf72268393/kube-rbac-proxy/0.log" Sep 17 23:46:46.021320 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:46.021299 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4nj87_2f26392a-99d2-4431-928c-30bf72268393/exporter/0.log" Sep 17 23:46:46.041260 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:46.041235 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4nj87_2f26392a-99d2-4431-928c-30bf72268393/extractor/0.log" Sep 17 23:46:47.985599 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:47.985557 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kg5sm/perf-node-gather-daemonset-m9wv7" Sep 17 23:46:48.238443 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:48.238362 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9gdrq_dad3084b-e040-483e-9c23-171a0573347c/manager/0.log" Sep 17 23:46:48.477523 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:48.477464 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-lw5wt_dd805404-17a7-41b6-9972-b3570a138562/manager/0.log" Sep 17 23:46:48.543259 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:48.543174 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7cd595f459-qk6cw_67fb385a-b198-4de6-968b-9e9996d869f6/manager/0.log" Sep 17 23:46:48.561883 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:48.561858 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-k7l6b_7c004f0a-2484-466f-8a7d-1f34096f8080/server/0.log" Sep 17 23:46:48.834051 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:48.833965 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-747fdf4bf5-xl5hw_60f5593a-58fb-4ff3-9df6-7b9a43eb49e2/manager/0.log" Sep 17 23:46:48.855795 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:48.855770 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-747fdf4bf5-xl5hw_60f5593a-58fb-4ff3-9df6-7b9a43eb49e2/copy-manifests/0.log" Sep 17 23:46:51.185737 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:51.185711 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d4c67b97b-lcbjm_f3db4a6f-b531-4ebe-bcd6-f56239784525/kube-auth-proxy/0.log" Sep 17 23:46:51.203988 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:51.203948 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-b66bc_7b3c4835-32b2-46a3-8d2d-f64071d509fd/istio-proxy/0.log" Sep 17 23:46:51.208279 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:51.208250 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-b66bc_7b3c4835-32b2-46a3-8d2d-f64071d509fd/istio-proxy/1.log" Sep 17 23:46:51.256772 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:51.256737 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-pxpjd_b6287a30-1142-48b6-a7ea-50b426e7b598/payload-processing/0.log" Sep 17 23:46:53.121493 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:53.121446 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d4c67b97b-lcbjm_f3db4a6f-b531-4ebe-bcd6-f56239784525/kube-auth-proxy/0.log" Sep 17 23:46:53.142577 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:53.142548 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-b66bc_7b3c4835-32b2-46a3-8d2d-f64071d509fd/istio-proxy/0.log" Sep 17 23:46:53.146326 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:53.146298 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-b66bc_7b3c4835-32b2-46a3-8d2d-f64071d509fd/istio-proxy/1.log" Sep 17 23:46:53.199988 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:53.199946 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-pxpjd_b6287a30-1142-48b6-a7ea-50b426e7b598/payload-processing/0.log" Sep 17 23:46:55.323127 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:55.323100 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-qkpcn_3a4ed9d0-4126-4045-92c6-d6162c3aa45e/migrator/0.log" Sep 17 23:46:55.342580 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:55.342552 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-qkpcn_3a4ed9d0-4126-4045-92c6-d6162c3aa45e/graceful-termination/0.log" Sep 17 23:46:55.675125 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:55.675044 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-d6px4_9ff60ca3-2533-4ed5-9cc1-56bad886cc5d/kube-storage-version-migrator-operator/1.log" Sep 17 23:46:55.676190 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:55.676167 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-d6px4_9ff60ca3-2533-4ed5-9cc1-56bad886cc5d/kube-storage-version-migrator-operator/0.log" Sep 17 23:46:56.706791 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:56.706761 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqrzv_d4c08617-40c6-47d5-bcc8-e4765ca7a477/kube-multus-additional-cni-plugins/0.log" Sep 17 23:46:56.727547 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:56.727522 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqrzv_d4c08617-40c6-47d5-bcc8-e4765ca7a477/egress-router-binary-copy/0.log" Sep 17 23:46:56.746640 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:56.746621 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqrzv_d4c08617-40c6-47d5-bcc8-e4765ca7a477/cni-plugins/0.log" Sep 17 23:46:56.765186 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:56.765167 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqrzv_d4c08617-40c6-47d5-bcc8-e4765ca7a477/bond-cni-plugin/0.log" Sep 17 23:46:56.784542 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:56.784487 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqrzv_d4c08617-40c6-47d5-bcc8-e4765ca7a477/routeoverride-cni/0.log" Sep 17 23:46:56.803853 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:56.803836 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqrzv_d4c08617-40c6-47d5-bcc8-e4765ca7a477/whereabouts-cni-bincopy/0.log" Sep 17 23:46:56.822952 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:56.822935 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqrzv_d4c08617-40c6-47d5-bcc8-e4765ca7a477/whereabouts-cni/0.log" Sep 17 23:46:57.301201 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:57.301171 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zzcb8_dd5f6c2c-5d7d-4863-aa8d-ab6c172114f6/kube-multus/0.log" Sep 17 23:46:57.416980 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:57.416940 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xg2sz_3752b568-d53f-49ac-9095-0f61cee96767/network-metrics-daemon/0.log" Sep 17 23:46:57.432899 ip-10-0-143-82 kubenswrapper[2582]: I0917 23:46:57.432877 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xg2sz_3752b568-d53f-49ac-9095-0f61cee96767/kube-rbac-proxy/0.log"