Sep 11 16:43:04.474819 ip-10-0-143-165 systemd[1]: Starting Kubernetes Kubelet... Sep 11 16:43:04.875448 ip-10-0-143-165 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 16:43:04.875448 ip-10-0-143-165 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 16:43:04.875448 ip-10-0-143-165 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 16:43:04.875448 ip-10-0-143-165 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 16:43:04.875448 ip-10-0-143-165 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 16:43:04.876949 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.876858 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 16:43:04.884091 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884065 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 16:43:04.884091 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884088 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 16:43:04.884091 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884093 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 16:43:04.884091 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884098 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884104 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884108 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884111 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884114 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884118 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884122 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884125 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884129 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884133 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884136 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884140 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884144 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884148 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884151 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884155 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884159 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884163 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884166 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884174 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 16:43:04.884361 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884179 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884183 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884187 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884192 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884196 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884200 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884204 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884207 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884211 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884214 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884218 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884222 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884226 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884229 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884234 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884238 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884244 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884248 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884253 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884257 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 16:43:04.885206 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884262 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884266 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884271 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884275 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884279 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884283 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884287 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884293 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884298 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884302 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884306 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884310 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884314 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884318 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884322 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884327 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884330 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884335 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884339 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884343 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 16:43:04.886082 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884347 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 16:43:04.886940 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884351 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 16:43:04.886940 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884355 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 16:43:04.886940 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884359 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 16:43:04.886940 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884363 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 16:43:04.886940 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884367 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 16:43:04.886940 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884373 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 16:43:04.886940 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884377 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 16:43:04.886940 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884383 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 16:43:04.886940 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884387 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 16:43:04.886940 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884392 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 16:43:04.886940 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884396 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 16:43:04.886940 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884400 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 16:43:04.886940 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884406 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 16:43:04.886940 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884410 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 16:43:04.886940 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884417 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 16:43:04.886940 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884422 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 16:43:04.886940 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884427 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 16:43:04.886940 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884432 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 16:43:04.886940 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884437 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 16:43:04.887781 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884444 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 16:43:04.887781 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884450 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 16:43:04.887781 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884455 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 16:43:04.887781 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884459 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 16:43:04.887781 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884464 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 16:43:04.887781 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884469 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 16:43:04.887781 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884476 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 16:43:04.887781 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884480 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 16:43:04.887781 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884485 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 16:43:04.887781 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884490 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 16:43:04.887781 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884511 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 16:43:04.887781 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884516 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 16:43:04.887781 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884520 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 16:43:04.887781 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884527 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 16:43:04.887781 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884532 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 16:43:04.887781 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.884537 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 16:43:04.887781 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885758 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 16:43:04.887781 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885769 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 16:43:04.887781 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885776 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 16:43:04.888609 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885781 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 16:43:04.888609 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885786 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 16:43:04.888609 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885790 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 16:43:04.888609 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885795 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 16:43:04.888609 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885799 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 16:43:04.888609 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885803 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 16:43:04.888609 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885807 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 16:43:04.888609 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885811 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 16:43:04.888609 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885816 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 16:43:04.888609 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885820 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 16:43:04.888609 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885824 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 16:43:04.888609 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885831 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 16:43:04.888609 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885836 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 16:43:04.888609 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885841 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 16:43:04.888609 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885855 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 16:43:04.888609 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885862 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 16:43:04.888609 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885868 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 16:43:04.888609 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885873 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 16:43:04.888609 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885877 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 16:43:04.889323 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885882 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 16:43:04.889323 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885886 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 16:43:04.889323 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885891 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 16:43:04.889323 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885895 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 16:43:04.889323 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885908 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 16:43:04.889323 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885912 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 16:43:04.889323 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885916 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 16:43:04.889323 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885921 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 16:43:04.889323 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885925 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 16:43:04.889323 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885929 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 16:43:04.889323 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885934 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 16:43:04.889323 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885939 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 16:43:04.889323 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885943 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 16:43:04.889323 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885948 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 16:43:04.889323 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885952 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 16:43:04.889323 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885956 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 16:43:04.889323 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885960 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 16:43:04.889323 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885964 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 16:43:04.889323 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885969 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885973 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885978 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885982 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885987 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885991 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885996 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.885999 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886004 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886008 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886013 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886017 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886021 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886027 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886034 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886038 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886042 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886047 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886051 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886056 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 16:43:04.889818 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886060 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886065 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886070 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886074 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886078 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886082 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886086 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886091 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886095 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886099 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886103 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886108 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886112 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886116 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886120 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886124 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886129 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886133 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886137 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886141 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 16:43:04.890412 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886146 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886150 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886154 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886159 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886163 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886167 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886171 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886175 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886180 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886184 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886188 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886193 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886197 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886201 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886205 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886210 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886214 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886218 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886223 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886895 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 16:43:04.890960 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886906 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 16:43:04.891637 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886911 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 16:43:04.891637 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886915 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 16:43:04.891637 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886920 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 16:43:04.891637 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886924 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 16:43:04.891637 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886928 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 16:43:04.891637 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886933 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 16:43:04.891637 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886937 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 16:43:04.891637 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886941 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 16:43:04.891637 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886945 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 16:43:04.891637 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886949 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 16:43:04.891637 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886954 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 16:43:04.891637 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886961 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 16:43:04.891637 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886966 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 16:43:04.891637 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886971 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 16:43:04.891637 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886977 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 16:43:04.891637 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886981 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 11 16:43:04.891637 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886985 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 16:43:04.891637 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886990 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 16:43:04.891637 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886994 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.886999 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887003 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887007 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887015 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887019 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887024 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887029 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887033 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887038 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887042 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887046 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887050 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887054 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887059 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887063 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887067 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887071 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887075 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887080 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 16:43:04.892173 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887084 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 16:43:04.892685 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887088 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 16:43:04.892685 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887092 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 16:43:04.892685 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887096 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 16:43:04.892685 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887100 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 16:43:04.892685 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887105 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 16:43:04.892685 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887110 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 16:43:04.892685 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887114 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 16:43:04.892685 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887118 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 16:43:04.892685 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887123 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 16:43:04.892685 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887127 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 16:43:04.892685 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887131 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 16:43:04.892685 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887138 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 16:43:04.892685 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887144 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 16:43:04.892685 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887149 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 16:43:04.892685 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887154 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 16:43:04.892685 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887160 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 16:43:04.892685 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887165 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 16:43:04.892685 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887169 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 16:43:04.892685 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887173 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887178 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887185 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887192 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887196 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887201 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887205 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887209 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887214 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887218 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887222 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887226 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887230 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887235 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887239 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887243 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887247 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887251 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887256 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887260 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 16:43:04.893252 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887264 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887269 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887273 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887279 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887283 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887287 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887292 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887296 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887300 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887306 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887310 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887314 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887318 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887322 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887326 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887331 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887335 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887339 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887344 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.887348 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887463 2577 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 16:43:04.893780 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887536 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887549 2577 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887556 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887563 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887568 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887575 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887581 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887587 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887592 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887597 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887603 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887608 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887613 2577 flags.go:64] FLAG: --cgroup-root="" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887618 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887623 2577 flags.go:64] FLAG: --client-ca-file="" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887634 2577 flags.go:64] FLAG: --cloud-provider="external" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887639 2577 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887645 2577 flags.go:64] FLAG: --cluster-domain="" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887650 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887656 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887664 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887670 2577 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887676 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 16:43:04.894320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887681 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887686 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887692 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887697 2577 flags.go:64] FLAG: --contention-profiling="false" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887701 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887706 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887711 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887716 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887722 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887727 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887732 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887737 2577 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887741 2577 flags.go:64] FLAG: --enable-server="true" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887746 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887754 2577 flags.go:64] FLAG: --event-burst="100" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887759 2577 flags.go:64] FLAG: --event-qps="50" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887765 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887770 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887774 2577 flags.go:64] FLAG: --eviction-hard="" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887781 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887786 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887791 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887795 2577 flags.go:64] FLAG: --eviction-soft="" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887800 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887805 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 16:43:04.894929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887810 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887815 2577 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887820 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887825 2577 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887831 2577 flags.go:64] FLAG: --feature-gates="" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887837 2577 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887842 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887847 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887852 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887857 2577 flags.go:64] FLAG: --healthz-port="10248" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887863 2577 flags.go:64] FLAG: --help="false" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887868 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-143-165.ec2.internal" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887873 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887878 2577 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887882 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887888 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887894 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887899 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887903 2577 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887908 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887913 2577 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887917 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887923 2577 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887928 2577 flags.go:64] FLAG: --kube-reserved="" Sep 11 16:43:04.895587 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887932 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887937 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887944 2577 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887950 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887955 2577 flags.go:64] FLAG: --lock-file="" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887959 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887964 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887969 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887978 2577 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887984 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887989 2577 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.887994 2577 flags.go:64] FLAG: --logging-format="text" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888000 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888005 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888010 2577 flags.go:64] FLAG: --manifest-url="" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888015 2577 flags.go:64] FLAG: --manifest-url-header="" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888023 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888028 2577 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888041 2577 flags.go:64] FLAG: --max-pods="110" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888046 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888051 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888056 2577 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888060 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888065 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888070 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 16:43:04.896263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888075 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888087 2577 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888091 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888096 2577 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888101 2577 flags.go:64] FLAG: --pod-cidr="" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888105 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888114 2577 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888119 2577 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888124 2577 flags.go:64] FLAG: --pods-per-core="0" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888130 2577 flags.go:64] FLAG: --port="10250" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888135 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888140 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0af27ee0e90f2bf34" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888145 2577 flags.go:64] FLAG: --qos-reserved="" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888150 2577 flags.go:64] FLAG: --read-only-port="10255" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888155 2577 flags.go:64] FLAG: --register-node="true" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888160 2577 flags.go:64] FLAG: --register-with-taints="" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888167 2577 flags.go:64] FLAG: --registry-burst="10" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888172 2577 flags.go:64] FLAG: --registry-qps="5" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888177 2577 flags.go:64] FLAG: --reserved-cpus="" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888181 2577 flags.go:64] FLAG: --reserved-memory="" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888189 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888194 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888199 2577 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888204 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888209 2577 flags.go:64] FLAG: --runonce="false" Sep 11 16:43:04.896918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888214 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888219 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888224 2577 flags.go:64] FLAG: --seccomp-default="false" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888228 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888233 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888238 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888243 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888248 2577 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888253 2577 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888258 2577 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888262 2577 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888267 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888271 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888276 2577 flags.go:64] FLAG: --system-cgroups="" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888281 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888290 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888296 2577 flags.go:64] FLAG: --tls-cert-file="" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888301 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888307 2577 flags.go:64] FLAG: --tls-min-version="" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888312 2577 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888316 2577 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888321 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888326 2577 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888331 2577 flags.go:64] FLAG: --v="2" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888338 2577 flags.go:64] FLAG: --version="false" Sep 11 16:43:04.897603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888345 2577 flags.go:64] FLAG: --vmodule="" Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888354 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.888374 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888574 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888582 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888587 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888591 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888597 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888601 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888606 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888610 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888614 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888618 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888622 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888626 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888630 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888634 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888638 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888643 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888647 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 16:43:04.898262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888651 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888656 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888661 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888668 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888672 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888676 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888682 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888689 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888694 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888698 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888702 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888707 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888711 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888717 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888723 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888727 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888732 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888737 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888742 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888746 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 16:43:04.898761 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888751 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888755 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888760 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888764 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888768 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888773 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888777 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888781 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888785 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888789 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888793 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888797 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888801 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888805 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888810 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888816 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888820 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888824 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888828 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888832 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 16:43:04.899262 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888837 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888844 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888849 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888854 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888859 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888863 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888870 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888875 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888880 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888884 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888888 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888892 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888896 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888901 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888905 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888909 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888913 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888917 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888921 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888926 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 16:43:04.899772 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888930 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 16:43:04.900255 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888934 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 16:43:04.900255 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888938 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 16:43:04.900255 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888942 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 16:43:04.900255 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888946 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 16:43:04.900255 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888950 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 16:43:04.900255 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888954 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 16:43:04.900255 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888960 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 16:43:04.900255 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888964 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 16:43:04.900255 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888968 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 16:43:04.900255 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888972 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 16:43:04.900255 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888978 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 16:43:04.900255 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888984 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 16:43:04.900255 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888989 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 16:43:04.900255 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.888995 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 16:43:04.900255 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.889001 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 16:43:04.900255 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.889005 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 16:43:04.900255 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.889009 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 16:43:04.900255 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.889015 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 16:43:04.900255 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.889029 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 16:43:04.900743 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.889034 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 16:43:04.900743 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.889038 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 16:43:04.900743 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.889966 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 16:43:04.900743 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.897586 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 16:43:04.900743 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.897602 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 16:43:04.900743 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.897623 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 16:43:04.900743 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.897629 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 16:43:04.900743 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897693 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 16:43:04.900743 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897699 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 16:43:04.900743 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897702 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 16:43:04.900743 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897705 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 16:43:04.900743 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897708 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 11 16:43:04.900743 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897711 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 16:43:04.900743 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897714 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 16:43:04.900743 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897717 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897720 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897722 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897725 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897729 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897731 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897734 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897736 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897739 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897742 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897745 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897747 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897750 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897753 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897756 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897758 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897761 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897765 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897768 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897770 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 16:43:04.901113 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897773 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897776 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897779 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897782 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897785 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897788 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897791 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897794 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897797 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897800 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897802 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897805 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897808 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897811 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897815 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897819 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897822 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897825 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897828 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897830 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 16:43:04.901663 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897833 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 16:43:04.902162 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897836 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 16:43:04.902162 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897838 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 16:43:04.902162 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897841 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 16:43:04.902162 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897844 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 16:43:04.902162 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897846 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 16:43:04.902162 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897849 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 16:43:04.902162 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897852 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 16:43:04.902162 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897855 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 16:43:04.902162 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897857 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 16:43:04.902162 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897866 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 16:43:04.902162 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897869 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 16:43:04.902162 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897871 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 16:43:04.902162 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897874 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 16:43:04.902162 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897879 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 16:43:04.902162 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897883 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 16:43:04.902162 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897886 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 16:43:04.902162 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897889 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 16:43:04.902162 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897892 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 16:43:04.902162 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897895 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897898 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897907 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897910 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897913 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897916 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897919 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897921 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897924 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897927 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897929 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897932 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897935 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897937 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897940 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897942 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897947 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897951 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897954 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897957 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 16:43:04.902696 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897960 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 16:43:04.903243 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897963 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 16:43:04.903243 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897966 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 16:43:04.903243 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897969 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 16:43:04.903243 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897977 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 16:43:04.903243 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897980 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 16:43:04.903243 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897983 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 16:43:04.903243 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897986 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 16:43:04.903243 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897989 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 16:43:04.903243 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897991 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 16:43:04.903243 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897994 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 16:43:04.903243 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897997 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 16:43:04.903243 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.897999 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 16:43:04.903243 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.898004 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 16:43:04.903243 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898150 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 16:43:04.903243 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898156 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 16:43:04.903636 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898159 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 16:43:04.903636 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898162 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 16:43:04.903636 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898165 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 16:43:04.903636 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898168 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 16:43:04.903636 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898171 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 16:43:04.903636 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898174 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 16:43:04.903636 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898177 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 16:43:04.903636 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898180 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 16:43:04.903636 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898182 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 16:43:04.903636 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898185 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 16:43:04.903636 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898189 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 16:43:04.903636 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898192 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 16:43:04.903636 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898195 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 16:43:04.903636 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898198 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 16:43:04.903636 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898200 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 16:43:04.903636 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898203 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 16:43:04.903636 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898206 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 16:43:04.903636 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898209 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 16:43:04.903636 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898211 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898214 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898217 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898225 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898228 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898230 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898234 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898236 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898239 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898242 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898244 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898247 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898250 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898252 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898255 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898257 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898260 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898262 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898265 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898268 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 16:43:04.904095 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898271 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898274 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898276 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898279 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898282 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898284 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898287 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898291 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898294 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898297 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898300 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898303 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898306 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898309 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898311 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898314 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898322 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898325 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898327 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898330 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 16:43:04.904595 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898333 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 16:43:04.905084 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898335 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 16:43:04.905084 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898339 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 16:43:04.905084 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898342 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 16:43:04.905084 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898345 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 16:43:04.905084 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898349 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 16:43:04.905084 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898353 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 16:43:04.905084 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898356 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 16:43:04.905084 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898359 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 16:43:04.905084 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898362 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 16:43:04.905084 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898365 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 16:43:04.905084 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898368 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 16:43:04.905084 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898371 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 16:43:04.905084 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898373 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 16:43:04.905084 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898376 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 16:43:04.905084 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898378 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 16:43:04.905084 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898381 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 16:43:04.905084 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898384 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 16:43:04.905084 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898386 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 16:43:04.905084 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898389 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 16:43:04.905569 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898391 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 16:43:04.905569 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898393 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 16:43:04.905569 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898396 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 16:43:04.905569 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898399 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 16:43:04.905569 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898401 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 16:43:04.905569 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898404 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 16:43:04.905569 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898406 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 16:43:04.905569 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898409 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 16:43:04.905569 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898411 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 16:43:04.905569 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898421 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 16:43:04.905569 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898424 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 16:43:04.905569 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898427 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 16:43:04.905569 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898429 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 16:43:04.905569 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898432 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 16:43:04.905569 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898434 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 16:43:04.905569 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898437 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 16:43:04.905569 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898440 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 16:43:04.905569 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898442 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 16:43:04.905569 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:04.898445 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 16:43:04.906039 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.898450 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 16:43:04.906039 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.898601 2577 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 16:43:04.906039 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.900866 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 16:43:04.906039 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.901739 2577 server.go:1019] "Starting client certificate rotation" Sep 11 16:43:04.906039 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.901845 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 16:43:04.906039 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.901879 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 16:43:04.906039 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.904528 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 16:43:04.906225 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.906067 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 16:43:04.915821 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.915802 2577 log.go:25] "Validated CRI v1 runtime API" Sep 11 16:43:04.918856 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.918841 2577 log.go:25] "Validated CRI v1 image API" Sep 11 16:43:04.919833 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.919817 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 16:43:04.923483 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.923463 2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b4291c21-71cb-4a47-b5e0-238849c36047:/dev/nvme0n1p4 c0c9c333-c656-4551-a0ae-de58312599bd:/dev/nvme0n1p3] Sep 11 16:43:04.923562 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.923483 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 16:43:04.928674 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.928562 2577 manager.go:216] Machine: {Timestamp:2026-09-11 16:43:04.927148365 +0000 UTC m=+0.335250645 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3132850 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2abf97be7d54a8fd9e6f208f487bcf SystemUUID:ec2abf97-be7d-54a8-fd9e-6f208f487bcf BootID:de83d362-7bf7-4a3f-ab9d-321d40a1dc2f Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:35:56:30:d3:31 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:35:56:30:d3:31 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:c1:dc:c2:40:de Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 16:43:04.928674 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.928672 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 16:43:04.928786 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.928759 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 16:43:04.931925 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.931895 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 16:43:04.932078 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.931927 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-165.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 16:43:04.932850 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.932840 2577 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 16:43:04.932884 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.932852 2577 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 16:43:04.932884 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.932860 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 16:43:04.932884 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.932873 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 16:43:04.933702 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.933692 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 16:43:04.937465 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.937452 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 11 16:43:04.937623 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.937613 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 16:43:04.937673 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.937663 2577 kubelet.go:486] "Attempting to sync node with API server" Sep 11 16:43:04.937710 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.937678 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 16:43:04.937710 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.937693 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 16:43:04.937710 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.937702 2577 kubelet.go:398] "Adding apiserver pod source" Sep 11 16:43:04.937710 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.937710 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 16:43:04.938107 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.938095 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 16:43:04.938156 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.938114 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 16:43:04.940449 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.940421 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 16:43:04.941136 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.941113 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 16:43:04.942321 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.942303 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 16:43:04.942410 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.942342 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 16:43:04.944173 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.944159 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 16:43:04.944252 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.944186 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 16:43:04.944252 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.944202 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 16:43:04.944252 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.944218 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 16:43:04.944252 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.944238 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 16:43:04.944445 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.944253 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 16:43:04.944445 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.944274 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 16:43:04.944445 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.944283 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 16:43:04.944445 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.944294 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 16:43:04.944445 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.944303 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 16:43:04.944445 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.944316 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 16:43:04.944445 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.944325 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 16:43:04.945111 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.945099 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 16:43:04.945157 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.945113 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 16:43:04.949213 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.949196 2577 server.go:1268] "Started kubelet" Sep 11 16:43:04.949302 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.949268 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 16:43:04.949444 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.949291 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 16:43:04.949486 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.949463 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 16:43:04.950000 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.949978 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-165.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 16:43:04.950246 ip-10-0-143-165 systemd[1]: Started Kubernetes Kubelet. Sep 11 16:43:04.950458 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:04.950329 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-165.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 16:43:04.950458 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:04.950385 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 16:43:04.951454 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.951432 2577 server.go:310] "Adding debug handlers to kubelet server" Sep 11 16:43:04.956531 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.956434 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 16:43:04.958666 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:04.958639 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 16:43:04.959759 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.959744 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 16:43:04.959837 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.959762 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 16:43:04.960021 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:04.959881 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-165.ec2.internal\" not found" Sep 11 16:43:04.960390 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.960371 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 16:43:04.960390 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.960390 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 16:43:04.960550 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.960532 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 16:43:04.960636 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.960625 2577 reconstruct.go:97] "Volume reconstruction finished" Sep 11 16:43:04.960636 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.960636 2577 reconciler.go:29] "Reconciler: start to sync state" Sep 11 16:43:04.960926 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.960907 2577 factory.go:55] Registering systemd factory Sep 11 16:43:04.960926 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.960923 2577 factory.go:223] Registration of the systemd container factory successfully Sep 11 16:43:04.961171 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.961152 2577 factory.go:153] Registering CRI-O factory Sep 11 16:43:04.961171 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.961169 2577 factory.go:223] Registration of the crio container factory successfully Sep 11 16:43:04.961306 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.961214 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 16:43:04.961306 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.961236 2577 factory.go:103] Registering Raw factory Sep 11 16:43:04.961306 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.961256 2577 manager.go:1196] Started watching for new ooms in manager Sep 11 16:43:04.961719 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.961707 2577 manager.go:349] Starting recovery of all containers Sep 11 16:43:04.963163 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:04.963118 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-165.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 16:43:04.963726 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:04.963700 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 16:43:04.965030 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:04.963387 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-165.ec2.internal.18d451e26f911b53 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-165.ec2.internal,UID:ip-10-0-143-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-165.ec2.internal,},FirstTimestamp:2026-09-11 16:43:04.949168979 +0000 UTC m=+0.357271239,LastTimestamp:2026-09-11 16:43:04.949168979 +0000 UTC m=+0.357271239,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-165.ec2.internal,}" Sep 11 16:43:04.972818 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.972798 2577 manager.go:354] Recovery completed Sep 11 16:43:04.977321 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.977305 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 16:43:04.979197 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.979177 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasSufficientMemory" Sep 11 16:43:04.979291 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.979215 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 16:43:04.979291 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.979231 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasSufficientPID" Sep 11 16:43:04.979886 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.979872 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 16:43:04.979886 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.979884 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 16:43:04.980019 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.979898 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 11 16:43:04.980019 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.979873 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 16:43:04.980714 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.980699 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasSufficientMemory" Sep 11 16:43:04.980796 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.980730 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 16:43:04.980796 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.980744 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasSufficientPID" Sep 11 16:43:04.981891 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.981878 2577 policy_none.go:49] "None policy: Start" Sep 11 16:43:04.981891 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.981894 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 16:43:04.982001 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.981911 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 16:43:04.982211 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:04.982139 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-165.ec2.internal.18d451e2715b4be1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-165.ec2.internal,UID:ip-10-0-143-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-165.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-165.ec2.internal,},FirstTimestamp:2026-09-11 16:43:04.979196897 +0000 UTC m=+0.387299157,LastTimestamp:2026-09-11 16:43:04.979196897 +0000 UTC m=+0.387299157,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-165.ec2.internal,}" Sep 11 16:43:04.983972 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:04.983961 2577 policy_none.go:47] "Start" Sep 11 16:43:04.993352 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:04.993136 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-165.ec2.internal.18d451e2715bae92 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-165.ec2.internal,UID:ip-10-0-143-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-165.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-165.ec2.internal,},FirstTimestamp:2026-09-11 16:43:04.979222162 +0000 UTC m=+0.387324424,LastTimestamp:2026-09-11 16:43:04.979222162 +0000 UTC m=+0.387324424,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-165.ec2.internal,}" Sep 11 16:43:05.003911 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:05.003840 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-165.ec2.internal.18d451e2715be7c3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-165.ec2.internal,UID:ip-10-0-143-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-165.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-165.ec2.internal,},FirstTimestamp:2026-09-11 16:43:04.979236803 +0000 UTC m=+0.387339064,LastTimestamp:2026-09-11 16:43:04.979236803 +0000 UTC m=+0.387339064,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-165.ec2.internal,}" Sep 11 16:43:05.004979 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.004959 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2zbjl" Sep 11 16:43:05.006275 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.006258 2577 manager.go:337] "Starting Device Plugin manager" Sep 11 16:43:05.006380 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:05.006303 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 16:43:05.006380 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.006316 2577 server.go:87] "Starting device plugin registration server" Sep 11 16:43:05.006705 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.006689 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 16:43:05.006801 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.006703 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 16:43:05.007222 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.006867 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 16:43:05.007222 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.006954 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 16:43:05.007222 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.006961 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 16:43:05.007605 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:05.007585 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 16:43:05.007760 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:05.007740 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-165.ec2.internal\" not found" Sep 11 16:43:05.013652 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:05.013580 2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-165.ec2.internal.18d451e2715b4be1\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-165.ec2.internal.18d451e2715b4be1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-165.ec2.internal,UID:ip-10-0-143-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-165.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-165.ec2.internal,},FirstTimestamp:2026-09-11 16:43:04.979196897 +0000 UTC m=+0.387299157,LastTimestamp:2026-09-11 16:43:04.980714671 +0000 UTC m=+0.388816935,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-165.ec2.internal,}" Sep 11 16:43:05.029686 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.015645 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2zbjl" Sep 11 16:43:05.085910 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.085860 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 16:43:05.087145 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.087129 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 16:43:05.087234 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.087190 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 16:43:05.087234 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.087223 2577 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 16:43:05.087359 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:05.087329 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 16:43:05.090390 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.090368 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 16:43:05.107813 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.107787 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 16:43:05.108857 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.108837 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasSufficientMemory" Sep 11 16:43:05.108978 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.108874 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 16:43:05.108978 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.108889 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasSufficientPID" Sep 11 16:43:05.108978 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.108917 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-165.ec2.internal" Sep 11 16:43:05.116172 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.116148 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-165.ec2.internal" Sep 11 16:43:05.116297 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:05.116177 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-165.ec2.internal\": node \"ip-10-0-143-165.ec2.internal\" not found" Sep 11 16:43:05.143956 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:05.143882 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-165.ec2.internal\" not found" Sep 11 16:43:05.188156 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.188121 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-165.ec2.internal"] Sep 11 16:43:05.188289 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.188207 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 16:43:05.189762 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.189746 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasSufficientMemory" Sep 11 16:43:05.189855 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.189774 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 16:43:05.189855 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.189784 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasSufficientPID" Sep 11 16:43:05.191060 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.191047 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 16:43:05.191197 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.191182 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal" Sep 11 16:43:05.191243 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.191216 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 16:43:05.191822 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.191804 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasSufficientMemory" Sep 11 16:43:05.191913 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.191808 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasSufficientMemory" Sep 11 16:43:05.191913 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.191837 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 16:43:05.191913 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.191851 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasSufficientPID" Sep 11 16:43:05.191913 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.191854 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 16:43:05.191913 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.191867 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasSufficientPID" Sep 11 16:43:05.193163 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.193148 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-165.ec2.internal" Sep 11 16:43:05.193243 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.193177 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 16:43:05.193953 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.193938 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasSufficientMemory" Sep 11 16:43:05.194060 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.193963 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 16:43:05.194060 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.193974 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasSufficientPID" Sep 11 16:43:05.223384 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:05.223357 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-165.ec2.internal\" not found" node="ip-10-0-143-165.ec2.internal" Sep 11 16:43:05.227932 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:05.227913 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-165.ec2.internal\" not found" node="ip-10-0-143-165.ec2.internal" Sep 11 16:43:05.244173 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:05.244143 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-165.ec2.internal\" not found" Sep 11 16:43:05.262964 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.262928 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81e835015ef18fd79319f1af18d7d65f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal\" (UID: \"81e835015ef18fd79319f1af18d7d65f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal" Sep 11 16:43:05.263091 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.262974 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81e835015ef18fd79319f1af18d7d65f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal\" (UID: \"81e835015ef18fd79319f1af18d7d65f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal" Sep 11 16:43:05.263091 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.263013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3752174baf855d01c4a6df678c4937e-config\") pod \"kube-apiserver-proxy-ip-10-0-143-165.ec2.internal\" (UID: \"a3752174baf855d01c4a6df678c4937e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-165.ec2.internal" Sep 11 16:43:05.344311 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:05.344273 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-165.ec2.internal\" not found" Sep 11 16:43:05.363689 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.363663 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81e835015ef18fd79319f1af18d7d65f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal\" (UID: \"81e835015ef18fd79319f1af18d7d65f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal" Sep 11 16:43:05.363689 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.363689 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81e835015ef18fd79319f1af18d7d65f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal\" (UID: \"81e835015ef18fd79319f1af18d7d65f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal" Sep 11 16:43:05.363865 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.363708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3752174baf855d01c4a6df678c4937e-config\") pod \"kube-apiserver-proxy-ip-10-0-143-165.ec2.internal\" (UID: \"a3752174baf855d01c4a6df678c4937e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-165.ec2.internal" Sep 11 16:43:05.363865 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.363761 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3752174baf855d01c4a6df678c4937e-config\") pod \"kube-apiserver-proxy-ip-10-0-143-165.ec2.internal\" (UID: \"a3752174baf855d01c4a6df678c4937e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-165.ec2.internal" Sep 11 16:43:05.363865 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.363782 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81e835015ef18fd79319f1af18d7d65f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal\" (UID: \"81e835015ef18fd79319f1af18d7d65f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal" Sep 11 16:43:05.363865 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.363783 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81e835015ef18fd79319f1af18d7d65f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal\" (UID: \"81e835015ef18fd79319f1af18d7d65f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal" Sep 11 16:43:05.445103 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:05.445032 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-165.ec2.internal\" not found" Sep 11 16:43:05.525435 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.525397 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal" Sep 11 16:43:05.530810 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.530790 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-165.ec2.internal" Sep 11 16:43:05.545263 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:05.545237 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-165.ec2.internal\" not found" Sep 11 16:43:05.645719 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:05.645680 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-165.ec2.internal\" not found" Sep 11 16:43:05.746153 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:05.746077 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-165.ec2.internal\" not found" Sep 11 16:43:05.846615 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:05.846588 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-165.ec2.internal\" not found" Sep 11 16:43:05.902004 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.901961 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 16:43:05.902574 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:05.902125 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 16:43:05.947339 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:05.947305 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-165.ec2.internal\" not found" Sep 11 16:43:06.003418 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.003335 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 16:43:06.020351 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.020316 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 16:38:04 +0000 UTC" deadline="2028-04-06 11:59:07.855443084 +0000 UTC" Sep 11 16:43:06.020351 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.020347 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13747h16m1.835099214s" Sep 11 16:43:06.047425 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:06.047395 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-165.ec2.internal\" not found" Sep 11 16:43:06.147658 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:06.147627 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-165.ec2.internal\" not found" Sep 11 16:43:06.165962 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.165932 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 16:43:06.167171 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.167150 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasSufficientMemory" Sep 11 16:43:06.167288 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.167187 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 16:43:06.167288 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.167198 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeHasSufficientPID" Sep 11 16:43:06.244957 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:06.244918 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e835015ef18fd79319f1af18d7d65f.slice/crio-39d60148a95855aa3bbe920b9b60c57cc713f5a346d10f155d053109f97fa426 WatchSource:0}: Error finding container 39d60148a95855aa3bbe920b9b60c57cc713f5a346d10f155d053109f97fa426: Status 404 returned error can't find the container with id 39d60148a95855aa3bbe920b9b60c57cc713f5a346d10f155d053109f97fa426 Sep 11 16:43:06.246698 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:06.246672 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3752174baf855d01c4a6df678c4937e.slice/crio-6d8be2f5d1be8d5f90a4c15b21be1680750edf0451172e8836e78b63e05a621d WatchSource:0}: Error finding container 6d8be2f5d1be8d5f90a4c15b21be1680750edf0451172e8836e78b63e05a621d: Status 404 returned error can't find the container with id 6d8be2f5d1be8d5f90a4c15b21be1680750edf0451172e8836e78b63e05a621d Sep 11 16:43:06.247818 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:06.247794 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-165.ec2.internal\" not found" Sep 11 16:43:06.248652 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.248639 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 16:43:06.341519 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.341417 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 16:43:06.360068 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.360039 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal" Sep 11 16:43:06.374219 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.374192 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 16:43:06.375745 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.375727 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-165.ec2.internal" Sep 11 16:43:06.384837 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.384818 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 16:43:06.526068 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.526037 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 16:43:06.938965 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.938932 2577 apiserver.go:52] "Watching apiserver" Sep 11 16:43:06.949857 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.949819 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 16:43:06.950896 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.950863 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-7vclm","openshift-ovn-kubernetes/ovnkube-node-mgslh","kube-system/global-pull-secret-syncer-kx9qk","kube-system/konnectivity-agent-vrc4d","openshift-cluster-node-tuning-operator/tuned-trfbj","openshift-dns/node-resolver-4ptxv","openshift-multus/multus-additional-cni-plugins-k7dfv","openshift-multus/multus-zgzsg","openshift-multus/network-metrics-daemon-68tln","openshift-network-operator/iptables-alerter-qrzv2","kube-system/kube-apiserver-proxy-ip-10-0-143-165.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5","openshift-image-registry/node-ca-r4sbm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal"] Sep 11 16:43:06.952506 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.952469 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:06.954059 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.954034 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r4sbm" Sep 11 16:43:06.955697 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.955646 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:06.955788 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:06.955713 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kx9qk" podUID="9c31fb9a-1977-49b8-8396-c462b1d9f47f" Sep 11 16:43:06.956690 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.956666 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vrc4d" Sep 11 16:43:06.956991 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.956972 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8kgpz\"" Sep 11 16:43:06.957076 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.957057 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 16:43:06.957198 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.957182 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 16:43:06.957293 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.957278 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 16:43:06.958519 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.958285 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:06.959333 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.959313 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4ptxv" Sep 11 16:43:06.963201 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.960058 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 16:43:06.963201 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.960680 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:06.963201 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:06.960748 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68tln" podUID="f1193c55-865c-4fa7-aabb-9ebbdadd11d2" Sep 11 16:43:06.963201 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.961659 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 16:43:06.963201 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.961785 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 16:43:06.963201 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.961976 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 16:43:06.963201 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.961982 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 16:43:06.963201 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.962235 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 16:43:06.963201 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.962272 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 16:43:06.963201 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.962601 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 16:43:06.963201 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.962617 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7rvnc\"" Sep 11 16:43:06.963201 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.962755 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 16:43:06.963201 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.962894 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 16:43:06.963201 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.962964 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 16:43:06.963201 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.963070 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kwmz6\"" Sep 11 16:43:06.963201 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.963170 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:06.963929 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:06.963224 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vclm" podUID="592424fa-0fb1-4974-8d4a-097697d013c0" Sep 11 16:43:06.963929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.963354 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 16:43:06.963929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.963658 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:06.964960 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.964753 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k5msh\"" Sep 11 16:43:06.965057 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.965032 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ccjpz\"" Sep 11 16:43:06.965609 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.965300 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qrzv2" Sep 11 16:43:06.966787 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.966749 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zgzsg" Sep 11 16:43:06.968512 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.968374 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 16:43:06.968512 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.968387 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 16:43:06.968656 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.968644 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 16:43:06.968819 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.968803 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 16:43:06.970045 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.969171 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 16:43:06.970147 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.969411 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zb52f\"" Sep 11 16:43:06.970218 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.969466 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 16:43:06.970218 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.969600 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 16:43:06.970314 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.969629 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:06.970314 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.969737 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 16:43:06.970416 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.969868 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 16:43:06.970416 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.969980 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2hfch\"" Sep 11 16:43:06.970620 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.970598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-var-lib-kubelet\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:06.970687 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.970634 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a9e4db82-8def-4848-983d-03de080ae127-etc-tuned\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:06.970687 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.970658 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9e4db82-8def-4848-983d-03de080ae127-tmp\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:06.970790 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.970692 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9g6c\" (UniqueName: \"kubernetes.io/projected/a9e4db82-8def-4848-983d-03de080ae127-kube-api-access-z9g6c\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:06.970790 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.970717 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:06.970790 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.970742 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c31fb9a-1977-49b8-8396-c462b1d9f47f-kubelet-config\") pod \"global-pull-secret-syncer-kx9qk\" (UID: \"9c31fb9a-1977-49b8-8396-c462b1d9f47f\") " pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:06.970790 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.970766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-etc-sysctl-d\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:06.970977 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.970808 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-sys\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:06.970977 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.970843 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:06.970977 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.970883 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-etc-modprobe-d\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:06.970977 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.970906 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c11370ff-491d-4494-861c-090e05beda2c-hosts-file\") pod \"node-resolver-4ptxv\" (UID: \"c11370ff-491d-4494-861c-090e05beda2c\") " pod="openshift-dns/node-resolver-4ptxv" Sep 11 16:43:06.970977 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.970933 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n277q\" (UniqueName: \"kubernetes.io/projected/c11370ff-491d-4494-861c-090e05beda2c-kube-api-access-n277q\") pod \"node-resolver-4ptxv\" (UID: \"c11370ff-491d-4494-861c-090e05beda2c\") " pod="openshift-dns/node-resolver-4ptxv" Sep 11 16:43:06.970977 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.970957 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-cnibin\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:06.971260 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.970985 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret\") pod \"global-pull-secret-syncer-kx9qk\" (UID: \"9c31fb9a-1977-49b8-8396-c462b1d9f47f\") " pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:06.971260 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971008 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-etc-kubernetes\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:06.971260 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971052 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc59x\" (UniqueName: \"kubernetes.io/projected/ef68ac95-3add-420a-85d4-31cef2cddd8a-kube-api-access-sc59x\") pod \"node-ca-r4sbm\" (UID: \"ef68ac95-3add-420a-85d4-31cef2cddd8a\") " pod="openshift-image-registry/node-ca-r4sbm" Sep 11 16:43:06.971260 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971089 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-system-cni-dir\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:06.971260 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:06.971260 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971153 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-etc-systemd\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:06.971260 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971181 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-run\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:06.971260 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971206 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-lib-modules\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:06.971660 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971263 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glbx7\" (UniqueName: \"kubernetes.io/projected/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-kube-api-access-glbx7\") pod \"network-metrics-daemon-68tln\" (UID: \"f1193c55-865c-4fa7-aabb-9ebbdadd11d2\") " pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:06.971660 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971293 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:06.971660 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w2fb\" (UniqueName: \"kubernetes.io/projected/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-kube-api-access-4w2fb\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:06.971660 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971369 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd648125-8d5e-455b-95bf-5834c01e4db6-agent-certs\") pod \"konnectivity-agent-vrc4d\" (UID: \"cd648125-8d5e-455b-95bf-5834c01e4db6\") " pod="kube-system/konnectivity-agent-vrc4d" Sep 11 16:43:06.971660 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971396 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs\") pod \"network-metrics-daemon-68tln\" (UID: \"f1193c55-865c-4fa7-aabb-9ebbdadd11d2\") " pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:06.971660 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971419 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c11370ff-491d-4494-861c-090e05beda2c-tmp-dir\") pod \"node-resolver-4ptxv\" (UID: \"c11370ff-491d-4494-861c-090e05beda2c\") " pod="openshift-dns/node-resolver-4ptxv" Sep 11 16:43:06.971660 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971475 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwqq2\" (UniqueName: \"kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2\") pod \"network-check-target-7vclm\" (UID: \"592424fa-0fb1-4974-8d4a-097697d013c0\") " pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:06.971660 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971538 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c31fb9a-1977-49b8-8396-c462b1d9f47f-dbus\") pod \"global-pull-secret-syncer-kx9qk\" (UID: \"9c31fb9a-1977-49b8-8396-c462b1d9f47f\") " pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:06.971660 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971566 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd648125-8d5e-455b-95bf-5834c01e4db6-konnectivity-ca\") pod \"konnectivity-agent-vrc4d\" (UID: \"cd648125-8d5e-455b-95bf-5834c01e4db6\") " pod="kube-system/konnectivity-agent-vrc4d" Sep 11 16:43:06.971660 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971588 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-etc-sysconfig\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:06.971660 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971613 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-etc-sysctl-conf\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:06.971660 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971651 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-host\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:06.972235 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971673 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef68ac95-3add-420a-85d4-31cef2cddd8a-host\") pod \"node-ca-r4sbm\" (UID: \"ef68ac95-3add-420a-85d4-31cef2cddd8a\") " pod="openshift-image-registry/node-ca-r4sbm" Sep 11 16:43:06.972235 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.971696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ef68ac95-3add-420a-85d4-31cef2cddd8a-serviceca\") pod \"node-ca-r4sbm\" (UID: \"ef68ac95-3add-420a-85d4-31cef2cddd8a\") " pod="openshift-image-registry/node-ca-r4sbm" Sep 11 16:43:06.976323 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.976285 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 16:43:06.976560 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.976531 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 16:43:06.976962 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.976939 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 16:43:06.977152 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.977137 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 16:43:06.977581 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.977561 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xbj4f\"" Sep 11 16:43:06.980398 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.980328 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kcjgg\"" Sep 11 16:43:06.990428 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:06.990397 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 16:43:07.006144 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.006085 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vrk8c" Sep 11 16:43:07.017285 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.017253 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vrk8c" Sep 11 16:43:07.062096 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.061968 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 16:43:07.072691 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.072649 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-system-cni-dir\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.072691 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.072682 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-multus-socket-dir-parent\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.072926 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.072703 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-host-var-lib-cni-multus\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.072926 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.072725 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-host-run-multus-certs\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.072926 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.072748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/257ca13e-8a75-4158-9b2c-5842fefa32bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.072926 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.072775 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w2fb\" (UniqueName: \"kubernetes.io/projected/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-kube-api-access-4w2fb\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:07.072926 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.072800 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-slash\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.072926 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.072913 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-run-ovn\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.073213 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.072944 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-cni-netd\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.073213 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.072965 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9a1a9f06-4b32-45eb-8838-43f5d0649c40-env-overrides\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.073213 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.072992 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c11370ff-491d-4494-861c-090e05beda2c-tmp-dir\") pod \"node-resolver-4ptxv\" (UID: \"c11370ff-491d-4494-861c-090e05beda2c\") " pod="openshift-dns/node-resolver-4ptxv" Sep 11 16:43:07.073213 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073035 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwqq2\" (UniqueName: \"kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2\") pod \"network-check-target-7vclm\" (UID: \"592424fa-0fb1-4974-8d4a-097697d013c0\") " pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:07.073213 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd648125-8d5e-455b-95bf-5834c01e4db6-konnectivity-ca\") pod \"konnectivity-agent-vrc4d\" (UID: \"cd648125-8d5e-455b-95bf-5834c01e4db6\") " pod="kube-system/konnectivity-agent-vrc4d" Sep 11 16:43:07.073213 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073100 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-etc-sysconfig\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.073213 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073124 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-host\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.073213 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073147 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-run-openvswitch\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.073213 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073183 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-log-socket\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.073690 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073205 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/257ca13e-8a75-4158-9b2c-5842fefa32bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.073690 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073254 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef68ac95-3add-420a-85d4-31cef2cddd8a-host\") pod \"node-ca-r4sbm\" (UID: \"ef68ac95-3add-420a-85d4-31cef2cddd8a\") " pod="openshift-image-registry/node-ca-r4sbm" Sep 11 16:43:07.073690 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073277 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtkdh\" (UniqueName: \"kubernetes.io/projected/9a1a9f06-4b32-45eb-8838-43f5d0649c40-kube-api-access-mtkdh\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.073690 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073303 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:07.073690 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073326 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-kubelet\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.073690 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073348 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9a1a9f06-4b32-45eb-8838-43f5d0649c40-ovnkube-script-lib\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.073690 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073369 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-multus-conf-dir\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.073690 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c11370ff-491d-4494-861c-090e05beda2c-hosts-file\") pod \"node-resolver-4ptxv\" (UID: \"c11370ff-491d-4494-861c-090e05beda2c\") " pod="openshift-dns/node-resolver-4ptxv" Sep 11 16:43:07.073690 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073411 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-cnibin\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:07.073690 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073446 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-run-systemd\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.073690 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073449 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c11370ff-491d-4494-861c-090e05beda2c-tmp-dir\") pod \"node-resolver-4ptxv\" (UID: \"c11370ff-491d-4494-861c-090e05beda2c\") " pod="openshift-dns/node-resolver-4ptxv" Sep 11 16:43:07.073690 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073475 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/257ca13e-8a75-4158-9b2c-5842fefa32bf-socket-dir\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.073690 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073516 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpq6k\" (UniqueName: \"kubernetes.io/projected/257ca13e-8a75-4158-9b2c-5842fefa32bf-kube-api-access-mpq6k\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.073690 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073546 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret\") pod \"global-pull-secret-syncer-kx9qk\" (UID: \"9c31fb9a-1977-49b8-8396-c462b1d9f47f\") " pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:07.074349 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073870 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-cnibin\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.074349 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073900 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a269739e-353a-4b14-ae8c-05de63045f05-cni-binary-copy\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.074349 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073926 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-host-var-lib-cni-bin\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.074349 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073950 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/257ca13e-8a75-4158-9b2c-5842fefa32bf-registration-dir\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.074349 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073972 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc59x\" (UniqueName: \"kubernetes.io/projected/ef68ac95-3add-420a-85d4-31cef2cddd8a-kube-api-access-sc59x\") pod \"node-ca-r4sbm\" (UID: \"ef68ac95-3add-420a-85d4-31cef2cddd8a\") " pod="openshift-image-registry/node-ca-r4sbm" Sep 11 16:43:07.074349 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.073988 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-system-cni-dir\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:07.074349 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:07.074349 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074027 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-etc-systemd\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.074349 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074043 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-lib-modules\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.074349 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074063 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-etc-kubernetes\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.074349 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glbx7\" (UniqueName: \"kubernetes.io/projected/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-kube-api-access-glbx7\") pod \"network-metrics-daemon-68tln\" (UID: \"f1193c55-865c-4fa7-aabb-9ebbdadd11d2\") " pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:07.074349 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074124 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:07.074349 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074153 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd648125-8d5e-455b-95bf-5834c01e4db6-agent-certs\") pod \"konnectivity-agent-vrc4d\" (UID: \"cd648125-8d5e-455b-95bf-5834c01e4db6\") " pod="kube-system/konnectivity-agent-vrc4d" Sep 11 16:43:07.074349 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074177 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-var-lib-openvswitch\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.074349 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074238 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/257ca13e-8a75-4158-9b2c-5842fefa32bf-sys-fs\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.074349 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074266 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs\") pod \"network-metrics-daemon-68tln\" (UID: \"f1193c55-865c-4fa7-aabb-9ebbdadd11d2\") " pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:07.074349 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074293 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c31fb9a-1977-49b8-8396-c462b1d9f47f-dbus\") pod \"global-pull-secret-syncer-kx9qk\" (UID: \"9c31fb9a-1977-49b8-8396-c462b1d9f47f\") " pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:07.075150 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074451 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-etc-sysctl-conf\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.075150 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074473 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a9e4db82-8def-4848-983d-03de080ae127-etc-tuned\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.075150 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074488 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9e4db82-8def-4848-983d-03de080ae127-tmp\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.075150 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074524 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-etc-openvswitch\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.075150 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074594 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c11370ff-491d-4494-861c-090e05beda2c-hosts-file\") pod \"node-resolver-4ptxv\" (UID: \"c11370ff-491d-4494-861c-090e05beda2c\") " pod="openshift-dns/node-resolver-4ptxv" Sep 11 16:43:07.075150 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074602 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-host\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.075150 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074595 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-cnibin\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:07.075150 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074648 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c31fb9a-1977-49b8-8396-c462b1d9f47f-dbus\") pod \"global-pull-secret-syncer-kx9qk\" (UID: \"9c31fb9a-1977-49b8-8396-c462b1d9f47f\") " pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:07.075150 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074654 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-etc-sysconfig\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.075150 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074693 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-system-cni-dir\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:07.075150 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:07.074753 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 16:43:07.075150 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074771 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:07.075150 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:07.074791 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 16:43:07.075150 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-run-ovn-kubernetes\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.075150 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:07.074858 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret podName:9c31fb9a-1977-49b8-8396-c462b1d9f47f nodeName:}" failed. No retries permitted until 2026-09-11 16:43:07.574835438 +0000 UTC m=+2.982937687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret") pod "global-pull-secret-syncer-kx9qk" (UID: "9c31fb9a-1977-49b8-8396-c462b1d9f47f") : object "kube-system"/"original-pull-secret" not registered Sep 11 16:43:07.075150 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.074967 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-lib-modules\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.075150 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075007 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-etc-systemd\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.075919 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:07.075013 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs podName:f1193c55-865c-4fa7-aabb-9ebbdadd11d2 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:07.574997494 +0000 UTC m=+2.983099745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs") pod "network-metrics-daemon-68tln" (UID: "f1193c55-865c-4fa7-aabb-9ebbdadd11d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 16:43:07.075919 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075058 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9a1a9f06-4b32-45eb-8838-43f5d0649c40-ovnkube-config\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.075919 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075062 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef68ac95-3add-420a-85d4-31cef2cddd8a-host\") pod \"node-ca-r4sbm\" (UID: \"ef68ac95-3add-420a-85d4-31cef2cddd8a\") " pod="openshift-image-registry/node-ca-r4sbm" Sep 11 16:43:07.075919 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075112 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ef68ac95-3add-420a-85d4-31cef2cddd8a-serviceca\") pod \"node-ca-r4sbm\" (UID: \"ef68ac95-3add-420a-85d4-31cef2cddd8a\") " pod="openshift-image-registry/node-ca-r4sbm" Sep 11 16:43:07.075919 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-var-lib-kubelet\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.075919 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9g6c\" (UniqueName: \"kubernetes.io/projected/a9e4db82-8def-4848-983d-03de080ae127-kube-api-access-z9g6c\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.075919 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075217 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-etc-sysctl-conf\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.075919 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075231 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-run-netns\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.075919 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075261 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.075919 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075286 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:07.075919 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e562cd0c-8abc-4d38-96d5-2960fb624672-iptables-alerter-script\") pod \"iptables-alerter-qrzv2\" (UID: \"e562cd0c-8abc-4d38-96d5-2960fb624672\") " pod="openshift-network-operator/iptables-alerter-qrzv2" Sep 11 16:43:07.075919 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075344 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-multus-cni-dir\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.075919 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075371 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-host-run-netns\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.075919 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075396 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 16:43:07.075919 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075515 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:07.075919 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075532 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd648125-8d5e-455b-95bf-5834c01e4db6-konnectivity-ca\") pod \"konnectivity-agent-vrc4d\" (UID: \"cd648125-8d5e-455b-95bf-5834c01e4db6\") " pod="kube-system/konnectivity-agent-vrc4d" Sep 11 16:43:07.076746 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075584 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c31fb9a-1977-49b8-8396-c462b1d9f47f-kubelet-config\") pod \"global-pull-secret-syncer-kx9qk\" (UID: \"9c31fb9a-1977-49b8-8396-c462b1d9f47f\") " pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:07.076746 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075598 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-var-lib-kubelet\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.076746 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075622 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-etc-sysctl-d\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.076746 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075648 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ef68ac95-3add-420a-85d4-31cef2cddd8a-serviceca\") pod \"node-ca-r4sbm\" (UID: \"ef68ac95-3add-420a-85d4-31cef2cddd8a\") " pod="openshift-image-registry/node-ca-r4sbm" Sep 11 16:43:07.076746 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075668 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-sys\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.076746 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075693 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c31fb9a-1977-49b8-8396-c462b1d9f47f-kubelet-config\") pod \"global-pull-secret-syncer-kx9qk\" (UID: \"9c31fb9a-1977-49b8-8396-c462b1d9f47f\") " pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:07.076746 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075721 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-etc-sysctl-d\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.076746 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075729 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgzm2\" (UniqueName: \"kubernetes.io/projected/e562cd0c-8abc-4d38-96d5-2960fb624672-kube-api-access-sgzm2\") pod \"iptables-alerter-qrzv2\" (UID: \"e562cd0c-8abc-4d38-96d5-2960fb624672\") " pod="openshift-network-operator/iptables-alerter-qrzv2" Sep 11 16:43:07.076746 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075767 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-sys\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.076746 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075813 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-host-var-lib-kubelet\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.076746 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075855 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-hostroot\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.076746 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075912 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a269739e-353a-4b14-ae8c-05de63045f05-multus-daemon-config\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.076746 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075951 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sxdx\" (UniqueName: \"kubernetes.io/projected/a269739e-353a-4b14-ae8c-05de63045f05-kube-api-access-9sxdx\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.076746 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.075982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:07.076746 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.076003 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-etc-modprobe-d\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.076746 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.076027 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-systemd-units\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.076746 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.076123 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e562cd0c-8abc-4d38-96d5-2960fb624672-host-slash\") pod \"iptables-alerter-qrzv2\" (UID: \"e562cd0c-8abc-4d38-96d5-2960fb624672\") " pod="openshift-network-operator/iptables-alerter-qrzv2" Sep 11 16:43:07.077552 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.076154 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/257ca13e-8a75-4158-9b2c-5842fefa32bf-device-dir\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.077552 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.076181 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n277q\" (UniqueName: \"kubernetes.io/projected/c11370ff-491d-4494-861c-090e05beda2c-kube-api-access-n277q\") pod \"node-resolver-4ptxv\" (UID: \"c11370ff-491d-4494-861c-090e05beda2c\") " pod="openshift-dns/node-resolver-4ptxv" Sep 11 16:43:07.077552 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.076185 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-etc-modprobe-d\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.077552 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.076222 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-host-run-k8s-cni-cncf-io\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.077552 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.076261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-etc-kubernetes\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.077552 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.076292 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-node-log\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.077552 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.076317 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-run\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.077552 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.076357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-cni-bin\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.077552 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.076397 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-run\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.077552 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.076397 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9e4db82-8def-4848-983d-03de080ae127-etc-kubernetes\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.077552 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.076391 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a1a9f06-4b32-45eb-8838-43f5d0649c40-ovn-node-metrics-cert\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.077552 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.076451 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:07.079927 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.079897 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9e4db82-8def-4848-983d-03de080ae127-tmp\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.080148 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.080125 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd648125-8d5e-455b-95bf-5834c01e4db6-agent-certs\") pod \"konnectivity-agent-vrc4d\" (UID: \"cd648125-8d5e-455b-95bf-5834c01e4db6\") " pod="kube-system/konnectivity-agent-vrc4d" Sep 11 16:43:07.080269 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.080243 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a9e4db82-8def-4848-983d-03de080ae127-etc-tuned\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.085513 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.085473 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w2fb\" (UniqueName: \"kubernetes.io/projected/9052d32a-f4da-4ace-9da5-6a63d6e7e0c2-kube-api-access-4w2fb\") pod \"multus-additional-cni-plugins-k7dfv\" (UID: \"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2\") " pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:07.087200 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:07.087184 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 16:43:07.087310 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:07.087208 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 16:43:07.087310 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:07.087219 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gwqq2 for pod openshift-network-diagnostics/network-check-target-7vclm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 16:43:07.087310 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:07.087275 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2 podName:592424fa-0fb1-4974-8d4a-097697d013c0 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:07.587258235 +0000 UTC m=+2.995360499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gwqq2" (UniqueName: "kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2") pod "network-check-target-7vclm" (UID: "592424fa-0fb1-4974-8d4a-097697d013c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 16:43:07.093605 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.093524 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-165.ec2.internal" event={"ID":"a3752174baf855d01c4a6df678c4937e","Type":"ContainerStarted","Data":"6d8be2f5d1be8d5f90a4c15b21be1680750edf0451172e8836e78b63e05a621d"} Sep 11 16:43:07.094924 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.094823 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n277q\" (UniqueName: \"kubernetes.io/projected/c11370ff-491d-4494-861c-090e05beda2c-kube-api-access-n277q\") pod \"node-resolver-4ptxv\" (UID: \"c11370ff-491d-4494-861c-090e05beda2c\") " pod="openshift-dns/node-resolver-4ptxv" Sep 11 16:43:07.095385 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.095360 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal" event={"ID":"81e835015ef18fd79319f1af18d7d65f","Type":"ContainerStarted","Data":"39d60148a95855aa3bbe920b9b60c57cc713f5a346d10f155d053109f97fa426"} Sep 11 16:43:07.095773 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.095750 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9g6c\" (UniqueName: \"kubernetes.io/projected/a9e4db82-8def-4848-983d-03de080ae127-kube-api-access-z9g6c\") pod \"tuned-trfbj\" (UID: \"a9e4db82-8def-4848-983d-03de080ae127\") " pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.096381 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.096356 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc59x\" (UniqueName: \"kubernetes.io/projected/ef68ac95-3add-420a-85d4-31cef2cddd8a-kube-api-access-sc59x\") pod \"node-ca-r4sbm\" (UID: \"ef68ac95-3add-420a-85d4-31cef2cddd8a\") " pod="openshift-image-registry/node-ca-r4sbm" Sep 11 16:43:07.097879 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.097846 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glbx7\" (UniqueName: \"kubernetes.io/projected/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-kube-api-access-glbx7\") pod \"network-metrics-daemon-68tln\" (UID: \"f1193c55-865c-4fa7-aabb-9ebbdadd11d2\") " pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:07.177232 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177180 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/257ca13e-8a75-4158-9b2c-5842fefa32bf-device-dir\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.177413 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177242 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-host-run-k8s-cni-cncf-io\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.177413 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177276 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-node-log\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.177413 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177304 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-cni-bin\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.177413 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177334 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a1a9f06-4b32-45eb-8838-43f5d0649c40-ovn-node-metrics-cert\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.177413 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177330 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/257ca13e-8a75-4158-9b2c-5842fefa32bf-device-dir\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.177413 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177358 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-system-cni-dir\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.177413 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177375 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-host-run-k8s-cni-cncf-io\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.177413 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177401 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-cni-bin\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.177808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177423 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-multus-socket-dir-parent\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.177808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177445 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-node-log\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.177808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177451 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-host-var-lib-cni-multus\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.177808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-host-run-multus-certs\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.177808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177523 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-system-cni-dir\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.177808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177519 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/257ca13e-8a75-4158-9b2c-5842fefa32bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.177808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177558 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-host-var-lib-cni-multus\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.177808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177576 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-slash\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.177808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177593 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-multus-socket-dir-parent\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.177808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177602 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-run-ovn\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.177808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177629 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-cni-netd\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.177808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177654 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9a1a9f06-4b32-45eb-8838-43f5d0649c40-env-overrides\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.177808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177664 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/257ca13e-8a75-4158-9b2c-5842fefa32bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.177808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177692 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-host-run-multus-certs\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.177808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177701 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-run-openvswitch\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.177808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177728 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-cni-netd\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.177808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-log-socket\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.177808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177754 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-slash\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.178697 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/257ca13e-8a75-4158-9b2c-5842fefa32bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.178697 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177778 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-run-ovn\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.178697 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177787 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtkdh\" (UniqueName: \"kubernetes.io/projected/9a1a9f06-4b32-45eb-8838-43f5d0649c40-kube-api-access-mtkdh\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.178697 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177802 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-run-openvswitch\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.178697 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177816 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-kubelet\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.178697 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177842 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9a1a9f06-4b32-45eb-8838-43f5d0649c40-ovnkube-script-lib\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.178697 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177868 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-multus-conf-dir\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.178697 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.177897 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-run-systemd\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.178697 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178073 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/257ca13e-8a75-4158-9b2c-5842fefa32bf-socket-dir\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.178697 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178130 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-log-socket\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.178697 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178184 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/257ca13e-8a75-4158-9b2c-5842fefa32bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.178697 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178230 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9a1a9f06-4b32-45eb-8838-43f5d0649c40-env-overrides\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.178697 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178262 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/257ca13e-8a75-4158-9b2c-5842fefa32bf-socket-dir\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.178697 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178284 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpq6k\" (UniqueName: \"kubernetes.io/projected/257ca13e-8a75-4158-9b2c-5842fefa32bf-kube-api-access-mpq6k\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.178697 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-cnibin\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.178697 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178336 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a269739e-353a-4b14-ae8c-05de63045f05-cni-binary-copy\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.178697 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178351 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-host-var-lib-cni-bin\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.179470 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178368 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/257ca13e-8a75-4158-9b2c-5842fefa32bf-registration-dir\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.179470 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178388 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-etc-kubernetes\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.179470 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-var-lib-openvswitch\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.179470 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178437 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/257ca13e-8a75-4158-9b2c-5842fefa32bf-sys-fs\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.179470 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178475 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-etc-openvswitch\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.179470 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178520 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-run-ovn-kubernetes\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.179470 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178552 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9a1a9f06-4b32-45eb-8838-43f5d0649c40-ovnkube-config\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.179470 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178579 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-run-netns\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.179470 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178596 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-multus-conf-dir\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.179470 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178608 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.179470 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178637 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e562cd0c-8abc-4d38-96d5-2960fb624672-iptables-alerter-script\") pod \"iptables-alerter-qrzv2\" (UID: \"e562cd0c-8abc-4d38-96d5-2960fb624672\") " pod="openshift-network-operator/iptables-alerter-qrzv2" Sep 11 16:43:07.179470 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178695 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.179470 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178702 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-run-netns\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.179470 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178734 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-run-ovn-kubernetes\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.179470 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178739 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-run-systemd\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.179470 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.178772 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-var-lib-openvswitch\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.179470 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179025 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-cnibin\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.180292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179019 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9a1a9f06-4b32-45eb-8838-43f5d0649c40-ovnkube-script-lib\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.180292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-multus-cni-dir\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.180292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179127 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-multus-cni-dir\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.180292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179162 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-host-var-lib-cni-bin\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.180292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179211 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/257ca13e-8a75-4158-9b2c-5842fefa32bf-sys-fs\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.180292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179233 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9a1a9f06-4b32-45eb-8838-43f5d0649c40-ovnkube-config\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.180292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179245 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-host-kubelet\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.180292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179283 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-etc-openvswitch\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.180292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179293 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/257ca13e-8a75-4158-9b2c-5842fefa32bf-registration-dir\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.180292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179328 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-etc-kubernetes\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.180292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179423 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-host-run-netns\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.180292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179514 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-host-run-netns\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.180292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179558 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgzm2\" (UniqueName: \"kubernetes.io/projected/e562cd0c-8abc-4d38-96d5-2960fb624672-kube-api-access-sgzm2\") pod \"iptables-alerter-qrzv2\" (UID: \"e562cd0c-8abc-4d38-96d5-2960fb624672\") " pod="openshift-network-operator/iptables-alerter-qrzv2" Sep 11 16:43:07.180292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179577 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e562cd0c-8abc-4d38-96d5-2960fb624672-iptables-alerter-script\") pod \"iptables-alerter-qrzv2\" (UID: \"e562cd0c-8abc-4d38-96d5-2960fb624672\") " pod="openshift-network-operator/iptables-alerter-qrzv2" Sep 11 16:43:07.180292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179605 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-host-var-lib-kubelet\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.180292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-hostroot\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.180292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a269739e-353a-4b14-ae8c-05de63045f05-multus-daemon-config\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.180292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179774 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sxdx\" (UniqueName: \"kubernetes.io/projected/a269739e-353a-4b14-ae8c-05de63045f05-kube-api-access-9sxdx\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.180994 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179779 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-host-var-lib-kubelet\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.180994 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179828 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-systemd-units\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.180994 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179831 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a269739e-353a-4b14-ae8c-05de63045f05-cni-binary-copy\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.180994 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e562cd0c-8abc-4d38-96d5-2960fb624672-host-slash\") pod \"iptables-alerter-qrzv2\" (UID: \"e562cd0c-8abc-4d38-96d5-2960fb624672\") " pod="openshift-network-operator/iptables-alerter-qrzv2" Sep 11 16:43:07.180994 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179910 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a269739e-353a-4b14-ae8c-05de63045f05-hostroot\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.180994 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179957 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9a1a9f06-4b32-45eb-8838-43f5d0649c40-systemd-units\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.180994 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.179994 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e562cd0c-8abc-4d38-96d5-2960fb624672-host-slash\") pod \"iptables-alerter-qrzv2\" (UID: \"e562cd0c-8abc-4d38-96d5-2960fb624672\") " pod="openshift-network-operator/iptables-alerter-qrzv2" Sep 11 16:43:07.180994 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.180210 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a269739e-353a-4b14-ae8c-05de63045f05-multus-daemon-config\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.180994 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.180345 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a1a9f06-4b32-45eb-8838-43f5d0649c40-ovn-node-metrics-cert\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.190100 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.188183 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtkdh\" (UniqueName: \"kubernetes.io/projected/9a1a9f06-4b32-45eb-8838-43f5d0649c40-kube-api-access-mtkdh\") pod \"ovnkube-node-mgslh\" (UID: \"9a1a9f06-4b32-45eb-8838-43f5d0649c40\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.190100 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.189485 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sxdx\" (UniqueName: \"kubernetes.io/projected/a269739e-353a-4b14-ae8c-05de63045f05-kube-api-access-9sxdx\") pod \"multus-zgzsg\" (UID: \"a269739e-353a-4b14-ae8c-05de63045f05\") " pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.190100 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.189524 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpq6k\" (UniqueName: \"kubernetes.io/projected/257ca13e-8a75-4158-9b2c-5842fefa32bf-kube-api-access-mpq6k\") pod \"aws-ebs-csi-driver-node-8nwt5\" (UID: \"257ca13e-8a75-4158-9b2c-5842fefa32bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.190100 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.189886 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgzm2\" (UniqueName: \"kubernetes.io/projected/e562cd0c-8abc-4d38-96d5-2960fb624672-kube-api-access-sgzm2\") pod \"iptables-alerter-qrzv2\" (UID: \"e562cd0c-8abc-4d38-96d5-2960fb624672\") " pod="openshift-network-operator/iptables-alerter-qrzv2" Sep 11 16:43:07.268239 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.268206 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k7dfv" Sep 11 16:43:07.278221 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.278194 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r4sbm" Sep 11 16:43:07.290151 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.290121 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vrc4d" Sep 11 16:43:07.297916 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.297890 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-trfbj" Sep 11 16:43:07.303780 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.303757 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4ptxv" Sep 11 16:43:07.311555 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.311530 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:07.320393 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.320361 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qrzv2" Sep 11 16:43:07.328380 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.328357 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zgzsg" Sep 11 16:43:07.335185 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.335156 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" Sep 11 16:43:07.492941 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.492857 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 16:43:07.583711 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.583677 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret\") pod \"global-pull-secret-syncer-kx9qk\" (UID: \"9c31fb9a-1977-49b8-8396-c462b1d9f47f\") " pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:07.583882 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.583739 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs\") pod \"network-metrics-daemon-68tln\" (UID: \"f1193c55-865c-4fa7-aabb-9ebbdadd11d2\") " pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:07.583882 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:07.583832 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 16:43:07.583981 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:07.583892 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 16:43:07.583981 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:07.583915 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret podName:9c31fb9a-1977-49b8-8396-c462b1d9f47f nodeName:}" failed. No retries permitted until 2026-09-11 16:43:08.583889456 +0000 UTC m=+3.991991724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret") pod "global-pull-secret-syncer-kx9qk" (UID: "9c31fb9a-1977-49b8-8396-c462b1d9f47f") : object "kube-system"/"original-pull-secret" not registered Sep 11 16:43:07.583981 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:07.583943 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs podName:f1193c55-865c-4fa7-aabb-9ebbdadd11d2 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:08.583929992 +0000 UTC m=+3.992032239 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs") pod "network-metrics-daemon-68tln" (UID: "f1193c55-865c-4fa7-aabb-9ebbdadd11d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 16:43:07.684078 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:07.684040 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwqq2\" (UniqueName: \"kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2\") pod \"network-check-target-7vclm\" (UID: \"592424fa-0fb1-4974-8d4a-097697d013c0\") " pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:07.684266 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:07.684229 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 16:43:07.684266 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:07.684255 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 16:43:07.684377 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:07.684270 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gwqq2 for pod openshift-network-diagnostics/network-check-target-7vclm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 16:43:07.684377 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:07.684353 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2 podName:592424fa-0fb1-4974-8d4a-097697d013c0 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:08.684334855 +0000 UTC m=+4.092437121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwqq2" (UniqueName: "kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2") pod "network-check-target-7vclm" (UID: "592424fa-0fb1-4974-8d4a-097697d013c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 16:43:08.019542 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:08.019474 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 16:38:07 +0000 UTC" deadline="2028-02-17 02:44:12.094219484 +0000 UTC" Sep 11 16:43:08.019542 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:08.019537 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12562h1m4.074686399s" Sep 11 16:43:08.206259 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:08.206044 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode562cd0c_8abc_4d38_96d5_2960fb624672.slice/crio-a443e7816fbad5c0baca38b6dcde5caa8a05a643f2c6cfa9da92a32b79a7218a WatchSource:0}: Error finding container a443e7816fbad5c0baca38b6dcde5caa8a05a643f2c6cfa9da92a32b79a7218a: Status 404 returned error can't find the container with id a443e7816fbad5c0baca38b6dcde5caa8a05a643f2c6cfa9da92a32b79a7218a Sep 11 16:43:08.211746 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:08.211715 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd648125_8d5e_455b_95bf_5834c01e4db6.slice/crio-006372f298e625ac3cb141ea952be99f0c1ea40f283d385be5477b8c726cd28c WatchSource:0}: Error finding container 006372f298e625ac3cb141ea952be99f0c1ea40f283d385be5477b8c726cd28c: Status 404 returned error can't find the container with id 006372f298e625ac3cb141ea952be99f0c1ea40f283d385be5477b8c726cd28c Sep 11 16:43:08.214642 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:08.214618 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda269739e_353a_4b14_ae8c_05de63045f05.slice/crio-355ecaaa4cde0e7baee5448b6418b0dbc2400ca79520a3d6d17f166b4c7fbdbb WatchSource:0}: Error finding container 355ecaaa4cde0e7baee5448b6418b0dbc2400ca79520a3d6d17f166b4c7fbdbb: Status 404 returned error can't find the container with id 355ecaaa4cde0e7baee5448b6418b0dbc2400ca79520a3d6d17f166b4c7fbdbb Sep 11 16:43:08.215600 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:08.215542 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1a9f06_4b32_45eb_8838_43f5d0649c40.slice/crio-2b7facb3331fe67b2abf232a9ace433521675cf59f084fd17aa1eb0347167422 WatchSource:0}: Error finding container 2b7facb3331fe67b2abf232a9ace433521675cf59f084fd17aa1eb0347167422: Status 404 returned error can't find the container with id 2b7facb3331fe67b2abf232a9ace433521675cf59f084fd17aa1eb0347167422 Sep 11 16:43:08.216306 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:08.216292 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e4db82_8def_4848_983d_03de080ae127.slice/crio-475390717a471edbcbf9f76abcd5bc52f6417358d52de0bd17ba1ab5392fc959 WatchSource:0}: Error finding container 475390717a471edbcbf9f76abcd5bc52f6417358d52de0bd17ba1ab5392fc959: Status 404 returned error can't find the container with id 475390717a471edbcbf9f76abcd5bc52f6417358d52de0bd17ba1ab5392fc959 Sep 11 16:43:08.218325 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:08.218272 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef68ac95_3add_420a_85d4_31cef2cddd8a.slice/crio-8faad07bc9219c5e90156cc3636a0f4c596ee6dc8f123780a647efa70ae2b62e WatchSource:0}: Error finding container 8faad07bc9219c5e90156cc3636a0f4c596ee6dc8f123780a647efa70ae2b62e: Status 404 returned error can't find the container with id 8faad07bc9219c5e90156cc3636a0f4c596ee6dc8f123780a647efa70ae2b62e Sep 11 16:43:08.590658 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:08.590556 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs\") pod \"network-metrics-daemon-68tln\" (UID: \"f1193c55-865c-4fa7-aabb-9ebbdadd11d2\") " pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:08.590658 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:08.590653 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret\") pod \"global-pull-secret-syncer-kx9qk\" (UID: \"9c31fb9a-1977-49b8-8396-c462b1d9f47f\") " pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:08.590852 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:08.590728 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 16:43:08.590852 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:08.590743 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 16:43:08.590852 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:08.590806 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs podName:f1193c55-865c-4fa7-aabb-9ebbdadd11d2 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:10.590785774 +0000 UTC m=+5.998888040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs") pod "network-metrics-daemon-68tln" (UID: "f1193c55-865c-4fa7-aabb-9ebbdadd11d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 16:43:08.590852 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:08.590829 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret podName:9c31fb9a-1977-49b8-8396-c462b1d9f47f nodeName:}" failed. No retries permitted until 2026-09-11 16:43:10.590818809 +0000 UTC m=+5.998921059 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret") pod "global-pull-secret-syncer-kx9qk" (UID: "9c31fb9a-1977-49b8-8396-c462b1d9f47f") : object "kube-system"/"original-pull-secret" not registered Sep 11 16:43:08.691565 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:08.691521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwqq2\" (UniqueName: \"kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2\") pod \"network-check-target-7vclm\" (UID: \"592424fa-0fb1-4974-8d4a-097697d013c0\") " pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:08.691738 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:08.691718 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 16:43:08.691794 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:08.691744 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 16:43:08.691794 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:08.691754 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gwqq2 for pod openshift-network-diagnostics/network-check-target-7vclm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 16:43:08.691861 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:08.691813 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2 podName:592424fa-0fb1-4974-8d4a-097697d013c0 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:10.691795041 +0000 UTC m=+6.099897300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwqq2" (UniqueName: "kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2") pod "network-check-target-7vclm" (UID: "592424fa-0fb1-4974-8d4a-097697d013c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 16:43:09.020284 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:09.020167 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 16:38:07 +0000 UTC" deadline="2028-03-10 05:06:08.219758392 +0000 UTC" Sep 11 16:43:09.020284 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:09.020234 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13092h22m59.199528342s" Sep 11 16:43:09.088677 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:09.088644 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:09.088861 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:09.088651 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:09.088861 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:09.088771 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vclm" podUID="592424fa-0fb1-4974-8d4a-097697d013c0" Sep 11 16:43:09.088861 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:09.088784 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:09.088861 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:09.088854 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kx9qk" podUID="9c31fb9a-1977-49b8-8396-c462b1d9f47f" Sep 11 16:43:09.089071 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:09.088933 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68tln" podUID="f1193c55-865c-4fa7-aabb-9ebbdadd11d2" Sep 11 16:43:09.118519 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:09.118114 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-165.ec2.internal" event={"ID":"a3752174baf855d01c4a6df678c4937e","Type":"ContainerStarted","Data":"9e862e9000b5d1371790a7b1314fcac8d45b3704401c88e6ccbe201ea0b0f140"} Sep 11 16:43:09.124418 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:09.124331 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7dfv" event={"ID":"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2","Type":"ContainerStarted","Data":"dacf67ed25e6263625f98ae8339afc944c6ffff614d32c6d3ce9fcec7a7d3daf"} Sep 11 16:43:09.134994 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:09.134959 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" event={"ID":"9a1a9f06-4b32-45eb-8838-43f5d0649c40","Type":"ContainerStarted","Data":"2b7facb3331fe67b2abf232a9ace433521675cf59f084fd17aa1eb0347167422"} Sep 11 16:43:09.142935 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:09.142851 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qrzv2" event={"ID":"e562cd0c-8abc-4d38-96d5-2960fb624672","Type":"ContainerStarted","Data":"a443e7816fbad5c0baca38b6dcde5caa8a05a643f2c6cfa9da92a32b79a7218a"} Sep 11 16:43:09.149688 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:09.149075 2577 generic.go:356] "Generic (PLEG): container finished" podID="81e835015ef18fd79319f1af18d7d65f" containerID="3e1c3cba7a8ec034198d3bbc8d9b3471a984f0bd717a884e6f4a0e4100cbbfb1" exitCode=0 Sep 11 16:43:09.149688 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:09.149163 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal" event={"ID":"81e835015ef18fd79319f1af18d7d65f","Type":"ContainerDied","Data":"3e1c3cba7a8ec034198d3bbc8d9b3471a984f0bd717a884e6f4a0e4100cbbfb1"} Sep 11 16:43:09.157398 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:09.157368 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r4sbm" event={"ID":"ef68ac95-3add-420a-85d4-31cef2cddd8a","Type":"ContainerStarted","Data":"8faad07bc9219c5e90156cc3636a0f4c596ee6dc8f123780a647efa70ae2b62e"} Sep 11 16:43:09.167574 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:09.167214 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-165.ec2.internal" podStartSLOduration=3.167194764 podStartE2EDuration="3.167194764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:09.137936614 +0000 UTC m=+4.546038884" watchObservedRunningTime="2026-09-11 16:43:09.167194764 +0000 UTC m=+4.575297034" Sep 11 16:43:09.169089 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:09.169070 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4ptxv" event={"ID":"c11370ff-491d-4494-861c-090e05beda2c","Type":"ContainerStarted","Data":"02f224f0bf3cceb31835ede3fed41c478e6140f59e912a6138febdb0802b936b"} Sep 11 16:43:09.174370 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:09.174341 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-trfbj" event={"ID":"a9e4db82-8def-4848-983d-03de080ae127","Type":"ContainerStarted","Data":"475390717a471edbcbf9f76abcd5bc52f6417358d52de0bd17ba1ab5392fc959"} Sep 11 16:43:09.176698 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:09.176645 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vrc4d" event={"ID":"cd648125-8d5e-455b-95bf-5834c01e4db6","Type":"ContainerStarted","Data":"006372f298e625ac3cb141ea952be99f0c1ea40f283d385be5477b8c726cd28c"} Sep 11 16:43:09.182682 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:09.182656 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zgzsg" event={"ID":"a269739e-353a-4b14-ae8c-05de63045f05","Type":"ContainerStarted","Data":"355ecaaa4cde0e7baee5448b6418b0dbc2400ca79520a3d6d17f166b4c7fbdbb"} Sep 11 16:43:09.185852 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:09.185828 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" event={"ID":"257ca13e-8a75-4158-9b2c-5842fefa32bf","Type":"ContainerStarted","Data":"17fcf161f1d604bfd8f3b620b64440687f6bfb050dd55f7c5be96a92c49be5bf"} Sep 11 16:43:10.213536 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:10.213435 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal" event={"ID":"81e835015ef18fd79319f1af18d7d65f","Type":"ContainerStarted","Data":"8a743c8576e7548512b5d2aa923f857ab43d577dc5194d1967d8518488558ce7"} Sep 11 16:43:10.610080 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:10.609982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret\") pod \"global-pull-secret-syncer-kx9qk\" (UID: \"9c31fb9a-1977-49b8-8396-c462b1d9f47f\") " pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:10.610080 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:10.610054 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs\") pod \"network-metrics-daemon-68tln\" (UID: \"f1193c55-865c-4fa7-aabb-9ebbdadd11d2\") " pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:10.610339 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:10.610297 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 16:43:10.610409 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:10.610362 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret podName:9c31fb9a-1977-49b8-8396-c462b1d9f47f nodeName:}" failed. No retries permitted until 2026-09-11 16:43:14.610344124 +0000 UTC m=+10.018446385 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret") pod "global-pull-secret-syncer-kx9qk" (UID: "9c31fb9a-1977-49b8-8396-c462b1d9f47f") : object "kube-system"/"original-pull-secret" not registered Sep 11 16:43:10.610799 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:10.610779 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 16:43:10.610901 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:10.610856 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs podName:f1193c55-865c-4fa7-aabb-9ebbdadd11d2 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:14.610839512 +0000 UTC m=+10.018941762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs") pod "network-metrics-daemon-68tln" (UID: "f1193c55-865c-4fa7-aabb-9ebbdadd11d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 16:43:10.711276 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:10.711240 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwqq2\" (UniqueName: \"kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2\") pod \"network-check-target-7vclm\" (UID: \"592424fa-0fb1-4974-8d4a-097697d013c0\") " pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:10.711475 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:10.711415 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 16:43:10.711475 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:10.711435 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 16:43:10.711475 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:10.711448 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gwqq2 for pod openshift-network-diagnostics/network-check-target-7vclm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 16:43:10.711703 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:10.711546 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2 podName:592424fa-0fb1-4974-8d4a-097697d013c0 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:14.711488365 +0000 UTC m=+10.119590619 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwqq2" (UniqueName: "kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2") pod "network-check-target-7vclm" (UID: "592424fa-0fb1-4974-8d4a-097697d013c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 16:43:11.088702 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:11.088597 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:11.088859 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:11.088609 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:11.088859 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:11.088737 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kx9qk" podUID="9c31fb9a-1977-49b8-8396-c462b1d9f47f" Sep 11 16:43:11.088859 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:11.088831 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68tln" podUID="f1193c55-865c-4fa7-aabb-9ebbdadd11d2" Sep 11 16:43:11.089026 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:11.088872 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:11.089026 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:11.088924 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vclm" podUID="592424fa-0fb1-4974-8d4a-097697d013c0" Sep 11 16:43:13.089196 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:13.089160 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:13.089196 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:13.089203 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:13.089967 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:13.089236 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:13.089967 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:13.089353 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kx9qk" podUID="9c31fb9a-1977-49b8-8396-c462b1d9f47f" Sep 11 16:43:13.089967 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:13.089479 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vclm" podUID="592424fa-0fb1-4974-8d4a-097697d013c0" Sep 11 16:43:13.089967 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:13.089637 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68tln" podUID="f1193c55-865c-4fa7-aabb-9ebbdadd11d2" Sep 11 16:43:14.644050 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:14.643530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret\") pod \"global-pull-secret-syncer-kx9qk\" (UID: \"9c31fb9a-1977-49b8-8396-c462b1d9f47f\") " pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:14.644050 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:14.643596 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs\") pod \"network-metrics-daemon-68tln\" (UID: \"f1193c55-865c-4fa7-aabb-9ebbdadd11d2\") " pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:14.644050 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:14.643755 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 16:43:14.644050 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:14.643813 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs podName:f1193c55-865c-4fa7-aabb-9ebbdadd11d2 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:22.643794861 +0000 UTC m=+18.051897122 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs") pod "network-metrics-daemon-68tln" (UID: "f1193c55-865c-4fa7-aabb-9ebbdadd11d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 16:43:14.644698 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:14.644128 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 16:43:14.644698 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:14.644179 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret podName:9c31fb9a-1977-49b8-8396-c462b1d9f47f nodeName:}" failed. No retries permitted until 2026-09-11 16:43:22.644162891 +0000 UTC m=+18.052265148 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret") pod "global-pull-secret-syncer-kx9qk" (UID: "9c31fb9a-1977-49b8-8396-c462b1d9f47f") : object "kube-system"/"original-pull-secret" not registered Sep 11 16:43:14.744129 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:14.744095 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwqq2\" (UniqueName: \"kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2\") pod \"network-check-target-7vclm\" (UID: \"592424fa-0fb1-4974-8d4a-097697d013c0\") " pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:14.744301 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:14.744239 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 16:43:14.744301 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:14.744258 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 16:43:14.744301 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:14.744270 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gwqq2 for pod openshift-network-diagnostics/network-check-target-7vclm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 16:43:14.744553 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:14.744330 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2 podName:592424fa-0fb1-4974-8d4a-097697d013c0 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:22.744311232 +0000 UTC m=+18.152413499 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwqq2" (UniqueName: "kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2") pod "network-check-target-7vclm" (UID: "592424fa-0fb1-4974-8d4a-097697d013c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 16:43:15.088776 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:15.088685 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:15.088943 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:15.088862 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68tln" podUID="f1193c55-865c-4fa7-aabb-9ebbdadd11d2" Sep 11 16:43:15.088943 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:15.088910 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:15.089068 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:15.088979 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vclm" podUID="592424fa-0fb1-4974-8d4a-097697d013c0" Sep 11 16:43:15.089068 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:15.089022 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:15.089157 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:15.089092 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kx9qk" podUID="9c31fb9a-1977-49b8-8396-c462b1d9f47f" Sep 11 16:43:17.087977 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:17.087895 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:17.088449 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:17.088030 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vclm" podUID="592424fa-0fb1-4974-8d4a-097697d013c0" Sep 11 16:43:17.088449 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:17.088044 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:17.088449 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:17.088134 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kx9qk" podUID="9c31fb9a-1977-49b8-8396-c462b1d9f47f" Sep 11 16:43:17.088449 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:17.088163 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:17.088449 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:17.088244 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68tln" podUID="f1193c55-865c-4fa7-aabb-9ebbdadd11d2" Sep 11 16:43:19.087593 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:19.087551 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:19.088040 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:19.087551 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:19.088040 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:19.087681 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kx9qk" podUID="9c31fb9a-1977-49b8-8396-c462b1d9f47f" Sep 11 16:43:19.088040 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:19.087790 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68tln" podUID="f1193c55-865c-4fa7-aabb-9ebbdadd11d2" Sep 11 16:43:19.088040 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:19.087557 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:19.088040 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:19.087882 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vclm" podUID="592424fa-0fb1-4974-8d4a-097697d013c0" Sep 11 16:43:21.088118 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:21.087901 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:21.088597 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:21.087901 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:21.088597 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:21.088228 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68tln" podUID="f1193c55-865c-4fa7-aabb-9ebbdadd11d2" Sep 11 16:43:21.088597 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:21.087933 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:21.088597 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:21.088264 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kx9qk" podUID="9c31fb9a-1977-49b8-8396-c462b1d9f47f" Sep 11 16:43:21.088597 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:21.088372 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vclm" podUID="592424fa-0fb1-4974-8d4a-097697d013c0" Sep 11 16:43:22.701616 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:22.701582 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret\") pod \"global-pull-secret-syncer-kx9qk\" (UID: \"9c31fb9a-1977-49b8-8396-c462b1d9f47f\") " pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:22.702096 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:22.701627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs\") pod \"network-metrics-daemon-68tln\" (UID: \"f1193c55-865c-4fa7-aabb-9ebbdadd11d2\") " pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:22.702096 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:22.701766 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 16:43:22.702096 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:22.701841 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret podName:9c31fb9a-1977-49b8-8396-c462b1d9f47f nodeName:}" failed. No retries permitted until 2026-09-11 16:43:38.701820771 +0000 UTC m=+34.109923026 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret") pod "global-pull-secret-syncer-kx9qk" (UID: "9c31fb9a-1977-49b8-8396-c462b1d9f47f") : object "kube-system"/"original-pull-secret" not registered Sep 11 16:43:22.702096 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:22.701766 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 16:43:22.702096 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:22.701928 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs podName:f1193c55-865c-4fa7-aabb-9ebbdadd11d2 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:38.701909061 +0000 UTC m=+34.110011315 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs") pod "network-metrics-daemon-68tln" (UID: "f1193c55-865c-4fa7-aabb-9ebbdadd11d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 16:43:22.802259 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:22.802227 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwqq2\" (UniqueName: \"kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2\") pod \"network-check-target-7vclm\" (UID: \"592424fa-0fb1-4974-8d4a-097697d013c0\") " pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:22.802433 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:22.802389 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 16:43:22.802433 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:22.802408 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 16:43:22.802433 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:22.802421 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gwqq2 for pod openshift-network-diagnostics/network-check-target-7vclm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 16:43:22.802589 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:22.802468 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2 podName:592424fa-0fb1-4974-8d4a-097697d013c0 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:38.802455677 +0000 UTC m=+34.210557927 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwqq2" (UniqueName: "kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2") pod "network-check-target-7vclm" (UID: "592424fa-0fb1-4974-8d4a-097697d013c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 16:43:23.088077 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:23.087995 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:23.088077 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:23.088044 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:23.088302 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:23.087996 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:23.088302 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:23.088145 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68tln" podUID="f1193c55-865c-4fa7-aabb-9ebbdadd11d2" Sep 11 16:43:23.088302 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:23.088256 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vclm" podUID="592424fa-0fb1-4974-8d4a-097697d013c0" Sep 11 16:43:23.088451 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:23.088337 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kx9qk" podUID="9c31fb9a-1977-49b8-8396-c462b1d9f47f" Sep 11 16:43:25.088661 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:25.088613 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:25.089045 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:25.088719 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kx9qk" podUID="9c31fb9a-1977-49b8-8396-c462b1d9f47f" Sep 11 16:43:25.089045 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:25.088796 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:25.089045 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:25.088894 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68tln" podUID="f1193c55-865c-4fa7-aabb-9ebbdadd11d2" Sep 11 16:43:25.089045 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:25.088932 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:25.089045 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:25.088996 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vclm" podUID="592424fa-0fb1-4974-8d4a-097697d013c0" Sep 11 16:43:26.251969 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:26.251785 2577 generic.go:356] "Generic (PLEG): container finished" podID="9052d32a-f4da-4ace-9da5-6a63d6e7e0c2" containerID="fddfabe67a90e40f5ce5e7dbb0184229db42964a06f622dd9850007ec2e65a81" exitCode=0 Sep 11 16:43:26.252611 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:26.251865 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7dfv" event={"ID":"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2","Type":"ContainerDied","Data":"fddfabe67a90e40f5ce5e7dbb0184229db42964a06f622dd9850007ec2e65a81"} Sep 11 16:43:26.254632 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:26.254610 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" event={"ID":"9a1a9f06-4b32-45eb-8838-43f5d0649c40","Type":"ContainerStarted","Data":"cbff1231ddaa7ae5f3bb2a16124d1b9aa97731c4960f04f1116018a04ed82687"} Sep 11 16:43:26.254753 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:26.254687 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" event={"ID":"9a1a9f06-4b32-45eb-8838-43f5d0649c40","Type":"ContainerStarted","Data":"6526106bb330732c951c0864c1f2abe000a751eb5d66842877eab8df7fcc1fca"} Sep 11 16:43:26.254753 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:26.254739 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" event={"ID":"9a1a9f06-4b32-45eb-8838-43f5d0649c40","Type":"ContainerStarted","Data":"bf9f62fa7b9da51f36224c5ace722c370da7ef42061c86005204f5eb209caca6"} Sep 11 16:43:26.254853 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:26.254787 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" event={"ID":"9a1a9f06-4b32-45eb-8838-43f5d0649c40","Type":"ContainerStarted","Data":"88cb4567d8e0766e518b96ebf5752bb9ceaf92df3cf9d3f3fb392af86335f155"} Sep 11 16:43:26.256076 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:26.256040 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r4sbm" event={"ID":"ef68ac95-3add-420a-85d4-31cef2cddd8a","Type":"ContainerStarted","Data":"3de819778d5a694ab88619669475f137968f80d17d3f698511b126ddb9acfc35"} Sep 11 16:43:26.257441 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:26.257420 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4ptxv" event={"ID":"c11370ff-491d-4494-861c-090e05beda2c","Type":"ContainerStarted","Data":"db8510805c48d78bbbe1f8a238d64ef55e983a183a02b6fb6318e7451aa259da"} Sep 11 16:43:26.258896 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:26.258878 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-trfbj" event={"ID":"a9e4db82-8def-4848-983d-03de080ae127","Type":"ContainerStarted","Data":"ecc399199458dac5b85ea2bfc108cab864a220c376f1efd67e90b5a2a4815b34"} Sep 11 16:43:26.260264 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:26.260245 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vrc4d" event={"ID":"cd648125-8d5e-455b-95bf-5834c01e4db6","Type":"ContainerStarted","Data":"8b2fa17a5dd044f4dd60a6cd940db152e8742535470a79d1a5a3c722ee737a50"} Sep 11 16:43:26.261609 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:26.261584 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zgzsg" event={"ID":"a269739e-353a-4b14-ae8c-05de63045f05","Type":"ContainerStarted","Data":"8a9b6672b4cb97a0fb2a2b4841ba6b70085dba04d9e1d60a5cb77f240f4917d1"} Sep 11 16:43:26.263003 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:26.262984 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" event={"ID":"257ca13e-8a75-4158-9b2c-5842fefa32bf","Type":"ContainerStarted","Data":"50d62195341746f75d3f175b1fbdd63ffee568ef4aa37356fdfdcf167f4c185b"} Sep 11 16:43:26.276604 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:26.276562 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-165.ec2.internal" podStartSLOduration=20.276550685 podStartE2EDuration="20.276550685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:10.235855877 +0000 UTC m=+5.643958165" watchObservedRunningTime="2026-09-11 16:43:26.276550685 +0000 UTC m=+21.684652954" Sep 11 16:43:26.310469 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:26.310380 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4ptxv" podStartSLOduration=7.599268859 podStartE2EDuration="21.310364238s" totalImagesPullingTime="13.711095379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:26.292182445 +0000 UTC m=+21.700284714" watchObservedRunningTime="2026-09-11 16:43:26.310364238 +0000 UTC m=+21.718466486" Sep 11 16:43:26.310617 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:26.310556 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zgzsg" podStartSLOduration=3.6549062660000002 podStartE2EDuration="21.310548621s" totalImagesPullingTime="17.655642355s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:26.310433705 +0000 UTC m=+21.718535986" watchObservedRunningTime="2026-09-11 16:43:26.310548621 +0000 UTC m=+21.718650889" Sep 11 16:43:26.326509 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:26.326455 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r4sbm" podStartSLOduration=7.612137231 podStartE2EDuration="21.326439868s" totalImagesPullingTime="13.714302637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:26.32606406 +0000 UTC m=+21.734166350" watchObservedRunningTime="2026-09-11 16:43:26.326439868 +0000 UTC m=+21.734542134" Sep 11 16:43:26.343277 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:26.343239 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vrc4d" podStartSLOduration=7.621157411 podStartE2EDuration="21.343226657s" totalImagesPullingTime="13.722069246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:26.342752696 +0000 UTC m=+21.750854964" watchObservedRunningTime="2026-09-11 16:43:26.343226657 +0000 UTC m=+21.751328926" Sep 11 16:43:26.361675 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:26.361638 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-trfbj" podStartSLOduration=3.766580203 podStartE2EDuration="21.361624551s" totalImagesPullingTime="17.595044348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:26.361591311 +0000 UTC m=+21.769693599" watchObservedRunningTime="2026-09-11 16:43:26.361624551 +0000 UTC m=+21.769726819" Sep 11 16:43:27.091097 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:27.091022 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:27.091097 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:27.091041 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:27.091305 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:27.091131 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:27.091305 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:27.091141 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vclm" podUID="592424fa-0fb1-4974-8d4a-097697d013c0" Sep 11 16:43:27.091305 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:27.091257 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68tln" podUID="f1193c55-865c-4fa7-aabb-9ebbdadd11d2" Sep 11 16:43:27.091433 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:27.091335 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kx9qk" podUID="9c31fb9a-1977-49b8-8396-c462b1d9f47f" Sep 11 16:43:27.271576 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:27.271530 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" event={"ID":"9a1a9f06-4b32-45eb-8838-43f5d0649c40","Type":"ContainerStarted","Data":"d0514ca01b6b00ae917727f1624e95f3e4260195b85121bd045f1d47d63712a8"} Sep 11 16:43:27.272034 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:27.271638 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" event={"ID":"9a1a9f06-4b32-45eb-8838-43f5d0649c40","Type":"ContainerStarted","Data":"16cb0f651184e95b16cda70d6f538bdb16b30a6c7ea890ff263a73eca3571ddf"} Sep 11 16:43:27.273630 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:27.273605 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 16:43:27.273780 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:27.273753 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qrzv2" event={"ID":"e562cd0c-8abc-4d38-96d5-2960fb624672","Type":"ContainerStarted","Data":"676889d275a44b9aa6a3b737ec0bdfedc2698a0eda7d364ce1b0fced611e1e29"} Sep 11 16:43:28.018803 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:28.018646 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T16:43:27.273624375Z","UUID":"41d2c4f4-b091-4be6-a758-ebc1d7aa1fa4","Handler":null,"Name":"","Endpoint":""} Sep 11 16:43:28.020449 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:28.020283 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 16:43:28.020597 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:28.020465 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 16:43:28.281083 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:28.281010 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" event={"ID":"257ca13e-8a75-4158-9b2c-5842fefa32bf","Type":"ContainerStarted","Data":"d4542f1516f3dcfa8baf30059a2c0006e3c785572d1457f119008c6524ea9e42"} Sep 11 16:43:28.938788 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:28.938707 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vrc4d" Sep 11 16:43:28.939521 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:28.939481 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vrc4d" Sep 11 16:43:28.971190 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:28.971142 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qrzv2" podStartSLOduration=6.392494613 podStartE2EDuration="23.971125718s" totalImagesPullingTime="17.578631105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:27.291065593 +0000 UTC m=+22.699167883" watchObservedRunningTime="2026-09-11 16:43:28.971125718 +0000 UTC m=+24.379227987" Sep 11 16:43:29.088325 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:29.088293 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:29.088523 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:29.088293 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:29.088523 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:29.088416 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vclm" podUID="592424fa-0fb1-4974-8d4a-097697d013c0" Sep 11 16:43:29.088660 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:29.088545 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68tln" podUID="f1193c55-865c-4fa7-aabb-9ebbdadd11d2" Sep 11 16:43:29.088660 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:29.088293 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:29.088660 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:29.088613 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kx9qk" podUID="9c31fb9a-1977-49b8-8396-c462b1d9f47f" Sep 11 16:43:29.285570 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:29.285464 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" event={"ID":"257ca13e-8a75-4158-9b2c-5842fefa32bf","Type":"ContainerStarted","Data":"f5684b9e3accdf4938131be3f5dfc51dc2f481750989bce2c88904aa81ec1f04"} Sep 11 16:43:29.288752 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:29.288726 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" event={"ID":"9a1a9f06-4b32-45eb-8838-43f5d0649c40","Type":"ContainerStarted","Data":"056ede1a0416393ab8c5d1a9355612b03098ca8ef805ba6c1465322cf80e9493"} Sep 11 16:43:29.310777 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:29.310727 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nwt5" podStartSLOduration=4.429328448 podStartE2EDuration="24.310698117s" totalImagesPullingTime="19.881369669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:43:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:29.31061082 +0000 UTC m=+24.718713109" watchObservedRunningTime="2026-09-11 16:43:29.310698117 +0000 UTC m=+24.718800389" Sep 11 16:43:31.087987 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:31.087896 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:31.088421 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:31.088033 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:31.088421 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:31.088040 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vclm" podUID="592424fa-0fb1-4974-8d4a-097697d013c0" Sep 11 16:43:31.088421 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:31.088167 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68tln" podUID="f1193c55-865c-4fa7-aabb-9ebbdadd11d2" Sep 11 16:43:31.088421 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:31.088220 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:31.088421 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:31.088297 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kx9qk" podUID="9c31fb9a-1977-49b8-8396-c462b1d9f47f" Sep 11 16:43:32.297189 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:32.296948 2577 generic.go:356] "Generic (PLEG): container finished" podID="9052d32a-f4da-4ace-9da5-6a63d6e7e0c2" containerID="f1ab7994229afb24bb5b0424b3cd919b785a85177538904f3d723f2e7034faa3" exitCode=0 Sep 11 16:43:32.297189 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:32.297032 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7dfv" event={"ID":"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2","Type":"ContainerDied","Data":"f1ab7994229afb24bb5b0424b3cd919b785a85177538904f3d723f2e7034faa3"} Sep 11 16:43:32.301057 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:32.301032 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" event={"ID":"9a1a9f06-4b32-45eb-8838-43f5d0649c40","Type":"ContainerStarted","Data":"68aeb801be1b152516f87b4697396a7c3d27bc070c9ddf1527c67731bc8cd5c5"} Sep 11 16:43:32.301296 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:32.301278 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:32.301379 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:32.301351 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:32.316143 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:32.316118 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:32.316298 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:32.316284 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:32.364153 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:32.364105 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" podStartSLOduration=9.720478775 podStartE2EDuration="27.364089582s" totalImagesPullingTime="17.643610807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:32.363630416 +0000 UTC m=+27.771732704" watchObservedRunningTime="2026-09-11 16:43:32.364089582 +0000 UTC m=+27.772191851" Sep 11 16:43:33.088254 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:33.088220 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:33.088254 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:33.088239 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:33.088421 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:33.088226 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:33.088421 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:33.088359 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kx9qk" podUID="9c31fb9a-1977-49b8-8396-c462b1d9f47f" Sep 11 16:43:33.088528 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:33.088491 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68tln" podUID="f1193c55-865c-4fa7-aabb-9ebbdadd11d2" Sep 11 16:43:33.088586 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:33.088572 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vclm" podUID="592424fa-0fb1-4974-8d4a-097697d013c0" Sep 11 16:43:33.303801 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:33.303771 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 16:43:33.728415 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:33.728383 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-68tln"] Sep 11 16:43:33.728549 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:33.728528 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:33.728680 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:33.728652 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68tln" podUID="f1193c55-865c-4fa7-aabb-9ebbdadd11d2" Sep 11 16:43:33.731154 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:33.731127 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kx9qk"] Sep 11 16:43:33.731275 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:33.731204 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:33.731337 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:33.731275 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kx9qk" podUID="9c31fb9a-1977-49b8-8396-c462b1d9f47f" Sep 11 16:43:33.735941 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:33.735922 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7vclm"] Sep 11 16:43:33.736008 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:33.735992 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:33.736080 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:33.736065 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vclm" podUID="592424fa-0fb1-4974-8d4a-097697d013c0" Sep 11 16:43:34.307799 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:34.307768 2577 generic.go:356] "Generic (PLEG): container finished" podID="9052d32a-f4da-4ace-9da5-6a63d6e7e0c2" containerID="a51fa8e6915805780ff8cd4b7e3b04cd635c112c5d0c0674daa2306c1c2e3368" exitCode=0 Sep 11 16:43:34.308528 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:34.307848 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7dfv" event={"ID":"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2","Type":"ContainerDied","Data":"a51fa8e6915805780ff8cd4b7e3b04cd635c112c5d0c0674daa2306c1c2e3368"} Sep 11 16:43:34.308528 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:34.307928 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 16:43:35.088996 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:35.088911 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:35.089153 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:35.089026 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kx9qk" podUID="9c31fb9a-1977-49b8-8396-c462b1d9f47f" Sep 11 16:43:35.089153 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:35.089076 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:35.089254 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:35.089189 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vclm" podUID="592424fa-0fb1-4974-8d4a-097697d013c0" Sep 11 16:43:35.989743 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:35.989559 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vrc4d" Sep 11 16:43:35.990143 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:35.989844 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 16:43:35.990143 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:35.990123 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vrc4d" Sep 11 16:43:36.087452 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:36.087424 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:36.087634 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:36.087581 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68tln" podUID="f1193c55-865c-4fa7-aabb-9ebbdadd11d2" Sep 11 16:43:36.314556 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:36.314455 2577 generic.go:356] "Generic (PLEG): container finished" podID="9052d32a-f4da-4ace-9da5-6a63d6e7e0c2" containerID="7826ae36478dc556e3b99ab3f13cf7b7bcb43a83b641a336cc09378a3b01354d" exitCode=0 Sep 11 16:43:36.314556 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:36.314534 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7dfv" event={"ID":"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2","Type":"ContainerDied","Data":"7826ae36478dc556e3b99ab3f13cf7b7bcb43a83b641a336cc09378a3b01354d"} Sep 11 16:43:36.855821 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:36.855785 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:43:36.856076 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:36.856019 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 16:43:36.865976 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:36.865938 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" podUID="9a1a9f06-4b32-45eb-8838-43f5d0649c40" containerName="ovnkube-controller" probeResult="failure" output="" Sep 11 16:43:36.874611 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:36.874578 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" podUID="9a1a9f06-4b32-45eb-8838-43f5d0649c40" containerName="ovnkube-controller" probeResult="failure" output="" Sep 11 16:43:37.087567 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:37.087532 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:37.087911 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:37.087583 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:37.087911 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:37.087677 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kx9qk" podUID="9c31fb9a-1977-49b8-8396-c462b1d9f47f" Sep 11 16:43:37.087911 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:37.087771 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vclm" podUID="592424fa-0fb1-4974-8d4a-097697d013c0" Sep 11 16:43:38.087481 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:38.087446 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:38.087679 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:38.087583 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68tln" podUID="f1193c55-865c-4fa7-aabb-9ebbdadd11d2" Sep 11 16:43:38.714512 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:38.714468 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs\") pod \"network-metrics-daemon-68tln\" (UID: \"f1193c55-865c-4fa7-aabb-9ebbdadd11d2\") " pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:38.714690 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:38.714565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret\") pod \"global-pull-secret-syncer-kx9qk\" (UID: \"9c31fb9a-1977-49b8-8396-c462b1d9f47f\") " pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:38.714690 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:38.714627 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 16:43:38.714690 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:38.714665 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 16:43:38.714810 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:38.714694 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs podName:f1193c55-865c-4fa7-aabb-9ebbdadd11d2 nodeName:}" failed. No retries permitted until 2026-09-11 16:44:10.7146768 +0000 UTC m=+66.122779050 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs") pod "network-metrics-daemon-68tln" (UID: "f1193c55-865c-4fa7-aabb-9ebbdadd11d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 16:43:38.714810 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:38.714711 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret podName:9c31fb9a-1977-49b8-8396-c462b1d9f47f nodeName:}" failed. No retries permitted until 2026-09-11 16:44:10.714699966 +0000 UTC m=+66.122802213 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret") pod "global-pull-secret-syncer-kx9qk" (UID: "9c31fb9a-1977-49b8-8396-c462b1d9f47f") : object "kube-system"/"original-pull-secret" not registered Sep 11 16:43:38.815856 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:38.815820 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwqq2\" (UniqueName: \"kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2\") pod \"network-check-target-7vclm\" (UID: \"592424fa-0fb1-4974-8d4a-097697d013c0\") " pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:38.816025 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:38.815995 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 16:43:38.816025 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:38.816023 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 16:43:38.816115 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:38.816037 2577 projected.go:196] Error preparing data for projected volume kube-api-access-gwqq2 for pod openshift-network-diagnostics/network-check-target-7vclm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 16:43:38.816115 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:38.816102 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2 podName:592424fa-0fb1-4974-8d4a-097697d013c0 nodeName:}" failed. No retries permitted until 2026-09-11 16:44:10.816082666 +0000 UTC m=+66.224184932 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwqq2" (UniqueName: "kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2") pod "network-check-target-7vclm" (UID: "592424fa-0fb1-4974-8d4a-097697d013c0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 16:43:39.087936 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.087900 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:39.088325 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.087909 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:39.088325 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:39.088032 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vclm" podUID="592424fa-0fb1-4974-8d4a-097697d013c0" Sep 11 16:43:39.088325 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:39.088077 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kx9qk" podUID="9c31fb9a-1977-49b8-8396-c462b1d9f47f" Sep 11 16:43:39.872831 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.872798 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-165.ec2.internal" event="NodeReady" Sep 11 16:43:39.873010 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.872962 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 16:43:39.920968 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.920929 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7669dcd8d-whgmr"] Sep 11 16:43:39.924168 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.924077 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:39.925821 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.925764 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c679445b8-dxcgg"] Sep 11 16:43:39.929463 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.929443 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-fc9sn"] Sep 11 16:43:39.929603 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.929571 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:39.932874 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.932854 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zk4rf"] Sep 11 16:43:39.932994 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.932981 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-fc9sn" Sep 11 16:43:39.936191 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.936171 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zk4rf" Sep 11 16:43:39.936839 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.936814 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-jkhgd"] Sep 11 16:43:39.940059 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.940038 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jkhgd" Sep 11 16:43:39.942713 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.942694 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 11 16:43:39.942821 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.942715 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 11 16:43:39.942821 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.942730 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 16:43:39.942821 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.942734 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qc76w\"" Sep 11 16:43:39.942821 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.942779 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 16:43:39.942821 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.942792 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 11 16:43:39.943064 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.942702 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 11 16:43:39.943064 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.942950 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 16:43:39.943064 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.942997 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 16:43:39.943064 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.943013 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 16:43:39.943064 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.943047 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 11 16:43:39.943283 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.943271 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-d9hsz"] Sep 11 16:43:39.944239 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.944212 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 11 16:43:39.944341 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.944212 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ptp2z\"" Sep 11 16:43:39.944341 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.944211 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 11 16:43:39.944595 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.944578 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rnwnv\"" Sep 11 16:43:39.944678 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.944601 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 11 16:43:39.944853 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.944800 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 11 16:43:39.944853 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.944827 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 11 16:43:39.944853 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.944839 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 11 16:43:39.945067 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.945050 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ml4jp\"" Sep 11 16:43:39.945139 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.945125 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ch27t\"" Sep 11 16:43:39.945231 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.945213 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 11 16:43:39.946264 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.945918 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 11 16:43:39.946264 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.945918 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 11 16:43:39.946264 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.946250 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-tlxcd"] Sep 11 16:43:39.946597 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.946407 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:39.949644 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.949624 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9"] Sep 11 16:43:39.949752 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.949716 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" Sep 11 16:43:39.952797 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.952779 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh"] Sep 11 16:43:39.953042 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.952963 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9" Sep 11 16:43:39.953454 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.953245 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 11 16:43:39.953454 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.953265 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 16:43:39.953637 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.953601 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 16:43:39.953735 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.953672 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vfp7g\"" Sep 11 16:43:39.955061 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.955043 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 11 16:43:39.955813 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.955795 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-jkhgd"] Sep 11 16:43:39.955813 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.955826 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c679445b8-dxcgg"] Sep 11 16:43:39.955970 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.955902 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" Sep 11 16:43:39.957301 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.957096 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 11 16:43:39.958607 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.958586 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 11 16:43:39.958791 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.958776 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 11 16:43:39.959019 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.959002 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 11 16:43:39.959092 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.959028 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 11 16:43:39.959328 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.959309 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 11 16:43:39.969892 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.969864 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh"] Sep 11 16:43:39.970302 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.970233 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 11 16:43:39.970302 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.970244 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 16:43:39.970525 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.970233 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dl9zw\"" Sep 11 16:43:39.970685 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.970611 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-g9whx\"" Sep 11 16:43:39.970685 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.970639 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-twlct\"" Sep 11 16:43:39.970855 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.970703 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 11 16:43:39.970855 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.970770 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-fc9sn"] Sep 11 16:43:39.972216 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.972188 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 16:43:39.974257 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.972618 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 11 16:43:39.974257 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.973104 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 11 16:43:39.974257 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.973124 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7669dcd8d-whgmr"] Sep 11 16:43:39.974257 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.973160 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv"] Sep 11 16:43:39.974257 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.973532 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 11 16:43:39.974257 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.973829 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 16:43:39.977719 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.977341 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59989b64f6-flmxp"] Sep 11 16:43:39.978990 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.978961 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:39.980906 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.980888 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-w5wz5"] Sep 11 16:43:39.981018 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.981000 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59989b64f6-flmxp" Sep 11 16:43:39.982289 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.982232 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 11 16:43:39.982458 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.982438 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 11 16:43:39.982659 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.982639 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 11 16:43:39.982900 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.982881 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 11 16:43:39.983020 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.983001 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 11 16:43:39.983130 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.983113 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 11 16:43:39.983942 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.983921 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 11 16:43:39.984764 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.984353 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 11 16:43:39.984764 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.984693 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f5975844-4n7th"] Sep 11 16:43:39.985063 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.985044 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w5wz5" Sep 11 16:43:39.987839 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.987791 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 16:43:39.988035 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.988013 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7nrk4\"" Sep 11 16:43:39.988535 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.988519 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 16:43:39.988651 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.988636 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 16:43:39.989200 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.989182 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-tlxcd"] Sep 11 16:43:39.989238 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.989224 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-kqg8b"] Sep 11 16:43:39.989290 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.989279 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f5975844-4n7th" Sep 11 16:43:39.992388 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.992358 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9"] Sep 11 16:43:39.992549 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.992396 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-d9hsz"] Sep 11 16:43:39.992549 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.992379 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 11 16:43:39.992549 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.992418 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c4dn7"] Sep 11 16:43:39.992549 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.992479 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-kqg8b" Sep 11 16:43:39.993619 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.993596 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-pz9rf\"" Sep 11 16:43:39.996959 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.995322 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 16:43:39.997239 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:39.995989 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dlxv9\"" Sep 11 16:43:40.002552 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.001837 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w5wz5"] Sep 11 16:43:40.002552 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.001929 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 16:43:40.002552 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.001944 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c4dn7" Sep 11 16:43:40.002552 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.002457 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59989b64f6-flmxp"] Sep 11 16:43:40.004491 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.004461 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zk4rf"] Sep 11 16:43:40.005048 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.005016 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 16:43:40.005360 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.005211 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wbvg9\"" Sep 11 16:43:40.005360 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.005260 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 16:43:40.005360 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.005297 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-kqg8b"] Sep 11 16:43:40.005908 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.005891 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c4dn7"] Sep 11 16:43:40.007748 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.007732 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv"] Sep 11 16:43:40.009047 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.008861 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f5975844-4n7th"] Sep 11 16:43:40.026001 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.025969 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wcxg9\" (UID: \"50af7bf9-0c82-4d40-979b-11350f3018a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9" Sep 11 16:43:40.026128 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026012 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-default-certificate\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:40.026128 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026044 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:40.026128 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026074 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed61a19b-2fa7-466b-9772-3b7a1f4a5d25-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-fc9sn\" (UID: \"ed61a19b-2fa7-466b-9772-3b7a1f4a5d25\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-fc9sn" Sep 11 16:43:40.026128 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026122 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f76030c8-35a4-4b21-8a5d-ac0ffa450ce6-serving-cert\") pod \"insights-operator-5844b856db-d9hsz\" (UID: \"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6\") " pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:40.026300 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/593fb846-0d3d-42dd-8405-6ac2f57bd18f-image-registry-private-configuration\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.026300 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026186 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d41dfd70-3a89-4d1d-84de-d0b3964b8538-serving-cert\") pod \"service-ca-operator-cf7c776b7-jkhgd\" (UID: \"d41dfd70-3a89-4d1d-84de-d0b3964b8538\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jkhgd" Sep 11 16:43:40.026300 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026217 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24l27\" (UniqueName: \"kubernetes.io/projected/13cca68e-a749-4f4e-bbdf-21de50e37c6c-kube-api-access-24l27\") pod \"console-operator-696884fb9c-tlxcd\" (UID: \"13cca68e-a749-4f4e-bbdf-21de50e37c6c\") " pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" Sep 11 16:43:40.026300 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026253 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrjnq\" (UniqueName: \"kubernetes.io/projected/50af7bf9-0c82-4d40-979b-11350f3018a9-kube-api-access-lrjnq\") pod \"cluster-samples-operator-86499c89c4-wcxg9\" (UID: \"50af7bf9-0c82-4d40-979b-11350f3018a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9" Sep 11 16:43:40.026450 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026300 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96f4w\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-kube-api-access-96f4w\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.026450 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/593fb846-0d3d-42dd-8405-6ac2f57bd18f-installation-pull-secrets\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.026450 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026414 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xzrz\" (UniqueName: \"kubernetes.io/projected/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-kube-api-access-7xzrz\") pod \"cluster-monitoring-operator-6cd954b897-rsrzh\" (UID: \"fde3ddc0-fc6a-42c7-aab9-866ca084bd59\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" Sep 11 16:43:40.026450 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026444 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncbzb\" (UniqueName: \"kubernetes.io/projected/ed61a19b-2fa7-466b-9772-3b7a1f4a5d25-kube-api-access-ncbzb\") pod \"kube-storage-version-migrator-operator-6c8f84f897-fc9sn\" (UID: \"ed61a19b-2fa7-466b-9772-3b7a1f4a5d25\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-fc9sn" Sep 11 16:43:40.026646 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026517 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f76030c8-35a4-4b21-8a5d-ac0ffa450ce6-trusted-ca-bundle\") pod \"insights-operator-5844b856db-d9hsz\" (UID: \"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6\") " pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:40.026646 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026566 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-stats-auth\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:40.026646 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026594 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-certificates\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.026646 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026627 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqp6l\" (UniqueName: \"kubernetes.io/projected/d41dfd70-3a89-4d1d-84de-d0b3964b8538-kube-api-access-cqp6l\") pod \"service-ca-operator-cf7c776b7-jkhgd\" (UID: \"d41dfd70-3a89-4d1d-84de-d0b3964b8538\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jkhgd" Sep 11 16:43:40.026782 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026652 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f76030c8-35a4-4b21-8a5d-ac0ffa450ce6-snapshots\") pod \"insights-operator-5844b856db-d9hsz\" (UID: \"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6\") " pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:40.026782 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026724 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-bound-sa-token\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.026782 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026763 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7npvk\" (UniqueName: \"kubernetes.io/projected/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-kube-api-access-7npvk\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:40.026904 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026808 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/13cca68e-a749-4f4e-bbdf-21de50e37c6c-config\") pod \"console-operator-696884fb9c-tlxcd\" (UID: \"13cca68e-a749-4f4e-bbdf-21de50e37c6c\") " pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" Sep 11 16:43:40.026904 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026835 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d41dfd70-3a89-4d1d-84de-d0b3964b8538-config\") pod \"service-ca-operator-cf7c776b7-jkhgd\" (UID: \"d41dfd70-3a89-4d1d-84de-d0b3964b8538\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jkhgd" Sep 11 16:43:40.026904 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026875 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/593fb846-0d3d-42dd-8405-6ac2f57bd18f-trusted-ca\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.027005 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026902 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rsrzh\" (UID: \"fde3ddc0-fc6a-42c7-aab9-866ca084bd59\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" Sep 11 16:43:40.027005 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026931 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdgqk\" (UniqueName: \"kubernetes.io/projected/a575fca1-0433-4012-b079-5baa6ca5a060-kube-api-access-mdgqk\") pod \"volume-data-source-validator-86695b7cc7-zk4rf\" (UID: \"a575fca1-0433-4012-b079-5baa6ca5a060\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zk4rf" Sep 11 16:43:40.027005 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026957 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f76030c8-35a4-4b21-8a5d-ac0ffa450ce6-service-ca-bundle\") pod \"insights-operator-5844b856db-d9hsz\" (UID: \"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6\") " pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:40.027005 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.026982 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-rsrzh\" (UID: \"fde3ddc0-fc6a-42c7-aab9-866ca084bd59\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" Sep 11 16:43:40.027140 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.027006 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13cca68e-a749-4f4e-bbdf-21de50e37c6c-trusted-ca\") pod \"console-operator-696884fb9c-tlxcd\" (UID: \"13cca68e-a749-4f4e-bbdf-21de50e37c6c\") " pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" Sep 11 16:43:40.027140 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.027040 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13cca68e-a749-4f4e-bbdf-21de50e37c6c-serving-cert\") pod \"console-operator-696884fb9c-tlxcd\" (UID: \"13cca68e-a749-4f4e-bbdf-21de50e37c6c\") " pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" Sep 11 16:43:40.027140 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.027082 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/593fb846-0d3d-42dd-8405-6ac2f57bd18f-ca-trust-extracted\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.027140 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.027114 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g49g\" (UniqueName: \"kubernetes.io/projected/f76030c8-35a4-4b21-8a5d-ac0ffa450ce6-kube-api-access-7g49g\") pod \"insights-operator-5844b856db-d9hsz\" (UID: \"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6\") " pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:40.027261 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.027156 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:40.027261 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.027179 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ed61a19b-2fa7-466b-9772-3b7a1f4a5d25-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-fc9sn\" (UID: \"ed61a19b-2fa7-466b-9772-3b7a1f4a5d25\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-fc9sn" Sep 11 16:43:40.027261 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.027201 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f76030c8-35a4-4b21-8a5d-ac0ffa450ce6-tmp\") pod \"insights-operator-5844b856db-d9hsz\" (UID: \"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6\") " pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:40.027261 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.027232 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.088127 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.088095 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:43:40.091705 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.091678 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 16:43:40.091833 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.091716 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dw8q7\"" Sep 11 16:43:40.128134 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128044 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/13cca68e-a749-4f4e-bbdf-21de50e37c6c-config\") pod \"console-operator-696884fb9c-tlxcd\" (UID: \"13cca68e-a749-4f4e-bbdf-21de50e37c6c\") " pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" Sep 11 16:43:40.128134 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128088 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm75p\" (UniqueName: \"kubernetes.io/projected/24b90046-f333-4fa9-8feb-a739dc31f500-kube-api-access-jm75p\") pod \"cluster-proxy-proxy-agent-54cf5ff94b-b5kpv\" (UID: \"24b90046-f333-4fa9-8feb-a739dc31f500\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:40.128355 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128116 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d41dfd70-3a89-4d1d-84de-d0b3964b8538-config\") pod \"service-ca-operator-cf7c776b7-jkhgd\" (UID: \"d41dfd70-3a89-4d1d-84de-d0b3964b8538\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jkhgd" Sep 11 16:43:40.128355 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128187 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjtmj\" (UniqueName: \"kubernetes.io/projected/f66b05a4-ccaf-4c11-b071-8da98ebec68c-kube-api-access-gjtmj\") pod \"network-check-source-6f8b768455-kqg8b\" (UID: \"f66b05a4-ccaf-4c11-b071-8da98ebec68c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-kqg8b" Sep 11 16:43:40.128355 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/593fb846-0d3d-42dd-8405-6ac2f57bd18f-trusted-ca\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.128355 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128258 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rsrzh\" (UID: \"fde3ddc0-fc6a-42c7-aab9-866ca084bd59\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" Sep 11 16:43:40.128355 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128283 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdgqk\" (UniqueName: \"kubernetes.io/projected/a575fca1-0433-4012-b079-5baa6ca5a060-kube-api-access-mdgqk\") pod \"volume-data-source-validator-86695b7cc7-zk4rf\" (UID: \"a575fca1-0433-4012-b079-5baa6ca5a060\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zk4rf" Sep 11 16:43:40.128355 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128310 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f76030c8-35a4-4b21-8a5d-ac0ffa450ce6-service-ca-bundle\") pod \"insights-operator-5844b856db-d9hsz\" (UID: \"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6\") " pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:40.128355 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128334 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-rsrzh\" (UID: \"fde3ddc0-fc6a-42c7-aab9-866ca084bd59\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" Sep 11 16:43:40.128663 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128358 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13cca68e-a749-4f4e-bbdf-21de50e37c6c-trusted-ca\") pod \"console-operator-696884fb9c-tlxcd\" (UID: \"13cca68e-a749-4f4e-bbdf-21de50e37c6c\") " pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" Sep 11 16:43:40.128663 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128391 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13cca68e-a749-4f4e-bbdf-21de50e37c6c-serving-cert\") pod \"console-operator-696884fb9c-tlxcd\" (UID: \"13cca68e-a749-4f4e-bbdf-21de50e37c6c\") " pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" Sep 11 16:43:40.128663 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.128412 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 16:43:40.128663 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128435 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/593fb846-0d3d-42dd-8405-6ac2f57bd18f-ca-trust-extracted\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.128663 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128458 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g49g\" (UniqueName: \"kubernetes.io/projected/f76030c8-35a4-4b21-8a5d-ac0ffa450ce6-kube-api-access-7g49g\") pod \"insights-operator-5844b856db-d9hsz\" (UID: \"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6\") " pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:40.128663 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128486 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6sf4\" (UniqueName: \"kubernetes.io/projected/abde5dc2-8658-41a8-a00e-822e857f9200-kube-api-access-t6sf4\") pod \"managed-serviceaccount-addon-agent-59f5975844-4n7th\" (UID: \"abde5dc2-8658-41a8-a00e-822e857f9200\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f5975844-4n7th" Sep 11 16:43:40.128663 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128532 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:40.128663 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128558 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ed61a19b-2fa7-466b-9772-3b7a1f4a5d25-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-fc9sn\" (UID: \"ed61a19b-2fa7-466b-9772-3b7a1f4a5d25\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-fc9sn" Sep 11 16:43:40.128663 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f76030c8-35a4-4b21-8a5d-ac0ffa450ce6-tmp\") pod \"insights-operator-5844b856db-d9hsz\" (UID: \"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6\") " pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:40.128663 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128607 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls\") pod \"dns-default-c4dn7\" (UID: \"727f81c8-b4b4-4b6e-9570-5739b9698950\") " pod="openshift-dns/dns-default-c4dn7" Sep 11 16:43:40.128663 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.129126 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128676 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/24b90046-f333-4fa9-8feb-a739dc31f500-ca\") pod \"cluster-proxy-proxy-agent-54cf5ff94b-b5kpv\" (UID: \"24b90046-f333-4fa9-8feb-a739dc31f500\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:40.129126 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.128749 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls podName:fde3ddc0-fc6a-42c7-aab9-866ca084bd59 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:40.628729986 +0000 UTC m=+36.036832237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-rsrzh" (UID: "fde3ddc0-fc6a-42c7-aab9-866ca084bd59") : secret "cluster-monitoring-operator-tls" not found Sep 11 16:43:40.129126 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128806 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d41dfd70-3a89-4d1d-84de-d0b3964b8538-config\") pod \"service-ca-operator-cf7c776b7-jkhgd\" (UID: \"d41dfd70-3a89-4d1d-84de-d0b3964b8538\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jkhgd" Sep 11 16:43:40.129126 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.128871 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle podName:4cc211e5-b500-49d5-84be-13c2bfa7f7bc nodeName:}" failed. No retries permitted until 2026-09-11 16:43:40.62885759 +0000 UTC m=+36.036959859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle") pod "router-default-7669dcd8d-whgmr" (UID: "4cc211e5-b500-49d5-84be-13c2bfa7f7bc") : configmap references non-existent config key: service-ca.crt Sep 11 16:43:40.129126 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.128905 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/13cca68e-a749-4f4e-bbdf-21de50e37c6c-config\") pod \"console-operator-696884fb9c-tlxcd\" (UID: \"13cca68e-a749-4f4e-bbdf-21de50e37c6c\") " pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" Sep 11 16:43:40.129126 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129027 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wcxg9\" (UID: \"50af7bf9-0c82-4d40-979b-11350f3018a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9" Sep 11 16:43:40.129126 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129063 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-default-certificate\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:40.129126 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxqjf\" (UniqueName: \"kubernetes.io/projected/42fbe40d-e07f-429e-bfab-01b77d189a2a-kube-api-access-mxqjf\") pod \"ingress-canary-w5wz5\" (UID: \"42fbe40d-e07f-429e-bfab-01b77d189a2a\") " pod="openshift-ingress-canary/ingress-canary-w5wz5" Sep 11 16:43:40.129126 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:40.129578 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129152 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed61a19b-2fa7-466b-9772-3b7a1f4a5d25-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-fc9sn\" (UID: \"ed61a19b-2fa7-466b-9772-3b7a1f4a5d25\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-fc9sn" Sep 11 16:43:40.129578 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129178 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f76030c8-35a4-4b21-8a5d-ac0ffa450ce6-serving-cert\") pod \"insights-operator-5844b856db-d9hsz\" (UID: \"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6\") " pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:40.129578 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129212 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/593fb846-0d3d-42dd-8405-6ac2f57bd18f-image-registry-private-configuration\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.129578 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129217 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/593fb846-0d3d-42dd-8405-6ac2f57bd18f-ca-trust-extracted\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.129578 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129217 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/593fb846-0d3d-42dd-8405-6ac2f57bd18f-trusted-ca\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.129578 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129243 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d41dfd70-3a89-4d1d-84de-d0b3964b8538-serving-cert\") pod \"service-ca-operator-cf7c776b7-jkhgd\" (UID: \"d41dfd70-3a89-4d1d-84de-d0b3964b8538\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jkhgd" Sep 11 16:43:40.129578 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129275 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24l27\" (UniqueName: \"kubernetes.io/projected/13cca68e-a749-4f4e-bbdf-21de50e37c6c-kube-api-access-24l27\") pod \"console-operator-696884fb9c-tlxcd\" (UID: \"13cca68e-a749-4f4e-bbdf-21de50e37c6c\") " pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" Sep 11 16:43:40.129578 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129300 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eeb5215d-ddf9-4bc2-975e-ea70d1fe0687-tmp\") pod \"klusterlet-addon-workmgr-59989b64f6-flmxp\" (UID: \"eeb5215d-ddf9-4bc2-975e-ea70d1fe0687\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59989b64f6-flmxp" Sep 11 16:43:40.129578 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129328 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmmt2\" (UniqueName: \"kubernetes.io/projected/727f81c8-b4b4-4b6e-9570-5739b9698950-kube-api-access-nmmt2\") pod \"dns-default-c4dn7\" (UID: \"727f81c8-b4b4-4b6e-9570-5739b9698950\") " pod="openshift-dns/dns-default-c4dn7" Sep 11 16:43:40.129578 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129354 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/24b90046-f333-4fa9-8feb-a739dc31f500-hub\") pod \"cluster-proxy-proxy-agent-54cf5ff94b-b5kpv\" (UID: \"24b90046-f333-4fa9-8feb-a739dc31f500\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:40.129578 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129382 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/eeb5215d-ddf9-4bc2-975e-ea70d1fe0687-klusterlet-config\") pod \"klusterlet-addon-workmgr-59989b64f6-flmxp\" (UID: \"eeb5215d-ddf9-4bc2-975e-ea70d1fe0687\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59989b64f6-flmxp" Sep 11 16:43:40.129578 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129407 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/abde5dc2-8658-41a8-a00e-822e857f9200-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-59f5975844-4n7th\" (UID: \"abde5dc2-8658-41a8-a00e-822e857f9200\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f5975844-4n7th" Sep 11 16:43:40.129578 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129422 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ed61a19b-2fa7-466b-9772-3b7a1f4a5d25-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-fc9sn\" (UID: \"ed61a19b-2fa7-466b-9772-3b7a1f4a5d25\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-fc9sn" Sep 11 16:43:40.129578 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129441 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrjnq\" (UniqueName: \"kubernetes.io/projected/50af7bf9-0c82-4d40-979b-11350f3018a9-kube-api-access-lrjnq\") pod \"cluster-samples-operator-86499c89c4-wcxg9\" (UID: \"50af7bf9-0c82-4d40-979b-11350f3018a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9" Sep 11 16:43:40.129578 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129469 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96f4w\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-kube-api-access-96f4w\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.130269 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129520 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/593fb846-0d3d-42dd-8405-6ac2f57bd18f-installation-pull-secrets\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.130269 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.129549 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 16:43:40.130269 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129552 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xzrz\" (UniqueName: \"kubernetes.io/projected/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-kube-api-access-7xzrz\") pod \"cluster-monitoring-operator-6cd954b897-rsrzh\" (UID: \"fde3ddc0-fc6a-42c7-aab9-866ca084bd59\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" Sep 11 16:43:40.130269 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.129565 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c679445b8-dxcgg: secret "image-registry-tls" not found Sep 11 16:43:40.130269 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncbzb\" (UniqueName: \"kubernetes.io/projected/ed61a19b-2fa7-466b-9772-3b7a1f4a5d25-kube-api-access-ncbzb\") pod \"kube-storage-version-migrator-operator-6c8f84f897-fc9sn\" (UID: \"ed61a19b-2fa7-466b-9772-3b7a1f4a5d25\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-fc9sn" Sep 11 16:43:40.130269 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.129608 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls podName:593fb846-0d3d-42dd-8405-6ac2f57bd18f nodeName:}" failed. No retries permitted until 2026-09-11 16:43:40.62959202 +0000 UTC m=+36.037694270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls") pod "image-registry-7c679445b8-dxcgg" (UID: "593fb846-0d3d-42dd-8405-6ac2f57bd18f") : secret "image-registry-tls" not found Sep 11 16:43:40.130269 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129616 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13cca68e-a749-4f4e-bbdf-21de50e37c6c-trusted-ca\") pod \"console-operator-696884fb9c-tlxcd\" (UID: \"13cca68e-a749-4f4e-bbdf-21de50e37c6c\") " pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" Sep 11 16:43:40.130269 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129641 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/24b90046-f333-4fa9-8feb-a739dc31f500-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54cf5ff94b-b5kpv\" (UID: \"24b90046-f333-4fa9-8feb-a739dc31f500\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:40.130269 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129671 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vlhl\" (UniqueName: \"kubernetes.io/projected/eeb5215d-ddf9-4bc2-975e-ea70d1fe0687-kube-api-access-6vlhl\") pod \"klusterlet-addon-workmgr-59989b64f6-flmxp\" (UID: \"eeb5215d-ddf9-4bc2-975e-ea70d1fe0687\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59989b64f6-flmxp" Sep 11 16:43:40.130269 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129707 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f76030c8-35a4-4b21-8a5d-ac0ffa450ce6-trusted-ca-bundle\") pod \"insights-operator-5844b856db-d9hsz\" (UID: \"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6\") " pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:40.130269 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129753 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-stats-auth\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:40.130269 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129781 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-certificates\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.130269 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.129805 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 16:43:40.130269 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129818 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42fbe40d-e07f-429e-bfab-01b77d189a2a-cert\") pod \"ingress-canary-w5wz5\" (UID: \"42fbe40d-e07f-429e-bfab-01b77d189a2a\") " pod="openshift-ingress-canary/ingress-canary-w5wz5" Sep 11 16:43:40.130269 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.129864 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls podName:50af7bf9-0c82-4d40-979b-11350f3018a9 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:40.629851526 +0000 UTC m=+36.037953786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-wcxg9" (UID: "50af7bf9-0c82-4d40-979b-11350f3018a9") : secret "samples-operator-tls" not found Sep 11 16:43:40.130269 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.129975 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqp6l\" (UniqueName: \"kubernetes.io/projected/d41dfd70-3a89-4d1d-84de-d0b3964b8538-kube-api-access-cqp6l\") pod \"service-ca-operator-cf7c776b7-jkhgd\" (UID: \"d41dfd70-3a89-4d1d-84de-d0b3964b8538\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jkhgd" Sep 11 16:43:40.131198 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.130012 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b90046-f333-4fa9-8feb-a739dc31f500-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54cf5ff94b-b5kpv\" (UID: \"24b90046-f333-4fa9-8feb-a739dc31f500\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:40.131198 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.130040 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f76030c8-35a4-4b21-8a5d-ac0ffa450ce6-snapshots\") pod \"insights-operator-5844b856db-d9hsz\" (UID: \"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6\") " pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:40.131198 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.130066 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/24b90046-f333-4fa9-8feb-a739dc31f500-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54cf5ff94b-b5kpv\" (UID: \"24b90046-f333-4fa9-8feb-a739dc31f500\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:40.131198 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.130093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/727f81c8-b4b4-4b6e-9570-5739b9698950-config-volume\") pod \"dns-default-c4dn7\" (UID: \"727f81c8-b4b4-4b6e-9570-5739b9698950\") " pod="openshift-dns/dns-default-c4dn7" Sep 11 16:43:40.131198 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.130115 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-rsrzh\" (UID: \"fde3ddc0-fc6a-42c7-aab9-866ca084bd59\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" Sep 11 16:43:40.131198 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.130121 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-bound-sa-token\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.131198 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.130171 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7npvk\" (UniqueName: \"kubernetes.io/projected/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-kube-api-access-7npvk\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:40.131198 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.130198 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727f81c8-b4b4-4b6e-9570-5739b9698950-tmp-dir\") pod \"dns-default-c4dn7\" (UID: \"727f81c8-b4b4-4b6e-9570-5739b9698950\") " pod="openshift-dns/dns-default-c4dn7" Sep 11 16:43:40.131614 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.131284 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 16:43:40.131614 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.131329 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs podName:4cc211e5-b500-49d5-84be-13c2bfa7f7bc nodeName:}" failed. No retries permitted until 2026-09-11 16:43:40.631316039 +0000 UTC m=+36.039418308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs") pod "router-default-7669dcd8d-whgmr" (UID: "4cc211e5-b500-49d5-84be-13c2bfa7f7bc") : secret "router-metrics-certs-default" not found Sep 11 16:43:40.131765 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.131738 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f76030c8-35a4-4b21-8a5d-ac0ffa450ce6-service-ca-bundle\") pod \"insights-operator-5844b856db-d9hsz\" (UID: \"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6\") " pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:40.131899 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.131794 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f76030c8-35a4-4b21-8a5d-ac0ffa450ce6-tmp\") pod \"insights-operator-5844b856db-d9hsz\" (UID: \"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6\") " pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:40.131899 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.131847 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f76030c8-35a4-4b21-8a5d-ac0ffa450ce6-snapshots\") pod \"insights-operator-5844b856db-d9hsz\" (UID: \"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6\") " pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:40.132031 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.132008 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f76030c8-35a4-4b21-8a5d-ac0ffa450ce6-trusted-ca-bundle\") pod \"insights-operator-5844b856db-d9hsz\" (UID: \"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6\") " pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:40.132717 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.132610 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-certificates\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.134767 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.134741 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ed61a19b-2fa7-466b-9772-3b7a1f4a5d25-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-fc9sn\" (UID: \"ed61a19b-2fa7-466b-9772-3b7a1f4a5d25\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-fc9sn" Sep 11 16:43:40.135049 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.135026 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13cca68e-a749-4f4e-bbdf-21de50e37c6c-serving-cert\") pod \"console-operator-696884fb9c-tlxcd\" (UID: \"13cca68e-a749-4f4e-bbdf-21de50e37c6c\") " pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" Sep 11 16:43:40.135576 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.135517 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/593fb846-0d3d-42dd-8405-6ac2f57bd18f-installation-pull-secrets\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.135681 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.135576 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-default-certificate\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:40.136139 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.136111 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f76030c8-35a4-4b21-8a5d-ac0ffa450ce6-serving-cert\") pod \"insights-operator-5844b856db-d9hsz\" (UID: \"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6\") " pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:40.136872 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.136820 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d41dfd70-3a89-4d1d-84de-d0b3964b8538-serving-cert\") pod \"service-ca-operator-cf7c776b7-jkhgd\" (UID: \"d41dfd70-3a89-4d1d-84de-d0b3964b8538\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jkhgd" Sep 11 16:43:40.136988 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.136851 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-stats-auth\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:40.137326 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.137306 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/593fb846-0d3d-42dd-8405-6ac2f57bd18f-image-registry-private-configuration\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.146369 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.146197 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-bound-sa-token\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.150233 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.150186 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96f4w\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-kube-api-access-96f4w\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.150864 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.150808 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrjnq\" (UniqueName: \"kubernetes.io/projected/50af7bf9-0c82-4d40-979b-11350f3018a9-kube-api-access-lrjnq\") pod \"cluster-samples-operator-86499c89c4-wcxg9\" (UID: \"50af7bf9-0c82-4d40-979b-11350f3018a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9" Sep 11 16:43:40.150864 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.150821 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqp6l\" (UniqueName: \"kubernetes.io/projected/d41dfd70-3a89-4d1d-84de-d0b3964b8538-kube-api-access-cqp6l\") pod \"service-ca-operator-cf7c776b7-jkhgd\" (UID: \"d41dfd70-3a89-4d1d-84de-d0b3964b8538\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jkhgd" Sep 11 16:43:40.151252 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.151207 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncbzb\" (UniqueName: \"kubernetes.io/projected/ed61a19b-2fa7-466b-9772-3b7a1f4a5d25-kube-api-access-ncbzb\") pod \"kube-storage-version-migrator-operator-6c8f84f897-fc9sn\" (UID: \"ed61a19b-2fa7-466b-9772-3b7a1f4a5d25\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-fc9sn" Sep 11 16:43:40.151333 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.151289 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g49g\" (UniqueName: \"kubernetes.io/projected/f76030c8-35a4-4b21-8a5d-ac0ffa450ce6-kube-api-access-7g49g\") pod \"insights-operator-5844b856db-d9hsz\" (UID: \"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6\") " pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:40.151393 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.151338 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24l27\" (UniqueName: \"kubernetes.io/projected/13cca68e-a749-4f4e-bbdf-21de50e37c6c-kube-api-access-24l27\") pod \"console-operator-696884fb9c-tlxcd\" (UID: \"13cca68e-a749-4f4e-bbdf-21de50e37c6c\") " pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" Sep 11 16:43:40.151451 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.151402 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xzrz\" (UniqueName: \"kubernetes.io/projected/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-kube-api-access-7xzrz\") pod \"cluster-monitoring-operator-6cd954b897-rsrzh\" (UID: \"fde3ddc0-fc6a-42c7-aab9-866ca084bd59\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" Sep 11 16:43:40.151451 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.151421 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7npvk\" (UniqueName: \"kubernetes.io/projected/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-kube-api-access-7npvk\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:40.152767 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.152720 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdgqk\" (UniqueName: \"kubernetes.io/projected/a575fca1-0433-4012-b079-5baa6ca5a060-kube-api-access-mdgqk\") pod \"volume-data-source-validator-86695b7cc7-zk4rf\" (UID: \"a575fca1-0433-4012-b079-5baa6ca5a060\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zk4rf" Sep 11 16:43:40.230770 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.230735 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6sf4\" (UniqueName: \"kubernetes.io/projected/abde5dc2-8658-41a8-a00e-822e857f9200-kube-api-access-t6sf4\") pod \"managed-serviceaccount-addon-agent-59f5975844-4n7th\" (UID: \"abde5dc2-8658-41a8-a00e-822e857f9200\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f5975844-4n7th" Sep 11 16:43:40.230974 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.230792 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls\") pod \"dns-default-c4dn7\" (UID: \"727f81c8-b4b4-4b6e-9570-5739b9698950\") " pod="openshift-dns/dns-default-c4dn7" Sep 11 16:43:40.230974 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.230897 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 16:43:40.231091 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.230981 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls podName:727f81c8-b4b4-4b6e-9570-5739b9698950 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:40.730963099 +0000 UTC m=+36.139065347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls") pod "dns-default-c4dn7" (UID: "727f81c8-b4b4-4b6e-9570-5739b9698950") : secret "dns-default-metrics-tls" not found Sep 11 16:43:40.231091 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.231001 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/24b90046-f333-4fa9-8feb-a739dc31f500-ca\") pod \"cluster-proxy-proxy-agent-54cf5ff94b-b5kpv\" (UID: \"24b90046-f333-4fa9-8feb-a739dc31f500\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:40.231091 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.231041 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxqjf\" (UniqueName: \"kubernetes.io/projected/42fbe40d-e07f-429e-bfab-01b77d189a2a-kube-api-access-mxqjf\") pod \"ingress-canary-w5wz5\" (UID: \"42fbe40d-e07f-429e-bfab-01b77d189a2a\") " pod="openshift-ingress-canary/ingress-canary-w5wz5" Sep 11 16:43:40.231253 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.231091 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eeb5215d-ddf9-4bc2-975e-ea70d1fe0687-tmp\") pod \"klusterlet-addon-workmgr-59989b64f6-flmxp\" (UID: \"eeb5215d-ddf9-4bc2-975e-ea70d1fe0687\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59989b64f6-flmxp" Sep 11 16:43:40.231253 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.231117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmmt2\" (UniqueName: \"kubernetes.io/projected/727f81c8-b4b4-4b6e-9570-5739b9698950-kube-api-access-nmmt2\") pod \"dns-default-c4dn7\" (UID: \"727f81c8-b4b4-4b6e-9570-5739b9698950\") " pod="openshift-dns/dns-default-c4dn7" Sep 11 16:43:40.231253 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.231145 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/24b90046-f333-4fa9-8feb-a739dc31f500-hub\") pod \"cluster-proxy-proxy-agent-54cf5ff94b-b5kpv\" (UID: \"24b90046-f333-4fa9-8feb-a739dc31f500\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:40.231253 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.231171 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/eeb5215d-ddf9-4bc2-975e-ea70d1fe0687-klusterlet-config\") pod \"klusterlet-addon-workmgr-59989b64f6-flmxp\" (UID: \"eeb5215d-ddf9-4bc2-975e-ea70d1fe0687\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59989b64f6-flmxp" Sep 11 16:43:40.231253 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.231195 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/abde5dc2-8658-41a8-a00e-822e857f9200-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-59f5975844-4n7th\" (UID: \"abde5dc2-8658-41a8-a00e-822e857f9200\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f5975844-4n7th" Sep 11 16:43:40.231253 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.231230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/24b90046-f333-4fa9-8feb-a739dc31f500-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54cf5ff94b-b5kpv\" (UID: \"24b90046-f333-4fa9-8feb-a739dc31f500\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:40.231573 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.231254 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vlhl\" (UniqueName: \"kubernetes.io/projected/eeb5215d-ddf9-4bc2-975e-ea70d1fe0687-kube-api-access-6vlhl\") pod \"klusterlet-addon-workmgr-59989b64f6-flmxp\" (UID: \"eeb5215d-ddf9-4bc2-975e-ea70d1fe0687\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59989b64f6-flmxp" Sep 11 16:43:40.231573 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.231490 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42fbe40d-e07f-429e-bfab-01b77d189a2a-cert\") pod \"ingress-canary-w5wz5\" (UID: \"42fbe40d-e07f-429e-bfab-01b77d189a2a\") " pod="openshift-ingress-canary/ingress-canary-w5wz5" Sep 11 16:43:40.231573 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.231567 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b90046-f333-4fa9-8feb-a739dc31f500-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54cf5ff94b-b5kpv\" (UID: \"24b90046-f333-4fa9-8feb-a739dc31f500\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:40.231719 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.231594 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/24b90046-f333-4fa9-8feb-a739dc31f500-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54cf5ff94b-b5kpv\" (UID: \"24b90046-f333-4fa9-8feb-a739dc31f500\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:40.231719 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.231619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/727f81c8-b4b4-4b6e-9570-5739b9698950-config-volume\") pod \"dns-default-c4dn7\" (UID: \"727f81c8-b4b4-4b6e-9570-5739b9698950\") " pod="openshift-dns/dns-default-c4dn7" Sep 11 16:43:40.231719 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.231644 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727f81c8-b4b4-4b6e-9570-5739b9698950-tmp-dir\") pod \"dns-default-c4dn7\" (UID: \"727f81c8-b4b4-4b6e-9570-5739b9698950\") " pod="openshift-dns/dns-default-c4dn7" Sep 11 16:43:40.231719 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.231673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm75p\" (UniqueName: \"kubernetes.io/projected/24b90046-f333-4fa9-8feb-a739dc31f500-kube-api-access-jm75p\") pod \"cluster-proxy-proxy-agent-54cf5ff94b-b5kpv\" (UID: \"24b90046-f333-4fa9-8feb-a739dc31f500\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:40.231719 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.231703 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjtmj\" (UniqueName: \"kubernetes.io/projected/f66b05a4-ccaf-4c11-b071-8da98ebec68c-kube-api-access-gjtmj\") pod \"network-check-source-6f8b768455-kqg8b\" (UID: \"f66b05a4-ccaf-4c11-b071-8da98ebec68c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-kqg8b" Sep 11 16:43:40.231975 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.231785 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eeb5215d-ddf9-4bc2-975e-ea70d1fe0687-tmp\") pod \"klusterlet-addon-workmgr-59989b64f6-flmxp\" (UID: \"eeb5215d-ddf9-4bc2-975e-ea70d1fe0687\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59989b64f6-flmxp" Sep 11 16:43:40.232757 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.232345 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/727f81c8-b4b4-4b6e-9570-5739b9698950-config-volume\") pod \"dns-default-c4dn7\" (UID: \"727f81c8-b4b4-4b6e-9570-5739b9698950\") " pod="openshift-dns/dns-default-c4dn7" Sep 11 16:43:40.232757 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.232545 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727f81c8-b4b4-4b6e-9570-5739b9698950-tmp-dir\") pod \"dns-default-c4dn7\" (UID: \"727f81c8-b4b4-4b6e-9570-5739b9698950\") " pod="openshift-dns/dns-default-c4dn7" Sep 11 16:43:40.232757 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.232706 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b90046-f333-4fa9-8feb-a739dc31f500-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54cf5ff94b-b5kpv\" (UID: \"24b90046-f333-4fa9-8feb-a739dc31f500\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:40.234813 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.234788 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/24b90046-f333-4fa9-8feb-a739dc31f500-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54cf5ff94b-b5kpv\" (UID: \"24b90046-f333-4fa9-8feb-a739dc31f500\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:40.234930 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.234815 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/24b90046-f333-4fa9-8feb-a739dc31f500-hub\") pod \"cluster-proxy-proxy-agent-54cf5ff94b-b5kpv\" (UID: \"24b90046-f333-4fa9-8feb-a739dc31f500\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:40.234930 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.234793 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/abde5dc2-8658-41a8-a00e-822e857f9200-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-59f5975844-4n7th\" (UID: \"abde5dc2-8658-41a8-a00e-822e857f9200\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f5975844-4n7th" Sep 11 16:43:40.235197 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.235176 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/24b90046-f333-4fa9-8feb-a739dc31f500-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54cf5ff94b-b5kpv\" (UID: \"24b90046-f333-4fa9-8feb-a739dc31f500\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:40.235448 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.235422 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/eeb5215d-ddf9-4bc2-975e-ea70d1fe0687-klusterlet-config\") pod \"klusterlet-addon-workmgr-59989b64f6-flmxp\" (UID: \"eeb5215d-ddf9-4bc2-975e-ea70d1fe0687\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59989b64f6-flmxp" Sep 11 16:43:40.235964 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.235941 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42fbe40d-e07f-429e-bfab-01b77d189a2a-cert\") pod \"ingress-canary-w5wz5\" (UID: \"42fbe40d-e07f-429e-bfab-01b77d189a2a\") " pod="openshift-ingress-canary/ingress-canary-w5wz5" Sep 11 16:43:40.236395 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.236374 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/24b90046-f333-4fa9-8feb-a739dc31f500-ca\") pod \"cluster-proxy-proxy-agent-54cf5ff94b-b5kpv\" (UID: \"24b90046-f333-4fa9-8feb-a739dc31f500\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:40.242327 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.242300 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6sf4\" (UniqueName: \"kubernetes.io/projected/abde5dc2-8658-41a8-a00e-822e857f9200-kube-api-access-t6sf4\") pod \"managed-serviceaccount-addon-agent-59f5975844-4n7th\" (UID: \"abde5dc2-8658-41a8-a00e-822e857f9200\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f5975844-4n7th" Sep 11 16:43:40.249098 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.249078 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxqjf\" (UniqueName: \"kubernetes.io/projected/42fbe40d-e07f-429e-bfab-01b77d189a2a-kube-api-access-mxqjf\") pod \"ingress-canary-w5wz5\" (UID: \"42fbe40d-e07f-429e-bfab-01b77d189a2a\") " pod="openshift-ingress-canary/ingress-canary-w5wz5" Sep 11 16:43:40.249188 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.249081 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjtmj\" (UniqueName: \"kubernetes.io/projected/f66b05a4-ccaf-4c11-b071-8da98ebec68c-kube-api-access-gjtmj\") pod \"network-check-source-6f8b768455-kqg8b\" (UID: \"f66b05a4-ccaf-4c11-b071-8da98ebec68c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-kqg8b" Sep 11 16:43:40.250446 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.250415 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-fc9sn" Sep 11 16:43:40.250905 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.250886 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmmt2\" (UniqueName: \"kubernetes.io/projected/727f81c8-b4b4-4b6e-9570-5739b9698950-kube-api-access-nmmt2\") pod \"dns-default-c4dn7\" (UID: \"727f81c8-b4b4-4b6e-9570-5739b9698950\") " pod="openshift-dns/dns-default-c4dn7" Sep 11 16:43:40.251313 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.251295 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm75p\" (UniqueName: \"kubernetes.io/projected/24b90046-f333-4fa9-8feb-a739dc31f500-kube-api-access-jm75p\") pod \"cluster-proxy-proxy-agent-54cf5ff94b-b5kpv\" (UID: \"24b90046-f333-4fa9-8feb-a739dc31f500\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:40.258853 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.258833 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vlhl\" (UniqueName: \"kubernetes.io/projected/eeb5215d-ddf9-4bc2-975e-ea70d1fe0687-kube-api-access-6vlhl\") pod \"klusterlet-addon-workmgr-59989b64f6-flmxp\" (UID: \"eeb5215d-ddf9-4bc2-975e-ea70d1fe0687\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59989b64f6-flmxp" Sep 11 16:43:40.262509 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.262483 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zk4rf" Sep 11 16:43:40.272818 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.272791 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jkhgd" Sep 11 16:43:40.282584 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.282565 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-d9hsz" Sep 11 16:43:40.290795 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.290776 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" Sep 11 16:43:40.326604 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.326570 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:43:40.334628 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.334607 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59989b64f6-flmxp" Sep 11 16:43:40.341287 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.341269 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w5wz5" Sep 11 16:43:40.346902 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.346881 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f5975844-4n7th" Sep 11 16:43:40.368453 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.368425 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-kqg8b" Sep 11 16:43:40.636363 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.636320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rsrzh\" (UID: \"fde3ddc0-fc6a-42c7-aab9-866ca084bd59\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" Sep 11 16:43:40.636585 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.636385 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:40.636585 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.636431 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:40.636585 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.636449 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wcxg9\" (UID: \"50af7bf9-0c82-4d40-979b-11350f3018a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9" Sep 11 16:43:40.636585 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.636470 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:40.636585 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.636511 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 16:43:40.636585 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.636574 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 16:43:40.636901 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.636578 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 16:43:40.636901 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.636657 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c679445b8-dxcgg: secret "image-registry-tls" not found Sep 11 16:43:40.636901 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.636580 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle podName:4cc211e5-b500-49d5-84be-13c2bfa7f7bc nodeName:}" failed. No retries permitted until 2026-09-11 16:43:41.636559479 +0000 UTC m=+37.044661748 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle") pod "router-default-7669dcd8d-whgmr" (UID: "4cc211e5-b500-49d5-84be-13c2bfa7f7bc") : configmap references non-existent config key: service-ca.crt Sep 11 16:43:40.636901 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.636626 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 16:43:40.636901 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.636741 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls podName:fde3ddc0-fc6a-42c7-aab9-866ca084bd59 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:41.636721835 +0000 UTC m=+37.044824087 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-rsrzh" (UID: "fde3ddc0-fc6a-42c7-aab9-866ca084bd59") : secret "cluster-monitoring-operator-tls" not found Sep 11 16:43:40.636901 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.636770 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls podName:50af7bf9-0c82-4d40-979b-11350f3018a9 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:41.636760719 +0000 UTC m=+37.044862967 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-wcxg9" (UID: "50af7bf9-0c82-4d40-979b-11350f3018a9") : secret "samples-operator-tls" not found Sep 11 16:43:40.636901 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.636793 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls podName:593fb846-0d3d-42dd-8405-6ac2f57bd18f nodeName:}" failed. No retries permitted until 2026-09-11 16:43:41.636785758 +0000 UTC m=+37.044888006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls") pod "image-registry-7c679445b8-dxcgg" (UID: "593fb846-0d3d-42dd-8405-6ac2f57bd18f") : secret "image-registry-tls" not found Sep 11 16:43:40.636901 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.636811 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs podName:4cc211e5-b500-49d5-84be-13c2bfa7f7bc nodeName:}" failed. No retries permitted until 2026-09-11 16:43:41.636806037 +0000 UTC m=+37.044908284 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs") pod "router-default-7669dcd8d-whgmr" (UID: "4cc211e5-b500-49d5-84be-13c2bfa7f7bc") : secret "router-metrics-certs-default" not found Sep 11 16:43:40.738231 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:40.738200 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls\") pod \"dns-default-c4dn7\" (UID: \"727f81c8-b4b4-4b6e-9570-5739b9698950\") " pod="openshift-dns/dns-default-c4dn7" Sep 11 16:43:40.738367 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.738339 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 16:43:40.738427 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:40.738395 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls podName:727f81c8-b4b4-4b6e-9570-5739b9698950 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:41.738381633 +0000 UTC m=+37.146483880 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls") pod "dns-default-c4dn7" (UID: "727f81c8-b4b4-4b6e-9570-5739b9698950") : secret "dns-default-metrics-tls" not found Sep 11 16:43:41.088323 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.088286 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:43:41.088794 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.088286 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:43:41.092783 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.092577 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 16:43:41.092783 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.092638 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6xdv2\"" Sep 11 16:43:41.646416 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.646070 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rsrzh\" (UID: \"fde3ddc0-fc6a-42c7-aab9-866ca084bd59\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" Sep 11 16:43:41.646416 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.646134 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:41.646416 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.646177 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:41.646416 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.646202 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wcxg9\" (UID: \"50af7bf9-0c82-4d40-979b-11350f3018a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9" Sep 11 16:43:41.646416 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.646232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:41.646416 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:41.646371 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 16:43:41.647053 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:41.646440 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs podName:4cc211e5-b500-49d5-84be-13c2bfa7f7bc nodeName:}" failed. No retries permitted until 2026-09-11 16:43:43.646420074 +0000 UTC m=+39.054522324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs") pod "router-default-7669dcd8d-whgmr" (UID: "4cc211e5-b500-49d5-84be-13c2bfa7f7bc") : secret "router-metrics-certs-default" not found Sep 11 16:43:41.647053 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:41.646535 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 16:43:41.647053 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:41.646550 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c679445b8-dxcgg: secret "image-registry-tls" not found Sep 11 16:43:41.647053 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:41.646615 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 16:43:41.647053 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:41.646632 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 16:43:41.647053 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:41.646535 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle podName:4cc211e5-b500-49d5-84be-13c2bfa7f7bc nodeName:}" failed. No retries permitted until 2026-09-11 16:43:43.646510651 +0000 UTC m=+39.054612912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle") pod "router-default-7669dcd8d-whgmr" (UID: "4cc211e5-b500-49d5-84be-13c2bfa7f7bc") : configmap references non-existent config key: service-ca.crt Sep 11 16:43:41.647053 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:41.646680 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls podName:593fb846-0d3d-42dd-8405-6ac2f57bd18f nodeName:}" failed. No retries permitted until 2026-09-11 16:43:43.646666381 +0000 UTC m=+39.054768633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls") pod "image-registry-7c679445b8-dxcgg" (UID: "593fb846-0d3d-42dd-8405-6ac2f57bd18f") : secret "image-registry-tls" not found Sep 11 16:43:41.647053 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:41.646696 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls podName:fde3ddc0-fc6a-42c7-aab9-866ca084bd59 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:43.646687096 +0000 UTC m=+39.054789343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-rsrzh" (UID: "fde3ddc0-fc6a-42c7-aab9-866ca084bd59") : secret "cluster-monitoring-operator-tls" not found Sep 11 16:43:41.647053 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:41.646710 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls podName:50af7bf9-0c82-4d40-979b-11350f3018a9 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:43.646702589 +0000 UTC m=+39.054804837 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-wcxg9" (UID: "50af7bf9-0c82-4d40-979b-11350f3018a9") : secret "samples-operator-tls" not found Sep 11 16:43:41.748770 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.747901 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls\") pod \"dns-default-c4dn7\" (UID: \"727f81c8-b4b4-4b6e-9570-5739b9698950\") " pod="openshift-dns/dns-default-c4dn7" Sep 11 16:43:41.749400 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:41.748934 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 16:43:41.749400 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:41.749014 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls podName:727f81c8-b4b4-4b6e-9570-5739b9698950 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:43.748994203 +0000 UTC m=+39.157096469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls") pod "dns-default-c4dn7" (UID: "727f81c8-b4b4-4b6e-9570-5739b9698950") : secret "dns-default-metrics-tls" not found Sep 11 16:43:41.869816 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.869627 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-jkhgd"] Sep 11 16:43:41.874163 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.874131 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f5975844-4n7th"] Sep 11 16:43:41.884219 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.884191 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w5wz5"] Sep 11 16:43:41.888770 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.888745 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-kqg8b"] Sep 11 16:43:41.891599 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.891565 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-d9hsz"] Sep 11 16:43:41.892555 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.892536 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-tlxcd"] Sep 11 16:43:41.916185 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.916159 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zk4rf"] Sep 11 16:43:41.924926 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.924896 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59989b64f6-flmxp"] Sep 11 16:43:41.945087 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.945059 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-fc9sn"] Sep 11 16:43:41.977342 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:41.977290 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv"] Sep 11 16:43:41.980025 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:41.979999 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66b05a4_ccaf_4c11_b071_8da98ebec68c.slice/crio-84d80e1fd942ba3ce3dd1649fe8db2aa0e012255cce9a3c2c323f7ae47fa1b57 WatchSource:0}: Error finding container 84d80e1fd942ba3ce3dd1649fe8db2aa0e012255cce9a3c2c323f7ae47fa1b57: Status 404 returned error can't find the container with id 84d80e1fd942ba3ce3dd1649fe8db2aa0e012255cce9a3c2c323f7ae47fa1b57 Sep 11 16:43:41.980797 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:41.980762 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda575fca1_0433_4012_b079_5baa6ca5a060.slice/crio-1fd58ebb437479179ab2ad2020e413516d3ca70d637f8cc6973efac1a7196c9b WatchSource:0}: Error finding container 1fd58ebb437479179ab2ad2020e413516d3ca70d637f8cc6973efac1a7196c9b: Status 404 returned error can't find the container with id 1fd58ebb437479179ab2ad2020e413516d3ca70d637f8cc6973efac1a7196c9b Sep 11 16:43:41.981897 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:41.981851 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded61a19b_2fa7_466b_9772_3b7a1f4a5d25.slice/crio-3e1f78e0fb006aa248d09ad811da3c0dcb2b3bfd8fc581d1c5149f30289fdfa2 WatchSource:0}: Error finding container 3e1f78e0fb006aa248d09ad811da3c0dcb2b3bfd8fc581d1c5149f30289fdfa2: Status 404 returned error can't find the container with id 3e1f78e0fb006aa248d09ad811da3c0dcb2b3bfd8fc581d1c5149f30289fdfa2 Sep 11 16:43:41.983811 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:41.983111 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42fbe40d_e07f_429e_bfab_01b77d189a2a.slice/crio-e68b8481a45e8d6e44bda9617e9f208d7747795bee92c26ff7167579ea50d712 WatchSource:0}: Error finding container e68b8481a45e8d6e44bda9617e9f208d7747795bee92c26ff7167579ea50d712: Status 404 returned error can't find the container with id e68b8481a45e8d6e44bda9617e9f208d7747795bee92c26ff7167579ea50d712 Sep 11 16:43:41.984648 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:41.984251 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabde5dc2_8658_41a8_a00e_822e857f9200.slice/crio-f25c9ef5a994d0e972eac82c5427224e23c980965aa2106a2448d12bdcf6d30e WatchSource:0}: Error finding container f25c9ef5a994d0e972eac82c5427224e23c980965aa2106a2448d12bdcf6d30e: Status 404 returned error can't find the container with id f25c9ef5a994d0e972eac82c5427224e23c980965aa2106a2448d12bdcf6d30e Sep 11 16:43:41.985536 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:41.985233 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb5215d_ddf9_4bc2_975e_ea70d1fe0687.slice/crio-fcbe0db65fc6aaf1df99669be4bc1a26b8625ee4127766851e7d0930efe87003 WatchSource:0}: Error finding container fcbe0db65fc6aaf1df99669be4bc1a26b8625ee4127766851e7d0930efe87003: Status 404 returned error can't find the container with id fcbe0db65fc6aaf1df99669be4bc1a26b8625ee4127766851e7d0930efe87003 Sep 11 16:43:41.985778 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:41.985754 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41dfd70_3a89_4d1d_84de_d0b3964b8538.slice/crio-b1d0b0dfb83b0b1ad0c58d043f2860cfb95d80893b795632ba7785035b74a807 WatchSource:0}: Error finding container b1d0b0dfb83b0b1ad0c58d043f2860cfb95d80893b795632ba7785035b74a807: Status 404 returned error can't find the container with id b1d0b0dfb83b0b1ad0c58d043f2860cfb95d80893b795632ba7785035b74a807 Sep 11 16:43:41.986667 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:41.986637 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13cca68e_a749_4f4e_bbdf_21de50e37c6c.slice/crio-2a9727bbaca6df6777933c150ed0c6d9cb7503efb829ed5e1c1154f10de72cb5 WatchSource:0}: Error finding container 2a9727bbaca6df6777933c150ed0c6d9cb7503efb829ed5e1c1154f10de72cb5: Status 404 returned error can't find the container with id 2a9727bbaca6df6777933c150ed0c6d9cb7503efb829ed5e1c1154f10de72cb5 Sep 11 16:43:41.987442 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:41.987406 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76030c8_35a4_4b21_8a5d_ac0ffa450ce6.slice/crio-4a06a8154c80ff7b193bf6808681c1a76535fff1d32d2b06fede5a31d5750458 WatchSource:0}: Error finding container 4a06a8154c80ff7b193bf6808681c1a76535fff1d32d2b06fede5a31d5750458: Status 404 returned error can't find the container with id 4a06a8154c80ff7b193bf6808681c1a76535fff1d32d2b06fede5a31d5750458 Sep 11 16:43:41.988192 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:43:41.988170 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24b90046_f333_4fa9_8feb_a739dc31f500.slice/crio-9621203cad80279031605a0d773a24dd7ea29da9e9d2bc26ad0595d4e2b4cc31 WatchSource:0}: Error finding container 9621203cad80279031605a0d773a24dd7ea29da9e9d2bc26ad0595d4e2b4cc31: Status 404 returned error can't find the container with id 9621203cad80279031605a0d773a24dd7ea29da9e9d2bc26ad0595d4e2b4cc31 Sep 11 16:43:42.330639 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:42.330599 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59989b64f6-flmxp" event={"ID":"eeb5215d-ddf9-4bc2-975e-ea70d1fe0687","Type":"ContainerStarted","Data":"fcbe0db65fc6aaf1df99669be4bc1a26b8625ee4127766851e7d0930efe87003"} Sep 11 16:43:42.332165 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:42.332134 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f5975844-4n7th" event={"ID":"abde5dc2-8658-41a8-a00e-822e857f9200","Type":"ContainerStarted","Data":"f25c9ef5a994d0e972eac82c5427224e23c980965aa2106a2448d12bdcf6d30e"} Sep 11 16:43:42.333669 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:42.333646 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w5wz5" event={"ID":"42fbe40d-e07f-429e-bfab-01b77d189a2a","Type":"ContainerStarted","Data":"e68b8481a45e8d6e44bda9617e9f208d7747795bee92c26ff7167579ea50d712"} Sep 11 16:43:42.334856 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:42.334834 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-kqg8b" event={"ID":"f66b05a4-ccaf-4c11-b071-8da98ebec68c","Type":"ContainerStarted","Data":"84d80e1fd942ba3ce3dd1649fe8db2aa0e012255cce9a3c2c323f7ae47fa1b57"} Sep 11 16:43:42.335936 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:42.335916 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-d9hsz" event={"ID":"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6","Type":"ContainerStarted","Data":"4a06a8154c80ff7b193bf6808681c1a76535fff1d32d2b06fede5a31d5750458"} Sep 11 16:43:42.337518 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:42.337481 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" event={"ID":"24b90046-f333-4fa9-8feb-a739dc31f500","Type":"ContainerStarted","Data":"9621203cad80279031605a0d773a24dd7ea29da9e9d2bc26ad0595d4e2b4cc31"} Sep 11 16:43:42.338710 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:42.338690 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zk4rf" event={"ID":"a575fca1-0433-4012-b079-5baa6ca5a060","Type":"ContainerStarted","Data":"1fd58ebb437479179ab2ad2020e413516d3ca70d637f8cc6973efac1a7196c9b"} Sep 11 16:43:42.339785 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:42.339761 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jkhgd" event={"ID":"d41dfd70-3a89-4d1d-84de-d0b3964b8538","Type":"ContainerStarted","Data":"b1d0b0dfb83b0b1ad0c58d043f2860cfb95d80893b795632ba7785035b74a807"} Sep 11 16:43:42.342301 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:42.342280 2577 generic.go:356] "Generic (PLEG): container finished" podID="9052d32a-f4da-4ace-9da5-6a63d6e7e0c2" containerID="61b985362e187f158de6997029284c63c92307cb1feb84ac0983d6195a15946c" exitCode=0 Sep 11 16:43:42.342374 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:42.342360 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7dfv" event={"ID":"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2","Type":"ContainerDied","Data":"61b985362e187f158de6997029284c63c92307cb1feb84ac0983d6195a15946c"} Sep 11 16:43:42.343484 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:42.343463 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-fc9sn" event={"ID":"ed61a19b-2fa7-466b-9772-3b7a1f4a5d25","Type":"ContainerStarted","Data":"3e1f78e0fb006aa248d09ad811da3c0dcb2b3bfd8fc581d1c5149f30289fdfa2"} Sep 11 16:43:42.344605 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:42.344587 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" event={"ID":"13cca68e-a749-4f4e-bbdf-21de50e37c6c","Type":"ContainerStarted","Data":"2a9727bbaca6df6777933c150ed0c6d9cb7503efb829ed5e1c1154f10de72cb5"} Sep 11 16:43:43.373268 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:43.373230 2577 generic.go:356] "Generic (PLEG): container finished" podID="9052d32a-f4da-4ace-9da5-6a63d6e7e0c2" containerID="e9e36f94805d6a06b47ae5c4e3f90606eaf680b32baf9c5c43b2da9d8fe551f7" exitCode=0 Sep 11 16:43:43.373935 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:43.373298 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7dfv" event={"ID":"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2","Type":"ContainerDied","Data":"e9e36f94805d6a06b47ae5c4e3f90606eaf680b32baf9c5c43b2da9d8fe551f7"} Sep 11 16:43:43.668215 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:43.668180 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rsrzh\" (UID: \"fde3ddc0-fc6a-42c7-aab9-866ca084bd59\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" Sep 11 16:43:43.668343 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:43.668264 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:43.668343 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:43.668311 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:43.668474 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:43.668345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wcxg9\" (UID: \"50af7bf9-0c82-4d40-979b-11350f3018a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9" Sep 11 16:43:43.668474 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:43.668374 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:43.668629 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:43.668554 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 16:43:43.668629 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:43.668610 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs podName:4cc211e5-b500-49d5-84be-13c2bfa7f7bc nodeName:}" failed. No retries permitted until 2026-09-11 16:43:47.66859223 +0000 UTC m=+43.076694485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs") pod "router-default-7669dcd8d-whgmr" (UID: "4cc211e5-b500-49d5-84be-13c2bfa7f7bc") : secret "router-metrics-certs-default" not found Sep 11 16:43:43.669098 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:43.668991 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 16:43:43.669098 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:43.669047 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls podName:fde3ddc0-fc6a-42c7-aab9-866ca084bd59 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:47.669031278 +0000 UTC m=+43.077133530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-rsrzh" (UID: "fde3ddc0-fc6a-42c7-aab9-866ca084bd59") : secret "cluster-monitoring-operator-tls" not found Sep 11 16:43:43.669238 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:43.669116 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle podName:4cc211e5-b500-49d5-84be-13c2bfa7f7bc nodeName:}" failed. No retries permitted until 2026-09-11 16:43:47.669104027 +0000 UTC m=+43.077206287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle") pod "router-default-7669dcd8d-whgmr" (UID: "4cc211e5-b500-49d5-84be-13c2bfa7f7bc") : configmap references non-existent config key: service-ca.crt Sep 11 16:43:43.669238 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:43.669173 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 16:43:43.669238 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:43.669184 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c679445b8-dxcgg: secret "image-registry-tls" not found Sep 11 16:43:43.669238 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:43.669214 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls podName:593fb846-0d3d-42dd-8405-6ac2f57bd18f nodeName:}" failed. No retries permitted until 2026-09-11 16:43:47.66920273 +0000 UTC m=+43.077304982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls") pod "image-registry-7c679445b8-dxcgg" (UID: "593fb846-0d3d-42dd-8405-6ac2f57bd18f") : secret "image-registry-tls" not found Sep 11 16:43:43.669510 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:43.669273 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 16:43:43.669510 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:43.669300 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls podName:50af7bf9-0c82-4d40-979b-11350f3018a9 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:47.669291303 +0000 UTC m=+43.077393556 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-wcxg9" (UID: "50af7bf9-0c82-4d40-979b-11350f3018a9") : secret "samples-operator-tls" not found Sep 11 16:43:43.769981 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:43.769883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls\") pod \"dns-default-c4dn7\" (UID: \"727f81c8-b4b4-4b6e-9570-5739b9698950\") " pod="openshift-dns/dns-default-c4dn7" Sep 11 16:43:43.770473 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:43.770270 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 16:43:43.771095 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:43.771025 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls podName:727f81c8-b4b4-4b6e-9570-5739b9698950 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:47.770415517 +0000 UTC m=+43.178517772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls") pod "dns-default-c4dn7" (UID: "727f81c8-b4b4-4b6e-9570-5739b9698950") : secret "dns-default-metrics-tls" not found Sep 11 16:43:44.415783 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:44.415739 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7dfv" event={"ID":"9052d32a-f4da-4ace-9da5-6a63d6e7e0c2","Type":"ContainerStarted","Data":"c81d088b9963f5b3e737251c372815e443ecda658d6454a2a54f50a7957ab543"} Sep 11 16:43:47.713593 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:47.713549 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:47.713593 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:47.713595 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wcxg9\" (UID: \"50af7bf9-0c82-4d40-979b-11350f3018a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9" Sep 11 16:43:47.714109 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:47.713615 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:47.714109 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:47.713743 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 16:43:47.714109 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:47.713748 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 16:43:47.714109 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:47.713772 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c679445b8-dxcgg: secret "image-registry-tls" not found Sep 11 16:43:47.714109 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:47.713773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rsrzh\" (UID: \"fde3ddc0-fc6a-42c7-aab9-866ca084bd59\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" Sep 11 16:43:47.714109 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:47.713809 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs podName:4cc211e5-b500-49d5-84be-13c2bfa7f7bc nodeName:}" failed. No retries permitted until 2026-09-11 16:43:55.713795245 +0000 UTC m=+51.121897509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs") pod "router-default-7669dcd8d-whgmr" (UID: "4cc211e5-b500-49d5-84be-13c2bfa7f7bc") : secret "router-metrics-certs-default" not found Sep 11 16:43:47.714109 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:47.713742 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 16:43:47.714109 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:47.713829 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 16:43:47.714109 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:47.713843 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls podName:593fb846-0d3d-42dd-8405-6ac2f57bd18f nodeName:}" failed. No retries permitted until 2026-09-11 16:43:55.713824769 +0000 UTC m=+51.121927036 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls") pod "image-registry-7c679445b8-dxcgg" (UID: "593fb846-0d3d-42dd-8405-6ac2f57bd18f") : secret "image-registry-tls" not found Sep 11 16:43:47.714109 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:47.713884 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls podName:fde3ddc0-fc6a-42c7-aab9-866ca084bd59 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:55.713874739 +0000 UTC m=+51.121976986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-rsrzh" (UID: "fde3ddc0-fc6a-42c7-aab9-866ca084bd59") : secret "cluster-monitoring-operator-tls" not found Sep 11 16:43:47.714109 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:47.713899 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:47.714109 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:47.713940 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls podName:50af7bf9-0c82-4d40-979b-11350f3018a9 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:55.71393325 +0000 UTC m=+51.122035496 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-wcxg9" (UID: "50af7bf9-0c82-4d40-979b-11350f3018a9") : secret "samples-operator-tls" not found Sep 11 16:43:47.714109 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:47.713957 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle podName:4cc211e5-b500-49d5-84be-13c2bfa7f7bc nodeName:}" failed. No retries permitted until 2026-09-11 16:43:55.71394691 +0000 UTC m=+51.122049157 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle") pod "router-default-7669dcd8d-whgmr" (UID: "4cc211e5-b500-49d5-84be-13c2bfa7f7bc") : configmap references non-existent config key: service-ca.crt Sep 11 16:43:47.814950 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:47.814910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls\") pod \"dns-default-c4dn7\" (UID: \"727f81c8-b4b4-4b6e-9570-5739b9698950\") " pod="openshift-dns/dns-default-c4dn7" Sep 11 16:43:47.815127 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:47.815073 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 16:43:47.815185 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:47.815150 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls podName:727f81c8-b4b4-4b6e-9570-5739b9698950 nodeName:}" failed. No retries permitted until 2026-09-11 16:43:55.815129666 +0000 UTC m=+51.223231928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls") pod "dns-default-c4dn7" (UID: "727f81c8-b4b4-4b6e-9570-5739b9698950") : secret "dns-default-metrics-tls" not found Sep 11 16:43:55.454566 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.454526 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jkhgd" event={"ID":"d41dfd70-3a89-4d1d-84de-d0b3964b8538","Type":"ContainerStarted","Data":"d9e763c72f40553d21f196a3283f0c856828e88ae517377884c186a2975aee88"} Sep 11 16:43:55.456077 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.456042 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-fc9sn" event={"ID":"ed61a19b-2fa7-466b-9772-3b7a1f4a5d25","Type":"ContainerStarted","Data":"6715d093c2ea108ec021f42c2f9e009b661cd2d7d1e5a4183fa670370d81f7ce"} Sep 11 16:43:55.458380 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.458353 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/0.log" Sep 11 16:43:55.458524 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.458395 2577 generic.go:356] "Generic (PLEG): container finished" podID="13cca68e-a749-4f4e-bbdf-21de50e37c6c" containerID="45efc6067671d146d1312fc688ceeff4a2228edc000b86a89727a837e68fc167" exitCode=255 Sep 11 16:43:55.458574 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.458538 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" event={"ID":"13cca68e-a749-4f4e-bbdf-21de50e37c6c","Type":"ContainerDied","Data":"45efc6067671d146d1312fc688ceeff4a2228edc000b86a89727a837e68fc167"} Sep 11 16:43:55.458873 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.458855 2577 scope.go:117] "RemoveContainer" containerID="45efc6067671d146d1312fc688ceeff4a2228edc000b86a89727a837e68fc167" Sep 11 16:43:55.460592 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.460564 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59989b64f6-flmxp" event={"ID":"eeb5215d-ddf9-4bc2-975e-ea70d1fe0687","Type":"ContainerStarted","Data":"1a934c47044cc7cf6ac0caa74c2cf77137050aedb6cf84e4efa9ecd14359fb0f"} Sep 11 16:43:55.462243 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.462220 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59989b64f6-flmxp" Sep 11 16:43:55.463331 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.463311 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59989b64f6-flmxp" Sep 11 16:43:55.463943 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.463923 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f5975844-4n7th" event={"ID":"abde5dc2-8658-41a8-a00e-822e857f9200","Type":"ContainerStarted","Data":"f06d58e49be77f2cd5b4cb9cb3abdd8b38126b251264e72be5bd9361f71d6285"} Sep 11 16:43:55.465933 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.465906 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w5wz5" event={"ID":"42fbe40d-e07f-429e-bfab-01b77d189a2a","Type":"ContainerStarted","Data":"b955cc88da5edfd6cde3af3282cbec628423e5ed4697ffe6d54871b2d3d9c877"} Sep 11 16:43:55.467529 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.467507 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-kqg8b" event={"ID":"f66b05a4-ccaf-4c11-b071-8da98ebec68c","Type":"ContainerStarted","Data":"5cc80d568541b3f3657fefb693c11ede1582f2aff541d2b3b97a683f546eac45"} Sep 11 16:43:55.469118 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.469097 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-d9hsz" event={"ID":"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6","Type":"ContainerStarted","Data":"de0a1d5376068b734fc1de54d74e03246d765bfd67998f16135f4349fea8ea1a"} Sep 11 16:43:55.470828 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.470809 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zk4rf" event={"ID":"a575fca1-0433-4012-b079-5baa6ca5a060","Type":"ContainerStarted","Data":"a47feeddfa37f4f23d209f04788d3367c879f554deebb3b61b50912a65fc8883"} Sep 11 16:43:55.476401 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.476354 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k7dfv" podStartSLOduration=19.647440354 podStartE2EDuration="50.476338952s" totalImagesPullingTime="30.828898598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:43:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:44.46113473 +0000 UTC m=+39.869237016" watchObservedRunningTime="2026-09-11 16:43:55.476338952 +0000 UTC m=+50.884441239" Sep 11 16:43:55.476917 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.476876 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jkhgd" podStartSLOduration=6.990906242 podStartE2EDuration="19.476868569s" totalImagesPullingTime="12.485962327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:55.475938754 +0000 UTC m=+50.884041041" watchObservedRunningTime="2026-09-11 16:43:55.476868569 +0000 UTC m=+50.884970832" Sep 11 16:43:55.539005 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.538956 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-d9hsz" podStartSLOduration=7.121199658 podStartE2EDuration="19.53893824s" totalImagesPullingTime="12.417738582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:55.538159555 +0000 UTC m=+50.946261844" watchObservedRunningTime="2026-09-11 16:43:55.53893824 +0000 UTC m=+50.947040547" Sep 11 16:43:55.539342 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.539306 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zk4rf" podStartSLOduration=7.705571447 podStartE2EDuration="19.539294495s" totalImagesPullingTime="11.833723048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:55.517754562 +0000 UTC m=+50.925856838" watchObservedRunningTime="2026-09-11 16:43:55.539294495 +0000 UTC m=+50.947396763" Sep 11 16:43:55.559743 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.559564 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-w5wz5" podStartSLOduration=4.07015626 podStartE2EDuration="16.559537015s" totalImagesPullingTime="12.489380755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:55.557758268 +0000 UTC m=+50.965860540" watchObservedRunningTime="2026-09-11 16:43:55.559537015 +0000 UTC m=+50.967639286" Sep 11 16:43:55.607958 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.607387 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f5975844-4n7th" podStartSLOduration=5.122569222 podStartE2EDuration="17.607366432s" totalImagesPullingTime="12.48479721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:55.579715233 +0000 UTC m=+50.987817506" watchObservedRunningTime="2026-09-11 16:43:55.607366432 +0000 UTC m=+51.015468702" Sep 11 16:43:55.634336 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.634277 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-fc9sn" podStartSLOduration=7.2656930840000005 podStartE2EDuration="19.634255339s" totalImagesPullingTime="12.368562255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:55.632843868 +0000 UTC m=+51.040946137" watchObservedRunningTime="2026-09-11 16:43:55.634255339 +0000 UTC m=+51.042357610" Sep 11 16:43:55.634991 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.634721 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59989b64f6-flmxp" podStartSLOduration=5.132122903 podStartE2EDuration="17.634713047s" totalImagesPullingTime="12.502590144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:55.60780836 +0000 UTC m=+51.015910629" watchObservedRunningTime="2026-09-11 16:43:55.634713047 +0000 UTC m=+51.042815318" Sep 11 16:43:55.655456 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.655401 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-kqg8b" podStartSLOduration=7.162843927 podStartE2EDuration="19.65538336s" totalImagesPullingTime="12.492539433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:55.654282403 +0000 UTC m=+51.062384693" watchObservedRunningTime="2026-09-11 16:43:55.65538336 +0000 UTC m=+51.063485620" Sep 11 16:43:55.800797 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.800760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:55.800797 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.800813 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:43:55.801050 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.800831 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wcxg9\" (UID: \"50af7bf9-0c82-4d40-979b-11350f3018a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9" Sep 11 16:43:55.801050 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:55.800925 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 16:43:55.801050 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:55.800938 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 16:43:55.801050 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.800951 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:43:55.801050 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:55.800962 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c679445b8-dxcgg: secret "image-registry-tls" not found Sep 11 16:43:55.801050 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:55.800964 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle podName:4cc211e5-b500-49d5-84be-13c2bfa7f7bc nodeName:}" failed. No retries permitted until 2026-09-11 16:44:11.800942962 +0000 UTC m=+67.209045223 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle") pod "router-default-7669dcd8d-whgmr" (UID: "4cc211e5-b500-49d5-84be-13c2bfa7f7bc") : configmap references non-existent config key: service-ca.crt Sep 11 16:43:55.801050 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:55.801012 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls podName:50af7bf9-0c82-4d40-979b-11350f3018a9 nodeName:}" failed. No retries permitted until 2026-09-11 16:44:11.801001339 +0000 UTC m=+67.209103589 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-wcxg9" (UID: "50af7bf9-0c82-4d40-979b-11350f3018a9") : secret "samples-operator-tls" not found Sep 11 16:43:55.801050 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:55.801014 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 16:43:55.801050 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:55.801027 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls podName:593fb846-0d3d-42dd-8405-6ac2f57bd18f nodeName:}" failed. No retries permitted until 2026-09-11 16:44:11.801018556 +0000 UTC m=+67.209120803 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls") pod "image-registry-7c679445b8-dxcgg" (UID: "593fb846-0d3d-42dd-8405-6ac2f57bd18f") : secret "image-registry-tls" not found Sep 11 16:43:55.801050 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:55.801054 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs podName:4cc211e5-b500-49d5-84be-13c2bfa7f7bc nodeName:}" failed. No retries permitted until 2026-09-11 16:44:11.801046202 +0000 UTC m=+67.209148449 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs") pod "router-default-7669dcd8d-whgmr" (UID: "4cc211e5-b500-49d5-84be-13c2bfa7f7bc") : secret "router-metrics-certs-default" not found Sep 11 16:43:55.801611 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.801181 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rsrzh\" (UID: \"fde3ddc0-fc6a-42c7-aab9-866ca084bd59\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" Sep 11 16:43:55.801611 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:55.801259 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 16:43:55.801611 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:55.801291 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls podName:fde3ddc0-fc6a-42c7-aab9-866ca084bd59 nodeName:}" failed. No retries permitted until 2026-09-11 16:44:11.801283335 +0000 UTC m=+67.209385582 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-rsrzh" (UID: "fde3ddc0-fc6a-42c7-aab9-866ca084bd59") : secret "cluster-monitoring-operator-tls" not found Sep 11 16:43:55.902213 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:55.902183 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls\") pod \"dns-default-c4dn7\" (UID: \"727f81c8-b4b4-4b6e-9570-5739b9698950\") " pod="openshift-dns/dns-default-c4dn7" Sep 11 16:43:55.902378 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:55.902354 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 16:43:55.902446 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:55.902436 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls podName:727f81c8-b4b4-4b6e-9570-5739b9698950 nodeName:}" failed. No retries permitted until 2026-09-11 16:44:11.902415786 +0000 UTC m=+67.310518040 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls") pod "dns-default-c4dn7" (UID: "727f81c8-b4b4-4b6e-9570-5739b9698950") : secret "dns-default-metrics-tls" not found Sep 11 16:43:56.476518 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:56.476480 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 16:43:56.476997 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:56.476979 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/0.log" Sep 11 16:43:56.477073 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:56.477023 2577 generic.go:356] "Generic (PLEG): container finished" podID="13cca68e-a749-4f4e-bbdf-21de50e37c6c" containerID="aead4a86c800bad22f611d58fc882878d236ee7adc2c7e38970519d57a380e1d" exitCode=255 Sep 11 16:43:56.477152 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:56.477118 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" event={"ID":"13cca68e-a749-4f4e-bbdf-21de50e37c6c","Type":"ContainerDied","Data":"aead4a86c800bad22f611d58fc882878d236ee7adc2c7e38970519d57a380e1d"} Sep 11 16:43:56.477259 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:56.477236 2577 scope.go:117] "RemoveContainer" containerID="45efc6067671d146d1312fc688ceeff4a2228edc000b86a89727a837e68fc167" Sep 11 16:43:56.477380 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:56.477351 2577 scope.go:117] "RemoveContainer" containerID="aead4a86c800bad22f611d58fc882878d236ee7adc2c7e38970519d57a380e1d" Sep 11 16:43:56.477605 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:56.477579 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-tlxcd_openshift-console-operator(13cca68e-a749-4f4e-bbdf-21de50e37c6c)\"" pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" podUID="13cca68e-a749-4f4e-bbdf-21de50e37c6c" Sep 11 16:43:56.809568 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:56.809534 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fgkvg"] Sep 11 16:43:56.837292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:56.837254 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fgkvg"] Sep 11 16:43:56.837465 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:56.837380 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:43:56.840323 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:56.840295 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 16:43:56.840456 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:56.840322 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 16:43:56.840456 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:56.840368 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9zkcq\"" Sep 11 16:43:56.909052 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:56.909022 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lntf7\" (UniqueName: \"kubernetes.io/projected/31d2da56-fa00-4468-b597-209e9817349b-kube-api-access-lntf7\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:43:56.909208 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:56.909061 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31d2da56-fa00-4468-b597-209e9817349b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:43:56.909208 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:56.909118 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:43:56.909208 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:56.909159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31d2da56-fa00-4468-b597-209e9817349b-data-volume\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:43:56.909208 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:56.909184 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31d2da56-fa00-4468-b597-209e9817349b-crio-socket\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:43:57.009660 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:57.009626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:43:57.009816 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:57.009679 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31d2da56-fa00-4468-b597-209e9817349b-data-volume\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:43:57.009816 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:57.009717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31d2da56-fa00-4468-b597-209e9817349b-crio-socket\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:43:57.009816 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:57.009776 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 16:43:57.009913 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:57.009829 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31d2da56-fa00-4468-b597-209e9817349b-crio-socket\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:43:57.009913 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:57.009836 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls podName:31d2da56-fa00-4468-b597-209e9817349b nodeName:}" failed. No retries permitted until 2026-09-11 16:43:57.50982123 +0000 UTC m=+52.917923477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fgkvg" (UID: "31d2da56-fa00-4468-b597-209e9817349b") : secret "insights-runtime-extractor-tls" not found Sep 11 16:43:57.009990 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:57.009910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lntf7\" (UniqueName: \"kubernetes.io/projected/31d2da56-fa00-4468-b597-209e9817349b-kube-api-access-lntf7\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:43:57.009990 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:57.009951 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31d2da56-fa00-4468-b597-209e9817349b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:43:57.010061 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:57.010031 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 16:43:57.010101 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:57.010088 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31d2da56-fa00-4468-b597-209e9817349b-kube-rbac-proxy-cm podName:31d2da56-fa00-4468-b597-209e9817349b nodeName:}" failed. No retries permitted until 2026-09-11 16:43:57.510070618 +0000 UTC m=+52.918172877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/31d2da56-fa00-4468-b597-209e9817349b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fgkvg" (UID: "31d2da56-fa00-4468-b597-209e9817349b") : configmap "kube-rbac-proxy" not found Sep 11 16:43:57.010138 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:57.010106 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31d2da56-fa00-4468-b597-209e9817349b-data-volume\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:43:57.019929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:57.019899 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lntf7\" (UniqueName: \"kubernetes.io/projected/31d2da56-fa00-4468-b597-209e9817349b-kube-api-access-lntf7\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:43:57.483514 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:57.483473 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 16:43:57.484516 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:57.484339 2577 scope.go:117] "RemoveContainer" containerID="aead4a86c800bad22f611d58fc882878d236ee7adc2c7e38970519d57a380e1d" Sep 11 16:43:57.484732 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:57.484703 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-tlxcd_openshift-console-operator(13cca68e-a749-4f4e-bbdf-21de50e37c6c)\"" pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" podUID="13cca68e-a749-4f4e-bbdf-21de50e37c6c" Sep 11 16:43:57.514550 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:57.514508 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31d2da56-fa00-4468-b597-209e9817349b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:43:57.514730 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:57.514588 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:43:57.514943 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:57.514919 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 16:43:57.515024 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:57.514985 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls podName:31d2da56-fa00-4468-b597-209e9817349b nodeName:}" failed. No retries permitted until 2026-09-11 16:43:58.514969646 +0000 UTC m=+53.923071896 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fgkvg" (UID: "31d2da56-fa00-4468-b597-209e9817349b") : secret "insights-runtime-extractor-tls" not found Sep 11 16:43:57.515329 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:57.515300 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 16:43:57.515437 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:57.515351 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31d2da56-fa00-4468-b597-209e9817349b-kube-rbac-proxy-cm podName:31d2da56-fa00-4468-b597-209e9817349b nodeName:}" failed. No retries permitted until 2026-09-11 16:43:58.515334897 +0000 UTC m=+53.923437161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/31d2da56-fa00-4468-b597-209e9817349b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fgkvg" (UID: "31d2da56-fa00-4468-b597-209e9817349b") : configmap "kube-rbac-proxy" not found Sep 11 16:43:58.525062 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:58.525024 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:43:58.525609 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:58.525193 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 16:43:58.525609 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:58.525223 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31d2da56-fa00-4468-b597-209e9817349b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:43:58.525609 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:58.525271 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls podName:31d2da56-fa00-4468-b597-209e9817349b nodeName:}" failed. No retries permitted until 2026-09-11 16:44:00.525250609 +0000 UTC m=+55.933352857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fgkvg" (UID: "31d2da56-fa00-4468-b597-209e9817349b") : secret "insights-runtime-extractor-tls" not found Sep 11 16:43:58.525609 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:58.525311 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 16:43:58.525609 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:43:58.525391 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31d2da56-fa00-4468-b597-209e9817349b-kube-rbac-proxy-cm podName:31d2da56-fa00-4468-b597-209e9817349b nodeName:}" failed. No retries permitted until 2026-09-11 16:44:00.525374429 +0000 UTC m=+55.933476698 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/31d2da56-fa00-4468-b597-209e9817349b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fgkvg" (UID: "31d2da56-fa00-4468-b597-209e9817349b") : configmap "kube-rbac-proxy" not found Sep 11 16:43:58.992205 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:43:58.992179 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4ptxv_c11370ff-491d-4494-861c-090e05beda2c/dns-node-resolver/0.log" Sep 11 16:44:00.291059 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:00.291028 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" Sep 11 16:44:00.291445 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:00.291119 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" Sep 11 16:44:00.291445 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:00.291412 2577 scope.go:117] "RemoveContainer" containerID="aead4a86c800bad22f611d58fc882878d236ee7adc2c7e38970519d57a380e1d" Sep 11 16:44:00.291596 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:44:00.291578 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-tlxcd_openshift-console-operator(13cca68e-a749-4f4e-bbdf-21de50e37c6c)\"" pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" podUID="13cca68e-a749-4f4e-bbdf-21de50e37c6c" Sep 11 16:44:00.384952 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:00.384925 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r4sbm_ef68ac95-3add-420a-85d4-31cef2cddd8a/node-ca/0.log" Sep 11 16:44:00.496992 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:00.496953 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" event={"ID":"24b90046-f333-4fa9-8feb-a739dc31f500","Type":"ContainerStarted","Data":"9c03175a0123006c2b55afd16f8de7a07eb89130ff99dd59c7328042bbab5e4b"} Sep 11 16:44:00.496992 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:00.497000 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" event={"ID":"24b90046-f333-4fa9-8feb-a739dc31f500","Type":"ContainerStarted","Data":"d4e2038fbf69fbe38aed0dbb5f1068f6f4fbd8dadb9bd4adfe0dcfbd3a757713"} Sep 11 16:44:00.497219 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:00.497012 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" event={"ID":"24b90046-f333-4fa9-8feb-a739dc31f500","Type":"ContainerStarted","Data":"45457a769fbf02f1b1366608578242e91fa0f6490143a44d911059b4e75bf213"} Sep 11 16:44:00.497286 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:00.497270 2577 scope.go:117] "RemoveContainer" containerID="aead4a86c800bad22f611d58fc882878d236ee7adc2c7e38970519d57a380e1d" Sep 11 16:44:00.497420 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:44:00.497403 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-tlxcd_openshift-console-operator(13cca68e-a749-4f4e-bbdf-21de50e37c6c)\"" pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" podUID="13cca68e-a749-4f4e-bbdf-21de50e37c6c" Sep 11 16:44:00.545006 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:00.544931 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31d2da56-fa00-4468-b597-209e9817349b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:44:00.545006 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:00.544987 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:44:00.545237 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:44:00.545057 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 16:44:00.545237 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:44:00.545113 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 16:44:00.545237 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:44:00.545117 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31d2da56-fa00-4468-b597-209e9817349b-kube-rbac-proxy-cm podName:31d2da56-fa00-4468-b597-209e9817349b nodeName:}" failed. No retries permitted until 2026-09-11 16:44:04.545098663 +0000 UTC m=+59.953200913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/31d2da56-fa00-4468-b597-209e9817349b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fgkvg" (UID: "31d2da56-fa00-4468-b597-209e9817349b") : configmap "kube-rbac-proxy" not found Sep 11 16:44:00.545397 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:44:00.545258 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls podName:31d2da56-fa00-4468-b597-209e9817349b nodeName:}" failed. No retries permitted until 2026-09-11 16:44:04.545243513 +0000 UTC m=+59.953345774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fgkvg" (UID: "31d2da56-fa00-4468-b597-209e9817349b") : secret "insights-runtime-extractor-tls" not found Sep 11 16:44:00.980534 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:00.980420 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w5wz5_42fbe40d-e07f-429e-bfab-01b77d189a2a/serve-healthcheck-canary/0.log" Sep 11 16:44:01.580785 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:01.580754 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-fc9sn_ed61a19b-2fa7-466b-9772-3b7a1f4a5d25/kube-storage-version-migrator-operator/0.log" Sep 11 16:44:04.583130 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:04.583094 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31d2da56-fa00-4468-b597-209e9817349b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:44:04.583561 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:04.583142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:44:04.583561 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:44:04.583212 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 16:44:04.583561 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:44:04.583240 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 16:44:04.583561 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:44:04.583280 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31d2da56-fa00-4468-b597-209e9817349b-kube-rbac-proxy-cm podName:31d2da56-fa00-4468-b597-209e9817349b nodeName:}" failed. No retries permitted until 2026-09-11 16:44:12.58326391 +0000 UTC m=+67.991366156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/31d2da56-fa00-4468-b597-209e9817349b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fgkvg" (UID: "31d2da56-fa00-4468-b597-209e9817349b") : configmap "kube-rbac-proxy" not found Sep 11 16:44:04.583561 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:44:04.583294 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls podName:31d2da56-fa00-4468-b597-209e9817349b nodeName:}" failed. No retries permitted until 2026-09-11 16:44:12.583288451 +0000 UTC m=+67.991390698 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fgkvg" (UID: "31d2da56-fa00-4468-b597-209e9817349b") : secret "insights-runtime-extractor-tls" not found Sep 11 16:44:06.876039 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:06.876006 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgslh" Sep 11 16:44:06.906995 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:06.906945 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" podStartSLOduration=10.993338463 podStartE2EDuration="28.906927034s" totalImagesPullingTime="17.913588571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:00.526632174 +0000 UTC m=+55.934734442" watchObservedRunningTime="2026-09-11 16:44:06.906927034 +0000 UTC m=+62.315029309" Sep 11 16:44:10.738440 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:10.738400 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs\") pod \"network-metrics-daemon-68tln\" (UID: \"f1193c55-865c-4fa7-aabb-9ebbdadd11d2\") " pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:44:10.738924 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:10.738605 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret\") pod \"global-pull-secret-syncer-kx9qk\" (UID: \"9c31fb9a-1977-49b8-8396-c462b1d9f47f\") " pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:44:10.741859 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:10.741836 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 16:44:10.741984 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:10.741843 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 16:44:10.749008 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:44:10.748990 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 11 16:44:10.749084 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:44:10.749065 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs podName:f1193c55-865c-4fa7-aabb-9ebbdadd11d2 nodeName:}" failed. No retries permitted until 2026-09-11 16:45:14.749045412 +0000 UTC m=+130.157147673 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs") pod "network-metrics-daemon-68tln" (UID: "f1193c55-865c-4fa7-aabb-9ebbdadd11d2") : secret "metrics-daemon-secret" not found Sep 11 16:44:10.751008 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:10.750986 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c31fb9a-1977-49b8-8396-c462b1d9f47f-original-pull-secret\") pod \"global-pull-secret-syncer-kx9qk\" (UID: \"9c31fb9a-1977-49b8-8396-c462b1d9f47f\") " pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:44:10.806275 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:10.806235 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kx9qk" Sep 11 16:44:10.839257 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:10.839223 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwqq2\" (UniqueName: \"kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2\") pod \"network-check-target-7vclm\" (UID: \"592424fa-0fb1-4974-8d4a-097697d013c0\") " pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:44:10.841724 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:10.841701 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwqq2\" (UniqueName: \"kubernetes.io/projected/592424fa-0fb1-4974-8d4a-097697d013c0-kube-api-access-gwqq2\") pod \"network-check-target-7vclm\" (UID: \"592424fa-0fb1-4974-8d4a-097697d013c0\") " pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:44:11.080472 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.080389 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kx9qk"] Sep 11 16:44:11.083643 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:44:11.083615 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c31fb9a_1977_49b8_8396_c462b1d9f47f.slice/crio-27bf086d5496f81b6e837718cec962f060652de739bc1e81bc23ebbb4fe4330f WatchSource:0}: Error finding container 27bf086d5496f81b6e837718cec962f060652de739bc1e81bc23ebbb4fe4330f: Status 404 returned error can't find the container with id 27bf086d5496f81b6e837718cec962f060652de739bc1e81bc23ebbb4fe4330f Sep 11 16:44:11.104106 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.104086 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6xdv2\"" Sep 11 16:44:11.111797 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.111781 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:44:11.381025 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.380954 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7vclm"] Sep 11 16:44:11.383889 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:44:11.383865 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592424fa_0fb1_4974_8d4a_097697d013c0.slice/crio-eda66ef0878dee9f5055d1e2ee8b6474fccd20851d9cecf74d99303cc20986c4 WatchSource:0}: Error finding container eda66ef0878dee9f5055d1e2ee8b6474fccd20851d9cecf74d99303cc20986c4: Status 404 returned error can't find the container with id eda66ef0878dee9f5055d1e2ee8b6474fccd20851d9cecf74d99303cc20986c4 Sep 11 16:44:11.536591 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.536549 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7vclm" event={"ID":"592424fa-0fb1-4974-8d4a-097697d013c0","Type":"ContainerStarted","Data":"92123ed0401c2a5870cbe974d907ede3a7b125d7022fcc24d315c4a8fd86d41a"} Sep 11 16:44:11.536764 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.536691 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:44:11.536764 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.536728 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7vclm" event={"ID":"592424fa-0fb1-4974-8d4a-097697d013c0","Type":"ContainerStarted","Data":"eda66ef0878dee9f5055d1e2ee8b6474fccd20851d9cecf74d99303cc20986c4"} Sep 11 16:44:11.537685 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.537660 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kx9qk" event={"ID":"9c31fb9a-1977-49b8-8396-c462b1d9f47f","Type":"ContainerStarted","Data":"27bf086d5496f81b6e837718cec962f060652de739bc1e81bc23ebbb4fe4330f"} Sep 11 16:44:11.850993 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.850958 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rsrzh\" (UID: \"fde3ddc0-fc6a-42c7-aab9-866ca084bd59\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" Sep 11 16:44:11.851441 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.851019 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:44:11.851441 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.851050 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:44:11.851441 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.851074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wcxg9\" (UID: \"50af7bf9-0c82-4d40-979b-11350f3018a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9" Sep 11 16:44:11.851441 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.851106 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:44:11.851871 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.851848 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-service-ca-bundle\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:44:11.854164 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.854137 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50af7bf9-0c82-4d40-979b-11350f3018a9-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wcxg9\" (UID: \"50af7bf9-0c82-4d40-979b-11350f3018a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9" Sep 11 16:44:11.854164 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.854155 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cc211e5-b500-49d5-84be-13c2bfa7f7bc-metrics-certs\") pod \"router-default-7669dcd8d-whgmr\" (UID: \"4cc211e5-b500-49d5-84be-13c2bfa7f7bc\") " pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:44:11.854314 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.854168 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fde3ddc0-fc6a-42c7-aab9-866ca084bd59-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rsrzh\" (UID: \"fde3ddc0-fc6a-42c7-aab9-866ca084bd59\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" Sep 11 16:44:11.854661 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.854639 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls\") pod \"image-registry-7c679445b8-dxcgg\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:44:11.952263 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.952225 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls\") pod \"dns-default-c4dn7\" (UID: \"727f81c8-b4b4-4b6e-9570-5739b9698950\") " pod="openshift-dns/dns-default-c4dn7" Sep 11 16:44:11.954862 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:11.954835 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/727f81c8-b4b4-4b6e-9570-5739b9698950-metrics-tls\") pod \"dns-default-c4dn7\" (UID: \"727f81c8-b4b4-4b6e-9570-5739b9698950\") " pod="openshift-dns/dns-default-c4dn7" Sep 11 16:44:12.039648 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.039616 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qc76w\"" Sep 11 16:44:12.045059 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.045032 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ptp2z\"" Sep 11 16:44:12.047138 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.047110 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:44:12.052317 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.052296 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:44:12.107694 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.107667 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-twlct\"" Sep 11 16:44:12.113618 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.112635 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9" Sep 11 16:44:12.114570 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.114549 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-g9whx\"" Sep 11 16:44:12.126758 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.123117 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" Sep 11 16:44:12.178582 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.177830 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wbvg9\"" Sep 11 16:44:12.185345 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.185112 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c4dn7" Sep 11 16:44:12.342210 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.342045 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7vclm" podStartSLOduration=67.342024361 podStartE2EDuration="1m7.342024361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:11.559961426 +0000 UTC m=+66.968063714" watchObservedRunningTime="2026-09-11 16:44:12.342024361 +0000 UTC m=+67.750126629" Sep 11 16:44:12.342424 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.342300 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7669dcd8d-whgmr"] Sep 11 16:44:12.344617 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.344594 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c679445b8-dxcgg"] Sep 11 16:44:12.345315 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:44:12.345283 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc211e5_b500_49d5_84be_13c2bfa7f7bc.slice/crio-2440d1024cc0320aab663fe9131d5e6e1f3c5d76cfeac7fb5ab6576792955839 WatchSource:0}: Error finding container 2440d1024cc0320aab663fe9131d5e6e1f3c5d76cfeac7fb5ab6576792955839: Status 404 returned error can't find the container with id 2440d1024cc0320aab663fe9131d5e6e1f3c5d76cfeac7fb5ab6576792955839 Sep 11 16:44:12.347276 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:44:12.347253 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593fb846_0d3d_42dd_8405_6ac2f57bd18f.slice/crio-3de15decdf67ff64a1e4ddeebb0c2bd16dc1fa9dbc8c56140102e6733a00a071 WatchSource:0}: Error finding container 3de15decdf67ff64a1e4ddeebb0c2bd16dc1fa9dbc8c56140102e6733a00a071: Status 404 returned error can't find the container with id 3de15decdf67ff64a1e4ddeebb0c2bd16dc1fa9dbc8c56140102e6733a00a071 Sep 11 16:44:12.421208 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.421133 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9"] Sep 11 16:44:12.423108 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.423054 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh"] Sep 11 16:44:12.427899 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:44:12.427864 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde3ddc0_fc6a_42c7_aab9_866ca084bd59.slice/crio-e0ac184b3fdcaa27c749b5a2b6c840733d597828adb5b81916cf3772051c2ce9 WatchSource:0}: Error finding container e0ac184b3fdcaa27c749b5a2b6c840733d597828adb5b81916cf3772051c2ce9: Status 404 returned error can't find the container with id e0ac184b3fdcaa27c749b5a2b6c840733d597828adb5b81916cf3772051c2ce9 Sep 11 16:44:12.474718 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.474692 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c4dn7"] Sep 11 16:44:12.477003 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:44:12.476971 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727f81c8_b4b4_4b6e_9570_5739b9698950.slice/crio-0f66939237f5bd56a6a0332b9f837b0ae3ba06d59c9c21599610703545b540f5 WatchSource:0}: Error finding container 0f66939237f5bd56a6a0332b9f837b0ae3ba06d59c9c21599610703545b540f5: Status 404 returned error can't find the container with id 0f66939237f5bd56a6a0332b9f837b0ae3ba06d59c9c21599610703545b540f5 Sep 11 16:44:12.543083 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.543046 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9" event={"ID":"50af7bf9-0c82-4d40-979b-11350f3018a9","Type":"ContainerStarted","Data":"f012c2f7e8d9ea0a3fc45eafbc84b0ecfe769c4c4a8adcf737c05e7645ecd488"} Sep 11 16:44:12.544545 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.544479 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" event={"ID":"fde3ddc0-fc6a-42c7-aab9-866ca084bd59","Type":"ContainerStarted","Data":"e0ac184b3fdcaa27c749b5a2b6c840733d597828adb5b81916cf3772051c2ce9"} Sep 11 16:44:12.546389 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.546358 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7669dcd8d-whgmr" event={"ID":"4cc211e5-b500-49d5-84be-13c2bfa7f7bc","Type":"ContainerStarted","Data":"3019b44c4055271e260289842299515bd9d03813e82b77e69622ab046da584f4"} Sep 11 16:44:12.546539 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.546405 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7669dcd8d-whgmr" event={"ID":"4cc211e5-b500-49d5-84be-13c2bfa7f7bc","Type":"ContainerStarted","Data":"2440d1024cc0320aab663fe9131d5e6e1f3c5d76cfeac7fb5ab6576792955839"} Sep 11 16:44:12.547944 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.547914 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c4dn7" event={"ID":"727f81c8-b4b4-4b6e-9570-5739b9698950","Type":"ContainerStarted","Data":"0f66939237f5bd56a6a0332b9f837b0ae3ba06d59c9c21599610703545b540f5"} Sep 11 16:44:12.550051 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.549993 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" event={"ID":"593fb846-0d3d-42dd-8405-6ac2f57bd18f","Type":"ContainerStarted","Data":"13e8784ec4f2c6c17aeaf8fcf486017442163e736ad37bb5a916b4c11a2553ed"} Sep 11 16:44:12.550201 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.550181 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:44:12.550268 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.550228 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" event={"ID":"593fb846-0d3d-42dd-8405-6ac2f57bd18f","Type":"ContainerStarted","Data":"3de15decdf67ff64a1e4ddeebb0c2bd16dc1fa9dbc8c56140102e6733a00a071"} Sep 11 16:44:12.571058 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.570995 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7669dcd8d-whgmr" podStartSLOduration=36.570974049 podStartE2EDuration="36.570974049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:12.56984155 +0000 UTC m=+67.977943844" watchObservedRunningTime="2026-09-11 16:44:12.570974049 +0000 UTC m=+67.979076320" Sep 11 16:44:12.595144 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.595083 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" podStartSLOduration=67.595066455 podStartE2EDuration="1m7.595066455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:12.593976929 +0000 UTC m=+68.002079225" watchObservedRunningTime="2026-09-11 16:44:12.595066455 +0000 UTC m=+68.003168723" Sep 11 16:44:12.659343 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.659268 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31d2da56-fa00-4468-b597-209e9817349b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:44:12.659473 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.659387 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:44:12.659828 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:12.659809 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31d2da56-fa00-4468-b597-209e9817349b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:44:12.660443 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:44:12.660423 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 16:44:12.660547 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:44:12.660531 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls podName:31d2da56-fa00-4468-b597-209e9817349b nodeName:}" failed. No retries permitted until 2026-09-11 16:44:28.660475181 +0000 UTC m=+84.068577441 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fgkvg" (UID: "31d2da56-fa00-4468-b597-209e9817349b") : secret "insights-runtime-extractor-tls" not found Sep 11 16:44:13.047630 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:13.047595 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:44:13.050786 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:13.050751 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:44:13.088923 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:13.088888 2577 scope.go:117] "RemoveContainer" containerID="aead4a86c800bad22f611d58fc882878d236ee7adc2c7e38970519d57a380e1d" Sep 11 16:44:13.560063 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:13.560031 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 16:44:13.560247 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:13.560171 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" event={"ID":"13cca68e-a749-4f4e-bbdf-21de50e37c6c","Type":"ContainerStarted","Data":"24d98400a6b8d04c761dd70f55b98c1032f8e404efe2ea1b8fa335bcb442bd06"} Sep 11 16:44:13.561062 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:13.561034 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:44:13.562642 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:13.562534 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7669dcd8d-whgmr" Sep 11 16:44:13.583870 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:13.583152 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" podStartSLOduration=25.097698047 podStartE2EDuration="37.583129117s" totalImagesPullingTime="12.48543107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:13.582911645 +0000 UTC m=+68.991013933" watchObservedRunningTime="2026-09-11 16:44:13.583129117 +0000 UTC m=+68.991231386" Sep 11 16:44:17.577959 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:17.577923 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9" event={"ID":"50af7bf9-0c82-4d40-979b-11350f3018a9","Type":"ContainerStarted","Data":"9e90b3415ec5828a04273054562640f572e7fa9527e911b44b31502f3b9b3149"} Sep 11 16:44:17.580131 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:17.580081 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" event={"ID":"fde3ddc0-fc6a-42c7-aab9-866ca084bd59","Type":"ContainerStarted","Data":"a78b405e5a82611083fbec5e9e453711949409d06d317fe74723f1b5a3a904e6"} Sep 11 16:44:17.585566 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:17.585481 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c4dn7" event={"ID":"727f81c8-b4b4-4b6e-9570-5739b9698950","Type":"ContainerStarted","Data":"3b088c9ed3226332d59caee45ccbe42098cd092c1161117c70291f122011d505"} Sep 11 16:44:17.603932 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:17.603879 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rsrzh" podStartSLOduration=36.621649405 podStartE2EDuration="41.603859445s" totalImagesPullingTime="4.98221004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:17.602604132 +0000 UTC m=+73.010706426" watchObservedRunningTime="2026-09-11 16:44:17.603859445 +0000 UTC m=+73.011961715" Sep 11 16:44:18.590848 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:18.590812 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kx9qk" event={"ID":"9c31fb9a-1977-49b8-8396-c462b1d9f47f","Type":"ContainerStarted","Data":"d0ceee6ced6a43c8fd78064b49d81338782ffe1cdba7634242684e28eb20184b"} Sep 11 16:44:18.592624 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:18.592597 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9" event={"ID":"50af7bf9-0c82-4d40-979b-11350f3018a9","Type":"ContainerStarted","Data":"34933615e1bf52d4b752d7fb366fd55ac99f583c1354de78a51742018598dd7b"} Sep 11 16:44:18.594568 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:18.594542 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c4dn7" event={"ID":"727f81c8-b4b4-4b6e-9570-5739b9698950","Type":"ContainerStarted","Data":"2136b6636632596d07dc2a0f3b0d9e5505ab8a81f7f8d3fe3ca1bc63af0b240a"} Sep 11 16:44:18.607605 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:18.607552 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kx9qk" podStartSLOduration=67.266431723 podStartE2EDuration="1m13.607537716s" totalImagesPullingTime="6.341105993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:18.607200444 +0000 UTC m=+74.015302712" watchObservedRunningTime="2026-09-11 16:44:18.607537716 +0000 UTC m=+74.015639982" Sep 11 16:44:18.625825 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:18.625776 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wcxg9" podStartSLOduration=37.686327243 podStartE2EDuration="42.625757635s" totalImagesPullingTime="4.939430392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:18.625589022 +0000 UTC m=+74.033691344" watchObservedRunningTime="2026-09-11 16:44:18.625757635 +0000 UTC m=+74.033859905" Sep 11 16:44:18.643089 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:18.643046 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c4dn7" podStartSLOduration=34.708427828 podStartE2EDuration="39.643028197s" totalImagesPullingTime="4.934600369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:18.642437716 +0000 UTC m=+74.050540016" watchObservedRunningTime="2026-09-11 16:44:18.643028197 +0000 UTC m=+74.051130466" Sep 11 16:44:19.602309 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:19.602274 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c4dn7" Sep 11 16:44:23.561057 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:23.561013 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" Sep 11 16:44:23.564946 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:23.564918 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-tlxcd" Sep 11 16:44:28.708357 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:28.708229 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:44:28.710569 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:28.710549 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31d2da56-fa00-4468-b597-209e9817349b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fgkvg\" (UID: \"31d2da56-fa00-4468-b597-209e9817349b\") " pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:44:28.951291 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:28.951257 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9zkcq\"" Sep 11 16:44:28.958834 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:28.958778 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fgkvg" Sep 11 16:44:29.261350 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:29.261314 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fgkvg"] Sep 11 16:44:29.264300 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:44:29.264274 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d2da56_fa00_4468_b597_209e9817349b.slice/crio-3b2eb13795e455d7c17ca3cc9e212ed8df9555fc4a27247af5be33ef24dc19e2 WatchSource:0}: Error finding container 3b2eb13795e455d7c17ca3cc9e212ed8df9555fc4a27247af5be33ef24dc19e2: Status 404 returned error can't find the container with id 3b2eb13795e455d7c17ca3cc9e212ed8df9555fc4a27247af5be33ef24dc19e2 Sep 11 16:44:29.607950 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:29.607870 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c4dn7" Sep 11 16:44:29.638149 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:29.638116 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fgkvg" event={"ID":"31d2da56-fa00-4468-b597-209e9817349b","Type":"ContainerStarted","Data":"90e810c36c573c9aa0f178f29d1a6038899b5e15e804a42ea191c541ba3ed1ea"} Sep 11 16:44:29.638297 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:29.638181 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fgkvg" event={"ID":"31d2da56-fa00-4468-b597-209e9817349b","Type":"ContainerStarted","Data":"3b2eb13795e455d7c17ca3cc9e212ed8df9555fc4a27247af5be33ef24dc19e2"} Sep 11 16:44:30.455442 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.455411 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jfk88"] Sep 11 16:44:30.476436 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.476407 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.479434 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.479409 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 16:44:30.479584 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.479515 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 16:44:30.479658 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.479590 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7dzm5\"" Sep 11 16:44:30.479658 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.479519 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 16:44:30.480675 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.480656 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 16:44:30.623718 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.623661 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/96a37056-4433-4acf-863e-6f1725d53ee5-root\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.623718 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.623706 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96a37056-4433-4acf-863e-6f1725d53ee5-sys\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.623920 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.623745 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/96a37056-4433-4acf-863e-6f1725d53ee5-node-exporter-wtmp\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.623920 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.623779 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqlg8\" (UniqueName: \"kubernetes.io/projected/96a37056-4433-4acf-863e-6f1725d53ee5-kube-api-access-cqlg8\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.623920 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.623856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/96a37056-4433-4acf-863e-6f1725d53ee5-node-exporter-accelerators-collector-config\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.623920 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.623895 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/96a37056-4433-4acf-863e-6f1725d53ee5-node-exporter-tls\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.624115 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.623979 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/96a37056-4433-4acf-863e-6f1725d53ee5-node-exporter-textfile\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.624115 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.624013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96a37056-4433-4acf-863e-6f1725d53ee5-metrics-client-ca\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.624115 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.624047 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96a37056-4433-4acf-863e-6f1725d53ee5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.725179 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.725092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/96a37056-4433-4acf-863e-6f1725d53ee5-root\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.725179 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.725129 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96a37056-4433-4acf-863e-6f1725d53ee5-sys\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.725179 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.725153 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/96a37056-4433-4acf-863e-6f1725d53ee5-node-exporter-wtmp\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.725179 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.725174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqlg8\" (UniqueName: \"kubernetes.io/projected/96a37056-4433-4acf-863e-6f1725d53ee5-kube-api-access-cqlg8\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.725533 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.725203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/96a37056-4433-4acf-863e-6f1725d53ee5-node-exporter-accelerators-collector-config\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.725533 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.725208 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/96a37056-4433-4acf-863e-6f1725d53ee5-root\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.725533 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.725226 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/96a37056-4433-4acf-863e-6f1725d53ee5-node-exporter-tls\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.725533 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.725214 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96a37056-4433-4acf-863e-6f1725d53ee5-sys\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.725533 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.725301 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/96a37056-4433-4acf-863e-6f1725d53ee5-node-exporter-wtmp\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.725533 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.725315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/96a37056-4433-4acf-863e-6f1725d53ee5-node-exporter-textfile\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.725533 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.725351 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96a37056-4433-4acf-863e-6f1725d53ee5-metrics-client-ca\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.725533 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.725387 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96a37056-4433-4acf-863e-6f1725d53ee5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.725881 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.725733 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/96a37056-4433-4acf-863e-6f1725d53ee5-node-exporter-textfile\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.725881 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.725822 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/96a37056-4433-4acf-863e-6f1725d53ee5-node-exporter-accelerators-collector-config\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.725961 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.725907 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96a37056-4433-4acf-863e-6f1725d53ee5-metrics-client-ca\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.727690 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.727661 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/96a37056-4433-4acf-863e-6f1725d53ee5-node-exporter-tls\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.727792 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.727690 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96a37056-4433-4acf-863e-6f1725d53ee5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.734203 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.734181 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqlg8\" (UniqueName: \"kubernetes.io/projected/96a37056-4433-4acf-863e-6f1725d53ee5-kube-api-access-cqlg8\") pod \"node-exporter-jfk88\" (UID: \"96a37056-4433-4acf-863e-6f1725d53ee5\") " pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.786222 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:30.786190 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jfk88" Sep 11 16:44:30.821387 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:44:30.821358 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a37056_4433_4acf_863e_6f1725d53ee5.slice/crio-00f4374e38b8c992416576dd7aa95ecaf66f63b3ba0e395bb506610b35eb8cf2 WatchSource:0}: Error finding container 00f4374e38b8c992416576dd7aa95ecaf66f63b3ba0e395bb506610b35eb8cf2: Status 404 returned error can't find the container with id 00f4374e38b8c992416576dd7aa95ecaf66f63b3ba0e395bb506610b35eb8cf2 Sep 11 16:44:31.647575 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:31.647541 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jfk88" event={"ID":"96a37056-4433-4acf-863e-6f1725d53ee5","Type":"ContainerStarted","Data":"00f4374e38b8c992416576dd7aa95ecaf66f63b3ba0e395bb506610b35eb8cf2"} Sep 11 16:44:31.649831 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:31.649794 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fgkvg" event={"ID":"31d2da56-fa00-4468-b597-209e9817349b","Type":"ContainerStarted","Data":"9eee5870d69942f740f0ffeb604da2e951162f9efed8ba370a1081f8478605bd"} Sep 11 16:44:32.656945 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:32.656822 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fgkvg" event={"ID":"31d2da56-fa00-4468-b597-209e9817349b","Type":"ContainerStarted","Data":"c99f1b3a248b0037e542ff476a226f29818e35a3b387438af4e78b659cef62e8"} Sep 11 16:44:32.658668 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:32.658638 2577 generic.go:356] "Generic (PLEG): container finished" podID="96a37056-4433-4acf-863e-6f1725d53ee5" containerID="10ee74bfe80a774d25d0bdc49683c5348e6c4f0a97e66181384f51f40fdb2369" exitCode=0 Sep 11 16:44:32.658782 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:32.658678 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jfk88" event={"ID":"96a37056-4433-4acf-863e-6f1725d53ee5","Type":"ContainerDied","Data":"10ee74bfe80a774d25d0bdc49683c5348e6c4f0a97e66181384f51f40fdb2369"} Sep 11 16:44:32.680620 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:32.680573 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fgkvg" podStartSLOduration=33.902167797 podStartE2EDuration="36.680556507s" totalImagesPullingTime="2.77838871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:43:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:32.679606682 +0000 UTC m=+88.087708934" watchObservedRunningTime="2026-09-11 16:44:32.680556507 +0000 UTC m=+88.088658775" Sep 11 16:44:33.564790 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:33.564760 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:44:33.669785 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:33.669747 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jfk88" event={"ID":"96a37056-4433-4acf-863e-6f1725d53ee5","Type":"ContainerStarted","Data":"6dbae61753bbd0b8246c51e095579e267ca55e2bc57e8ab894fed37059ffb3af"} Sep 11 16:44:33.670214 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:33.669834 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jfk88" event={"ID":"96a37056-4433-4acf-863e-6f1725d53ee5","Type":"ContainerStarted","Data":"1b1d28146b3a7d6269e5eaecea88190cb1d4b34e3e6e9d265ed936d48d71fbf8"} Sep 11 16:44:33.692463 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:33.692419 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jfk88" podStartSLOduration=2.175910034 podStartE2EDuration="3.692404917s" totalImagesPullingTime="1.449277128s" totalInitContainerRuntime="67.217755ms" isStatefulPod=true podCreationTimestamp="2026-09-11 16:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:33.690692255 +0000 UTC m=+89.098794546" watchObservedRunningTime="2026-09-11 16:44:33.692404917 +0000 UTC m=+89.100507186" Sep 11 16:44:42.552851 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:42.552818 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7vclm" Sep 11 16:44:45.571015 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:44:45.570979 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c679445b8-dxcgg"] Sep 11 16:45:10.327808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.327757 2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" podUID="24b90046-f333-4fa9-8feb-a739dc31f500" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 11 16:45:10.590460 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.590366 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" podUID="593fb846-0d3d-42dd-8405-6ac2f57bd18f" containerName="registry" containerID="cri-o://13e8784ec4f2c6c17aeaf8fcf486017442163e736ad37bb5a916b4c11a2553ed" gracePeriod=30 Sep 11 16:45:10.808681 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.808649 2577 generic.go:356] "Generic (PLEG): container finished" podID="593fb846-0d3d-42dd-8405-6ac2f57bd18f" containerID="13e8784ec4f2c6c17aeaf8fcf486017442163e736ad37bb5a916b4c11a2553ed" exitCode=0 Sep 11 16:45:10.808842 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.808725 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" event={"ID":"593fb846-0d3d-42dd-8405-6ac2f57bd18f","Type":"ContainerDied","Data":"13e8784ec4f2c6c17aeaf8fcf486017442163e736ad37bb5a916b4c11a2553ed"} Sep 11 16:45:10.836529 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.836491 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:45:10.974343 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.974260 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/593fb846-0d3d-42dd-8405-6ac2f57bd18f-image-registry-private-configuration\") pod \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " Sep 11 16:45:10.974343 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.974305 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-bound-sa-token\") pod \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " Sep 11 16:45:10.974343 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.974336 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/593fb846-0d3d-42dd-8405-6ac2f57bd18f-ca-trust-extracted\") pod \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " Sep 11 16:45:10.974620 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.974365 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls\") pod \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " Sep 11 16:45:10.974620 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.974403 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-certificates\") pod \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " Sep 11 16:45:10.974620 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.974453 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96f4w\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-kube-api-access-96f4w\") pod \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " Sep 11 16:45:10.974620 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.974522 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/593fb846-0d3d-42dd-8405-6ac2f57bd18f-trusted-ca\") pod \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " Sep 11 16:45:10.974620 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.974553 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/593fb846-0d3d-42dd-8405-6ac2f57bd18f-installation-pull-secrets\") pod \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\" (UID: \"593fb846-0d3d-42dd-8405-6ac2f57bd18f\") " Sep 11 16:45:10.975108 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.975020 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/593fb846-0d3d-42dd-8405-6ac2f57bd18f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "593fb846-0d3d-42dd-8405-6ac2f57bd18f" (UID: "593fb846-0d3d-42dd-8405-6ac2f57bd18f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:45:10.975389 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.975358 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "593fb846-0d3d-42dd-8405-6ac2f57bd18f" (UID: "593fb846-0d3d-42dd-8405-6ac2f57bd18f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:45:10.977003 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.976955 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/593fb846-0d3d-42dd-8405-6ac2f57bd18f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "593fb846-0d3d-42dd-8405-6ac2f57bd18f" (UID: "593fb846-0d3d-42dd-8405-6ac2f57bd18f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 16:45:10.977003 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.976965 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-kube-api-access-96f4w" (OuterVolumeSpecName: "kube-api-access-96f4w") pod "593fb846-0d3d-42dd-8405-6ac2f57bd18f" (UID: "593fb846-0d3d-42dd-8405-6ac2f57bd18f"). InnerVolumeSpecName "kube-api-access-96f4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:45:10.977193 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.977157 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "593fb846-0d3d-42dd-8405-6ac2f57bd18f" (UID: "593fb846-0d3d-42dd-8405-6ac2f57bd18f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:45:10.977251 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.977231 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "593fb846-0d3d-42dd-8405-6ac2f57bd18f" (UID: "593fb846-0d3d-42dd-8405-6ac2f57bd18f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:45:10.977455 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.977434 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/593fb846-0d3d-42dd-8405-6ac2f57bd18f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "593fb846-0d3d-42dd-8405-6ac2f57bd18f" (UID: "593fb846-0d3d-42dd-8405-6ac2f57bd18f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 16:45:10.983015 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:10.982992 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/593fb846-0d3d-42dd-8405-6ac2f57bd18f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "593fb846-0d3d-42dd-8405-6ac2f57bd18f" (UID: "593fb846-0d3d-42dd-8405-6ac2f57bd18f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:45:11.075831 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:11.075794 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-tls\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:45:11.075831 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:11.075826 2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/593fb846-0d3d-42dd-8405-6ac2f57bd18f-registry-certificates\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:45:11.075831 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:11.075837 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96f4w\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-kube-api-access-96f4w\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:45:11.076064 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:11.075847 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/593fb846-0d3d-42dd-8405-6ac2f57bd18f-trusted-ca\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:45:11.076064 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:11.075855 2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/593fb846-0d3d-42dd-8405-6ac2f57bd18f-installation-pull-secrets\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:45:11.076064 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:11.075864 2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/593fb846-0d3d-42dd-8405-6ac2f57bd18f-image-registry-private-configuration\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:45:11.076064 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:11.075875 2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/593fb846-0d3d-42dd-8405-6ac2f57bd18f-bound-sa-token\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:45:11.076064 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:11.075883 2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/593fb846-0d3d-42dd-8405-6ac2f57bd18f-ca-trust-extracted\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:45:11.813531 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:11.813491 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" Sep 11 16:45:11.813531 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:11.813491 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c679445b8-dxcgg" event={"ID":"593fb846-0d3d-42dd-8405-6ac2f57bd18f","Type":"ContainerDied","Data":"3de15decdf67ff64a1e4ddeebb0c2bd16dc1fa9dbc8c56140102e6733a00a071"} Sep 11 16:45:11.814089 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:11.813567 2577 scope.go:117] "RemoveContainer" containerID="13e8784ec4f2c6c17aeaf8fcf486017442163e736ad37bb5a916b4c11a2553ed" Sep 11 16:45:11.835038 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:11.835007 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c679445b8-dxcgg"] Sep 11 16:45:11.838356 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:11.838326 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c679445b8-dxcgg"] Sep 11 16:45:13.092744 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:13.092703 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="593fb846-0d3d-42dd-8405-6ac2f57bd18f" path="/var/lib/kubelet/pods/593fb846-0d3d-42dd-8405-6ac2f57bd18f/volumes" Sep 11 16:45:14.811696 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:14.811650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs\") pod \"network-metrics-daemon-68tln\" (UID: \"f1193c55-865c-4fa7-aabb-9ebbdadd11d2\") " pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:45:14.813947 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:14.813929 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1193c55-865c-4fa7-aabb-9ebbdadd11d2-metrics-certs\") pod \"network-metrics-daemon-68tln\" (UID: \"f1193c55-865c-4fa7-aabb-9ebbdadd11d2\") " pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:45:14.900568 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:14.900533 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dw8q7\"" Sep 11 16:45:14.908082 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:14.908058 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68tln" Sep 11 16:45:15.180832 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:15.180804 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-68tln"] Sep 11 16:45:15.183264 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:45:15.183240 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1193c55_865c_4fa7_aabb_9ebbdadd11d2.slice/crio-34faf41b4897e06957d2f06b575cb5fb8c87512dc6d07f4dfa5b5d768cad1ce6 WatchSource:0}: Error finding container 34faf41b4897e06957d2f06b575cb5fb8c87512dc6d07f4dfa5b5d768cad1ce6: Status 404 returned error can't find the container with id 34faf41b4897e06957d2f06b575cb5fb8c87512dc6d07f4dfa5b5d768cad1ce6 Sep 11 16:45:15.831213 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:15.831182 2577 generic.go:356] "Generic (PLEG): container finished" podID="ed61a19b-2fa7-466b-9772-3b7a1f4a5d25" containerID="6715d093c2ea108ec021f42c2f9e009b661cd2d7d1e5a4183fa670370d81f7ce" exitCode=0 Sep 11 16:45:15.831704 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:15.831263 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-fc9sn" event={"ID":"ed61a19b-2fa7-466b-9772-3b7a1f4a5d25","Type":"ContainerDied","Data":"6715d093c2ea108ec021f42c2f9e009b661cd2d7d1e5a4183fa670370d81f7ce"} Sep 11 16:45:15.831704 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:15.831696 2577 scope.go:117] "RemoveContainer" containerID="6715d093c2ea108ec021f42c2f9e009b661cd2d7d1e5a4183fa670370d81f7ce" Sep 11 16:45:15.832982 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:15.832950 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-68tln" event={"ID":"f1193c55-865c-4fa7-aabb-9ebbdadd11d2","Type":"ContainerStarted","Data":"34faf41b4897e06957d2f06b575cb5fb8c87512dc6d07f4dfa5b5d768cad1ce6"} Sep 11 16:45:15.834730 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:15.834707 2577 generic.go:356] "Generic (PLEG): container finished" podID="f76030c8-35a4-4b21-8a5d-ac0ffa450ce6" containerID="de0a1d5376068b734fc1de54d74e03246d765bfd67998f16135f4349fea8ea1a" exitCode=0 Sep 11 16:45:15.834820 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:15.834747 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-d9hsz" event={"ID":"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6","Type":"ContainerDied","Data":"de0a1d5376068b734fc1de54d74e03246d765bfd67998f16135f4349fea8ea1a"} Sep 11 16:45:15.835123 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:15.835103 2577 scope.go:117] "RemoveContainer" containerID="de0a1d5376068b734fc1de54d74e03246d765bfd67998f16135f4349fea8ea1a" Sep 11 16:45:16.840815 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:16.840779 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-68tln" event={"ID":"f1193c55-865c-4fa7-aabb-9ebbdadd11d2","Type":"ContainerStarted","Data":"38b071a6b3cf71d2974e13713dfe9e6981e00a2dd2e0caad05006b116a9c6a78"} Sep 11 16:45:16.841272 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:16.840836 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-68tln" event={"ID":"f1193c55-865c-4fa7-aabb-9ebbdadd11d2","Type":"ContainerStarted","Data":"0475cfb3da7d4240052d8bc92a9a5b8919a2bf16bc3ac061575cffb0496cf47b"} Sep 11 16:45:16.842550 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:16.842525 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-d9hsz" event={"ID":"f76030c8-35a4-4b21-8a5d-ac0ffa450ce6","Type":"ContainerStarted","Data":"5217d963d2ac018b8bb6d2905f45e11e44dd88dfaafa8e9a9e0154a66f2c6cc5"} Sep 11 16:45:16.844157 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:16.844134 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-fc9sn" event={"ID":"ed61a19b-2fa7-466b-9772-3b7a1f4a5d25","Type":"ContainerStarted","Data":"f410fc475c1a06e51f5623cda30ff12f042d1cc0a331eff5fcec947a52ae4fbd"} Sep 11 16:45:16.861555 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:16.861477 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-68tln" podStartSLOduration=130.815826203 podStartE2EDuration="2m11.861462783s" totalImagesPullingTime="1.04563658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:45:16.859391478 +0000 UTC m=+132.267493748" watchObservedRunningTime="2026-09-11 16:45:16.861462783 +0000 UTC m=+132.269565052" Sep 11 16:45:20.328520 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:20.328458 2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" podUID="24b90046-f333-4fa9-8feb-a739dc31f500" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 11 16:45:21.864480 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:21.864448 2577 generic.go:356] "Generic (PLEG): container finished" podID="d41dfd70-3a89-4d1d-84de-d0b3964b8538" containerID="d9e763c72f40553d21f196a3283f0c856828e88ae517377884c186a2975aee88" exitCode=0 Sep 11 16:45:21.864860 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:21.864528 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jkhgd" event={"ID":"d41dfd70-3a89-4d1d-84de-d0b3964b8538","Type":"ContainerDied","Data":"d9e763c72f40553d21f196a3283f0c856828e88ae517377884c186a2975aee88"} Sep 11 16:45:21.864860 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:21.864833 2577 scope.go:117] "RemoveContainer" containerID="d9e763c72f40553d21f196a3283f0c856828e88ae517377884c186a2975aee88" Sep 11 16:45:22.871009 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:22.870970 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jkhgd" event={"ID":"d41dfd70-3a89-4d1d-84de-d0b3964b8538","Type":"ContainerStarted","Data":"d6d9ca23fc88bf3ffe5a6790a5fa7e1919bd53fb222df7d75f77eb6ad436f006"} Sep 11 16:45:30.328600 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:30.328562 2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" podUID="24b90046-f333-4fa9-8feb-a739dc31f500" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 11 16:45:30.328958 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:30.328660 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" Sep 11 16:45:30.333369 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:30.329643 2577 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"9c03175a0123006c2b55afd16f8de7a07eb89130ff99dd59c7328042bbab5e4b"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" containerMessage="Container service-proxy failed liveness probe, will be restarted" Sep 11 16:45:30.333369 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:30.329722 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" podUID="24b90046-f333-4fa9-8feb-a739dc31f500" containerName="service-proxy" containerID="cri-o://9c03175a0123006c2b55afd16f8de7a07eb89130ff99dd59c7328042bbab5e4b" gracePeriod=30 Sep 11 16:45:30.913298 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:30.913256 2577 generic.go:356] "Generic (PLEG): container finished" podID="24b90046-f333-4fa9-8feb-a739dc31f500" containerID="9c03175a0123006c2b55afd16f8de7a07eb89130ff99dd59c7328042bbab5e4b" exitCode=2 Sep 11 16:45:30.913522 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:30.913336 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" event={"ID":"24b90046-f333-4fa9-8feb-a739dc31f500","Type":"ContainerDied","Data":"9c03175a0123006c2b55afd16f8de7a07eb89130ff99dd59c7328042bbab5e4b"} Sep 11 16:45:30.913522 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:45:30.913399 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54cf5ff94b-b5kpv" event={"ID":"24b90046-f333-4fa9-8feb-a739dc31f500","Type":"ContainerStarted","Data":"9265acafba6e820110f9fa44a32503c79dcb21b7b42df22ac2a79088a735cf00"} Sep 11 16:47:37.325133 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.325092 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms"] Sep 11 16:47:37.325762 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.325482 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="593fb846-0d3d-42dd-8405-6ac2f57bd18f" containerName="registry" Sep 11 16:47:37.325762 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.325516 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="593fb846-0d3d-42dd-8405-6ac2f57bd18f" containerName="registry" Sep 11 16:47:37.325762 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.325607 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="593fb846-0d3d-42dd-8405-6ac2f57bd18f" containerName="registry" Sep 11 16:47:37.328538 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.328517 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" Sep 11 16:47:37.345079 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.345061 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 16:47:37.345522 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.345492 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 16:47:37.345594 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.345559 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 16:47:37.345655 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.345637 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 16:47:37.345835 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.345820 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5x2wg\"" Sep 11 16:47:37.356129 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.356107 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms"] Sep 11 16:47:37.431853 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.431820 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5gl6\" (UniqueName: \"kubernetes.io/projected/c210ed9a-fa06-4b38-a9b1-537f9b44aa19-kube-api-access-c5gl6\") pod \"opendatahub-operator-controller-manager-7d67cb4f96-s9cms\" (UID: \"c210ed9a-fa06-4b38-a9b1-537f9b44aa19\") " pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" Sep 11 16:47:37.431853 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.431856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c210ed9a-fa06-4b38-a9b1-537f9b44aa19-manifests\") pod \"opendatahub-operator-controller-manager-7d67cb4f96-s9cms\" (UID: \"c210ed9a-fa06-4b38-a9b1-537f9b44aa19\") " pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" Sep 11 16:47:37.432084 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.431880 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c210ed9a-fa06-4b38-a9b1-537f9b44aa19-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d67cb4f96-s9cms\" (UID: \"c210ed9a-fa06-4b38-a9b1-537f9b44aa19\") " pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" Sep 11 16:47:37.432084 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.431954 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c210ed9a-fa06-4b38-a9b1-537f9b44aa19-tmp\") pod \"opendatahub-operator-controller-manager-7d67cb4f96-s9cms\" (UID: \"c210ed9a-fa06-4b38-a9b1-537f9b44aa19\") " pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" Sep 11 16:47:37.432084 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.432058 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c210ed9a-fa06-4b38-a9b1-537f9b44aa19-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d67cb4f96-s9cms\" (UID: \"c210ed9a-fa06-4b38-a9b1-537f9b44aa19\") " pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" Sep 11 16:47:37.532451 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.532410 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c210ed9a-fa06-4b38-a9b1-537f9b44aa19-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d67cb4f96-s9cms\" (UID: \"c210ed9a-fa06-4b38-a9b1-537f9b44aa19\") " pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" Sep 11 16:47:37.532668 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.532462 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5gl6\" (UniqueName: \"kubernetes.io/projected/c210ed9a-fa06-4b38-a9b1-537f9b44aa19-kube-api-access-c5gl6\") pod \"opendatahub-operator-controller-manager-7d67cb4f96-s9cms\" (UID: \"c210ed9a-fa06-4b38-a9b1-537f9b44aa19\") " pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" Sep 11 16:47:37.532668 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.532524 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c210ed9a-fa06-4b38-a9b1-537f9b44aa19-manifests\") pod \"opendatahub-operator-controller-manager-7d67cb4f96-s9cms\" (UID: \"c210ed9a-fa06-4b38-a9b1-537f9b44aa19\") " pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" Sep 11 16:47:37.532668 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.532553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c210ed9a-fa06-4b38-a9b1-537f9b44aa19-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d67cb4f96-s9cms\" (UID: \"c210ed9a-fa06-4b38-a9b1-537f9b44aa19\") " pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" Sep 11 16:47:37.532668 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.532582 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c210ed9a-fa06-4b38-a9b1-537f9b44aa19-tmp\") pod \"opendatahub-operator-controller-manager-7d67cb4f96-s9cms\" (UID: \"c210ed9a-fa06-4b38-a9b1-537f9b44aa19\") " pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" Sep 11 16:47:37.533002 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.532971 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c210ed9a-fa06-4b38-a9b1-537f9b44aa19-manifests\") pod \"opendatahub-operator-controller-manager-7d67cb4f96-s9cms\" (UID: \"c210ed9a-fa06-4b38-a9b1-537f9b44aa19\") " pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" Sep 11 16:47:37.534983 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.534957 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c210ed9a-fa06-4b38-a9b1-537f9b44aa19-tmp\") pod \"opendatahub-operator-controller-manager-7d67cb4f96-s9cms\" (UID: \"c210ed9a-fa06-4b38-a9b1-537f9b44aa19\") " pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" Sep 11 16:47:37.535095 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.535035 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c210ed9a-fa06-4b38-a9b1-537f9b44aa19-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d67cb4f96-s9cms\" (UID: \"c210ed9a-fa06-4b38-a9b1-537f9b44aa19\") " pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" Sep 11 16:47:37.535155 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.535133 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c210ed9a-fa06-4b38-a9b1-537f9b44aa19-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d67cb4f96-s9cms\" (UID: \"c210ed9a-fa06-4b38-a9b1-537f9b44aa19\") " pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" Sep 11 16:47:37.545323 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.545293 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5gl6\" (UniqueName: \"kubernetes.io/projected/c210ed9a-fa06-4b38-a9b1-537f9b44aa19-kube-api-access-c5gl6\") pod \"opendatahub-operator-controller-manager-7d67cb4f96-s9cms\" (UID: \"c210ed9a-fa06-4b38-a9b1-537f9b44aa19\") " pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" Sep 11 16:47:37.638206 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.638106 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" Sep 11 16:47:37.948373 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:47:37.948345 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc210ed9a_fa06_4b38_a9b1_537f9b44aa19.slice/crio-5f2af444b1e9275f1b5e5eee578d4248be2e2075af609d38553ef1a636f76fea WatchSource:0}: Error finding container 5f2af444b1e9275f1b5e5eee578d4248be2e2075af609d38553ef1a636f76fea: Status 404 returned error can't find the container with id 5f2af444b1e9275f1b5e5eee578d4248be2e2075af609d38553ef1a636f76fea Sep 11 16:47:37.949540 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:37.949509 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms"] Sep 11 16:47:38.393842 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:38.393792 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" event={"ID":"c210ed9a-fa06-4b38-a9b1-537f9b44aa19","Type":"ContainerStarted","Data":"5f2af444b1e9275f1b5e5eee578d4248be2e2075af609d38553ef1a636f76fea"} Sep 11 16:47:41.410024 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:41.409989 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" event={"ID":"c210ed9a-fa06-4b38-a9b1-537f9b44aa19","Type":"ContainerStarted","Data":"e1ebdb86d7c61f0991776b5ec61e77900d94882b0e1a71478c3c8398fb3c9743"} Sep 11 16:47:42.415935 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:42.415896 2577 generic.go:356] "Generic (PLEG): container finished" podID="c210ed9a-fa06-4b38-a9b1-537f9b44aa19" containerID="e1ebdb86d7c61f0991776b5ec61e77900d94882b0e1a71478c3c8398fb3c9743" exitCode=0 Sep 11 16:47:42.416376 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:42.415979 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" event={"ID":"c210ed9a-fa06-4b38-a9b1-537f9b44aa19","Type":"ContainerDied","Data":"e1ebdb86d7c61f0991776b5ec61e77900d94882b0e1a71478c3c8398fb3c9743"} Sep 11 16:47:43.422190 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:43.422155 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" event={"ID":"c210ed9a-fa06-4b38-a9b1-537f9b44aa19","Type":"ContainerStarted","Data":"f8b2f989316f896ab323f0c4749d4229ffa330c677eed19508c7e2c74d2fd91f"} Sep 11 16:47:43.422703 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:43.422291 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" Sep 11 16:47:43.452964 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:43.452904 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" podStartSLOduration=2.251682228 podStartE2EDuration="6.452886324s" totalImagesPullingTime="3.151041274s" totalInitContainerRuntime="1.050162822s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:47:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:47:43.450646021 +0000 UTC m=+278.858748310" watchObservedRunningTime="2026-09-11 16:47:43.452886324 +0000 UTC m=+278.860988593" Sep 11 16:47:54.429651 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:54.429614 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d67cb4f96-s9cms" Sep 11 16:47:56.689340 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:56.689307 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-75dc676c8d-z2798"] Sep 11 16:47:56.698985 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:56.698965 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-75dc676c8d-z2798" Sep 11 16:47:56.702573 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:56.702549 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 11 16:47:56.702688 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:56.702593 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6gs4n\"" Sep 11 16:47:56.702955 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:56.702939 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 11 16:47:56.710785 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:56.710762 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-75dc676c8d-z2798"] Sep 11 16:47:56.792208 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:56.792177 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcee08b9-9855-498b-88da-c1f8591a19f3-tmp\") pod \"kube-auth-proxy-75dc676c8d-z2798\" (UID: \"fcee08b9-9855-498b-88da-c1f8591a19f3\") " pod="openshift-ingress/kube-auth-proxy-75dc676c8d-z2798" Sep 11 16:47:56.792391 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:56.792220 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnwck\" (UniqueName: \"kubernetes.io/projected/fcee08b9-9855-498b-88da-c1f8591a19f3-kube-api-access-qnwck\") pod \"kube-auth-proxy-75dc676c8d-z2798\" (UID: \"fcee08b9-9855-498b-88da-c1f8591a19f3\") " pod="openshift-ingress/kube-auth-proxy-75dc676c8d-z2798" Sep 11 16:47:56.792391 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:56.792292 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcee08b9-9855-498b-88da-c1f8591a19f3-tls-certs\") pod \"kube-auth-proxy-75dc676c8d-z2798\" (UID: \"fcee08b9-9855-498b-88da-c1f8591a19f3\") " pod="openshift-ingress/kube-auth-proxy-75dc676c8d-z2798" Sep 11 16:47:56.893476 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:56.893444 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnwck\" (UniqueName: \"kubernetes.io/projected/fcee08b9-9855-498b-88da-c1f8591a19f3-kube-api-access-qnwck\") pod \"kube-auth-proxy-75dc676c8d-z2798\" (UID: \"fcee08b9-9855-498b-88da-c1f8591a19f3\") " pod="openshift-ingress/kube-auth-proxy-75dc676c8d-z2798" Sep 11 16:47:56.893619 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:56.893582 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcee08b9-9855-498b-88da-c1f8591a19f3-tls-certs\") pod \"kube-auth-proxy-75dc676c8d-z2798\" (UID: \"fcee08b9-9855-498b-88da-c1f8591a19f3\") " pod="openshift-ingress/kube-auth-proxy-75dc676c8d-z2798" Sep 11 16:47:56.893676 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:56.893619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcee08b9-9855-498b-88da-c1f8591a19f3-tmp\") pod \"kube-auth-proxy-75dc676c8d-z2798\" (UID: \"fcee08b9-9855-498b-88da-c1f8591a19f3\") " pod="openshift-ingress/kube-auth-proxy-75dc676c8d-z2798" Sep 11 16:47:56.893727 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:47:56.893711 2577 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 11 16:47:56.893792 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:47:56.893781 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcee08b9-9855-498b-88da-c1f8591a19f3-tls-certs podName:fcee08b9-9855-498b-88da-c1f8591a19f3 nodeName:}" failed. No retries permitted until 2026-09-11 16:47:57.393762273 +0000 UTC m=+292.801864523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fcee08b9-9855-498b-88da-c1f8591a19f3-tls-certs") pod "kube-auth-proxy-75dc676c8d-z2798" (UID: "fcee08b9-9855-498b-88da-c1f8591a19f3") : secret "kube-auth-proxy-tls" not found Sep 11 16:47:56.895939 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:56.895913 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcee08b9-9855-498b-88da-c1f8591a19f3-tmp\") pod \"kube-auth-proxy-75dc676c8d-z2798\" (UID: \"fcee08b9-9855-498b-88da-c1f8591a19f3\") " pod="openshift-ingress/kube-auth-proxy-75dc676c8d-z2798" Sep 11 16:47:56.908869 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:56.908844 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnwck\" (UniqueName: \"kubernetes.io/projected/fcee08b9-9855-498b-88da-c1f8591a19f3-kube-api-access-qnwck\") pod \"kube-auth-proxy-75dc676c8d-z2798\" (UID: \"fcee08b9-9855-498b-88da-c1f8591a19f3\") " pod="openshift-ingress/kube-auth-proxy-75dc676c8d-z2798" Sep 11 16:47:57.397561 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:57.397521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcee08b9-9855-498b-88da-c1f8591a19f3-tls-certs\") pod \"kube-auth-proxy-75dc676c8d-z2798\" (UID: \"fcee08b9-9855-498b-88da-c1f8591a19f3\") " pod="openshift-ingress/kube-auth-proxy-75dc676c8d-z2798" Sep 11 16:47:57.399901 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:57.399871 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcee08b9-9855-498b-88da-c1f8591a19f3-tls-certs\") pod \"kube-auth-proxy-75dc676c8d-z2798\" (UID: \"fcee08b9-9855-498b-88da-c1f8591a19f3\") " pod="openshift-ingress/kube-auth-proxy-75dc676c8d-z2798" Sep 11 16:47:57.608316 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:57.608283 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-75dc676c8d-z2798" Sep 11 16:47:57.887128 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:57.887102 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-75dc676c8d-z2798"] Sep 11 16:47:57.889123 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:47:57.889089 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcee08b9_9855_498b_88da_c1f8591a19f3.slice/crio-60f21a6d63da6398fb8d49d191139aac4c2f05c45a30099d9e9eb0e32632f9c5 WatchSource:0}: Error finding container 60f21a6d63da6398fb8d49d191139aac4c2f05c45a30099d9e9eb0e32632f9c5: Status 404 returned error can't find the container with id 60f21a6d63da6398fb8d49d191139aac4c2f05c45a30099d9e9eb0e32632f9c5 Sep 11 16:47:58.254083 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.254046 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4"] Sep 11 16:47:58.257871 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.257853 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.265790 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.265763 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 11 16:47:58.265913 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.265773 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 11 16:47:58.265913 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.265821 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-z6zqc\"" Sep 11 16:47:58.266088 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.266072 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 16:47:58.266144 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.266131 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 11 16:47:58.284081 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.284057 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4"] Sep 11 16:47:58.304062 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.304038 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.304180 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.304072 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxhfx\" (UniqueName: \"kubernetes.io/projected/00674c00-c4d9-441d-9c57-045701d9b7ff-kube-api-access-dxhfx\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.304180 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.304111 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.304180 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.304129 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.304180 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.304151 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00674c00-c4d9-441d-9c57-045701d9b7ff-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.304180 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.304175 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.304362 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.304252 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00674c00-c4d9-441d-9c57-045701d9b7ff-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.405333 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.405292 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.405556 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.405354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxhfx\" (UniqueName: \"kubernetes.io/projected/00674c00-c4d9-441d-9c57-045701d9b7ff-kube-api-access-dxhfx\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.405556 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.405405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.405556 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.405436 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.405556 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.405485 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00674c00-c4d9-441d-9c57-045701d9b7ff-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.405556 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.405527 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.405856 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.405559 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00674c00-c4d9-441d-9c57-045701d9b7ff-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.406029 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.406003 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.408679 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.408631 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.408679 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.408658 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00674c00-c4d9-441d-9c57-045701d9b7ff-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.408845 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.408742 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.409533 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.409485 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00674c00-c4d9-441d-9c57-045701d9b7ff-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.415337 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.415303 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxhfx\" (UniqueName: \"kubernetes.io/projected/00674c00-c4d9-441d-9c57-045701d9b7ff-kube-api-access-dxhfx\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.416767 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.416738 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-75sk4\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.492001 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.491951 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-75dc676c8d-z2798" event={"ID":"fcee08b9-9855-498b-88da-c1f8591a19f3","Type":"ContainerStarted","Data":"60f21a6d63da6398fb8d49d191139aac4c2f05c45a30099d9e9eb0e32632f9c5"} Sep 11 16:47:58.567762 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.567678 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:47:58.873223 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:58.873109 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4"] Sep 11 16:47:58.877455 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:47:58.877425 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00674c00_c4d9_441d_9c57_045701d9b7ff.slice/crio-2212d83e17c7309aa28bc1ad3880bd8397c788a0d062718f7bb810c7090fe699 WatchSource:0}: Error finding container 2212d83e17c7309aa28bc1ad3880bd8397c788a0d062718f7bb810c7090fe699: Status 404 returned error can't find the container with id 2212d83e17c7309aa28bc1ad3880bd8397c788a0d062718f7bb810c7090fe699 Sep 11 16:47:59.498280 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:47:59.498238 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" event={"ID":"00674c00-c4d9-441d-9c57-045701d9b7ff","Type":"ContainerStarted","Data":"2212d83e17c7309aa28bc1ad3880bd8397c788a0d062718f7bb810c7090fe699"} Sep 11 16:48:01.510376 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:01.510328 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-75dc676c8d-z2798" event={"ID":"fcee08b9-9855-498b-88da-c1f8591a19f3","Type":"ContainerStarted","Data":"cb871928dd1ee97e9d134dcdcc901966a907c45c35e58e608491ba9d2f29852a"} Sep 11 16:48:01.537194 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:01.537137 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-75dc676c8d-z2798" podStartSLOduration=2.969918921 podStartE2EDuration="5.537115061s" totalImagesPullingTime="2.56719614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:48:01.535712559 +0000 UTC m=+296.943814851" watchObservedRunningTime="2026-09-11 16:48:01.537115061 +0000 UTC m=+296.945217332" Sep 11 16:48:02.190048 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:02.190016 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:48:02.190108 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:02.190080 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:48:02.515903 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:02.515862 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" event={"ID":"00674c00-c4d9-441d-9c57-045701d9b7ff","Type":"ContainerStarted","Data":"5e075b46b778ecfd8e323907082efcf4d2f06628275b05ea15624394505379f8"} Sep 11 16:48:02.544175 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:02.544121 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" podStartSLOduration=1.234794153 podStartE2EDuration="4.544105203s" totalImagesPullingTime="3.30931105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:48:02.54199327 +0000 UTC m=+297.950095563" watchObservedRunningTime="2026-09-11 16:48:02.544105203 +0000 UTC m=+297.952207472" Sep 11 16:48:03.519804 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:03.519768 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:48:03.521706 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:03.521680 2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-75sk4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 16:48:03.521826 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:03.521733 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" podUID="00674c00-c4d9-441d-9c57-045701d9b7ff" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 16:48:04.524139 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:04.524112 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:48:04.976480 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:04.976401 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 16:48:04.977379 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:04.977352 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 16:48:04.988692 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:04.988670 2577 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 16:48:14.450840 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.450804 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td"] Sep 11 16:48:14.454219 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.454203 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.529420 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.529380 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td"] Sep 11 16:48:14.549682 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.549643 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjzdm\" (UniqueName: \"kubernetes.io/projected/c36344a8-30b6-4b52-b868-3a30bfa914e9-kube-api-access-vjzdm\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.549862 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.549689 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36344a8-30b6-4b52-b868-3a30bfa914e9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.549862 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.549742 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c36344a8-30b6-4b52-b868-3a30bfa914e9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.549862 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.549763 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c36344a8-30b6-4b52-b868-3a30bfa914e9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.550021 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.549892 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c36344a8-30b6-4b52-b868-3a30bfa914e9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.550021 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.549941 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c36344a8-30b6-4b52-b868-3a30bfa914e9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.550021 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.549973 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36344a8-30b6-4b52-b868-3a30bfa914e9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.650682 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.650647 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c36344a8-30b6-4b52-b868-3a30bfa914e9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.650682 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.650685 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36344a8-30b6-4b52-b868-3a30bfa914e9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.650889 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.650709 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjzdm\" (UniqueName: \"kubernetes.io/projected/c36344a8-30b6-4b52-b868-3a30bfa914e9-kube-api-access-vjzdm\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.650889 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.650826 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36344a8-30b6-4b52-b868-3a30bfa914e9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.650889 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.650872 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c36344a8-30b6-4b52-b868-3a30bfa914e9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.651033 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.650902 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c36344a8-30b6-4b52-b868-3a30bfa914e9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.651033 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.651002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c36344a8-30b6-4b52-b868-3a30bfa914e9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.651788 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.651757 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c36344a8-30b6-4b52-b868-3a30bfa914e9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.653249 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.653215 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36344a8-30b6-4b52-b868-3a30bfa914e9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.653249 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.653245 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c36344a8-30b6-4b52-b868-3a30bfa914e9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.653382 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.653255 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c36344a8-30b6-4b52-b868-3a30bfa914e9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.653382 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.653334 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c36344a8-30b6-4b52-b868-3a30bfa914e9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.662365 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.662342 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjzdm\" (UniqueName: \"kubernetes.io/projected/c36344a8-30b6-4b52-b868-3a30bfa914e9-kube-api-access-vjzdm\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.662593 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.662574 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36344a8-30b6-4b52-b868-3a30bfa914e9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vv6td\" (UID: \"c36344a8-30b6-4b52-b868-3a30bfa914e9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:14.763043 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:14.763010 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:15.077534 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:15.076679 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td"] Sep 11 16:48:15.081292 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:15.081256 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:48:15.081362 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:15.081328 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:48:15.571407 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:15.571372 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" event={"ID":"c36344a8-30b6-4b52-b868-3a30bfa914e9","Type":"ContainerStarted","Data":"15b79b9a3acb3df8eb69f9770dd254b63cbb14a8877182279526176c39edb591"} Sep 11 16:48:15.571407 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:15.571425 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" event={"ID":"c36344a8-30b6-4b52-b868-3a30bfa914e9","Type":"ContainerStarted","Data":"6abb0be69e81658fa1bc09fe34dce3df1fefbe12557261cddfd152123627fcae"} Sep 11 16:48:15.571967 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:15.571470 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:16.577250 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:16.577220 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" Sep 11 16:48:16.614679 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:16.614627 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vv6td" podStartSLOduration=2.614610295 podStartE2EDuration="2.614610295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:48:15.632886228 +0000 UTC m=+311.040988518" watchObservedRunningTime="2026-09-11 16:48:16.614610295 +0000 UTC m=+312.022712563" Sep 11 16:48:16.677837 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:16.677794 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4"] Sep 11 16:48:16.678021 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:16.677994 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" podUID="00674c00-c4d9-441d-9c57-045701d9b7ff" containerName="discovery" containerID="cri-o://5e075b46b778ecfd8e323907082efcf4d2f06628275b05ea15624394505379f8" gracePeriod=30 Sep 11 16:48:16.919546 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:16.919518 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:48:17.071955 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.071915 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00674c00-c4d9-441d-9c57-045701d9b7ff-local-certs\") pod \"00674c00-c4d9-441d-9c57-045701d9b7ff\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " Sep 11 16:48:17.071955 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.071958 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00674c00-c4d9-441d-9c57-045701d9b7ff-cacerts\") pod \"00674c00-c4d9-441d-9c57-045701d9b7ff\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " Sep 11 16:48:17.072220 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.072001 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-csr-ca-configmap\") pod \"00674c00-c4d9-441d-9c57-045701d9b7ff\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " Sep 11 16:48:17.072220 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.072028 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-kubeconfig\") pod \"00674c00-c4d9-441d-9c57-045701d9b7ff\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " Sep 11 16:48:17.072220 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.072057 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-token\") pod \"00674c00-c4d9-441d-9c57-045701d9b7ff\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " Sep 11 16:48:17.072220 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.072105 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxhfx\" (UniqueName: \"kubernetes.io/projected/00674c00-c4d9-441d-9c57-045701d9b7ff-kube-api-access-dxhfx\") pod \"00674c00-c4d9-441d-9c57-045701d9b7ff\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " Sep 11 16:48:17.072220 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.072137 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-csr-dns-cert\") pod \"00674c00-c4d9-441d-9c57-045701d9b7ff\" (UID: \"00674c00-c4d9-441d-9c57-045701d9b7ff\") " Sep 11 16:48:17.072546 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.072483 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "00674c00-c4d9-441d-9c57-045701d9b7ff" (UID: "00674c00-c4d9-441d-9c57-045701d9b7ff"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:48:17.074510 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.074449 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "00674c00-c4d9-441d-9c57-045701d9b7ff" (UID: "00674c00-c4d9-441d-9c57-045701d9b7ff"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 16:48:17.074617 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.074484 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00674c00-c4d9-441d-9c57-045701d9b7ff-local-certs" (OuterVolumeSpecName: "local-certs") pod "00674c00-c4d9-441d-9c57-045701d9b7ff" (UID: "00674c00-c4d9-441d-9c57-045701d9b7ff"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:48:17.074674 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.074648 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00674c00-c4d9-441d-9c57-045701d9b7ff-cacerts" (OuterVolumeSpecName: "cacerts") pod "00674c00-c4d9-441d-9c57-045701d9b7ff" (UID: "00674c00-c4d9-441d-9c57-045701d9b7ff"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 16:48:17.074734 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.074701 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "00674c00-c4d9-441d-9c57-045701d9b7ff" (UID: "00674c00-c4d9-441d-9c57-045701d9b7ff"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 16:48:17.074828 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.074809 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-token" (OuterVolumeSpecName: "istio-token") pod "00674c00-c4d9-441d-9c57-045701d9b7ff" (UID: "00674c00-c4d9-441d-9c57-045701d9b7ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:48:17.074884 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.074808 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00674c00-c4d9-441d-9c57-045701d9b7ff-kube-api-access-dxhfx" (OuterVolumeSpecName: "kube-api-access-dxhfx") pod "00674c00-c4d9-441d-9c57-045701d9b7ff" (UID: "00674c00-c4d9-441d-9c57-045701d9b7ff"). InnerVolumeSpecName "kube-api-access-dxhfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:48:17.173192 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.173156 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxhfx\" (UniqueName: \"kubernetes.io/projected/00674c00-c4d9-441d-9c57-045701d9b7ff-kube-api-access-dxhfx\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:48:17.173192 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.173196 2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-csr-dns-cert\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:48:17.173394 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.173205 2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00674c00-c4d9-441d-9c57-045701d9b7ff-local-certs\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:48:17.173394 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.173215 2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00674c00-c4d9-441d-9c57-045701d9b7ff-cacerts\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:48:17.173394 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.173223 2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-csr-ca-configmap\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:48:17.173394 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.173231 2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-kubeconfig\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:48:17.173394 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.173238 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00674c00-c4d9-441d-9c57-045701d9b7ff-istio-token\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:48:17.580777 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.580745 2577 generic.go:356] "Generic (PLEG): container finished" podID="00674c00-c4d9-441d-9c57-045701d9b7ff" containerID="5e075b46b778ecfd8e323907082efcf4d2f06628275b05ea15624394505379f8" exitCode=0 Sep 11 16:48:17.581182 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.580787 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" Sep 11 16:48:17.581182 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.580821 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" event={"ID":"00674c00-c4d9-441d-9c57-045701d9b7ff","Type":"ContainerDied","Data":"5e075b46b778ecfd8e323907082efcf4d2f06628275b05ea15624394505379f8"} Sep 11 16:48:17.581182 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.580866 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4" event={"ID":"00674c00-c4d9-441d-9c57-045701d9b7ff","Type":"ContainerDied","Data":"2212d83e17c7309aa28bc1ad3880bd8397c788a0d062718f7bb810c7090fe699"} Sep 11 16:48:17.581182 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.580885 2577 scope.go:117] "RemoveContainer" containerID="5e075b46b778ecfd8e323907082efcf4d2f06628275b05ea15624394505379f8" Sep 11 16:48:17.589109 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.589092 2577 scope.go:117] "RemoveContainer" containerID="5e075b46b778ecfd8e323907082efcf4d2f06628275b05ea15624394505379f8" Sep 11 16:48:17.589394 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:48:17.589371 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e075b46b778ecfd8e323907082efcf4d2f06628275b05ea15624394505379f8\": container with ID starting with 5e075b46b778ecfd8e323907082efcf4d2f06628275b05ea15624394505379f8 not found: ID does not exist" containerID="5e075b46b778ecfd8e323907082efcf4d2f06628275b05ea15624394505379f8" Sep 11 16:48:17.589458 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.589408 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e075b46b778ecfd8e323907082efcf4d2f06628275b05ea15624394505379f8"} err="failed to get container status \"5e075b46b778ecfd8e323907082efcf4d2f06628275b05ea15624394505379f8\": rpc error: code = NotFound desc = could not find container \"5e075b46b778ecfd8e323907082efcf4d2f06628275b05ea15624394505379f8\": container with ID starting with 5e075b46b778ecfd8e323907082efcf4d2f06628275b05ea15624394505379f8 not found: ID does not exist" Sep 11 16:48:17.622993 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.622958 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4"] Sep 11 16:48:17.631175 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:17.631151 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-75sk4"] Sep 11 16:48:18.715197 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:18.715155 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kjbdq"] Sep 11 16:48:18.715578 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:18.715472 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00674c00-c4d9-441d-9c57-045701d9b7ff" containerName="discovery" Sep 11 16:48:18.715578 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:18.715484 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="00674c00-c4d9-441d-9c57-045701d9b7ff" containerName="discovery" Sep 11 16:48:18.715578 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:18.715560 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="00674c00-c4d9-441d-9c57-045701d9b7ff" containerName="discovery" Sep 11 16:48:18.719353 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:18.719335 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kjbdq" Sep 11 16:48:18.723116 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:18.723095 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 11 16:48:18.723260 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:18.723185 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-9p2mh\"" Sep 11 16:48:18.772431 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:18.772393 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kjbdq"] Sep 11 16:48:18.891635 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:18.891594 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf9d2\" (UniqueName: \"kubernetes.io/projected/0ec9a480-9bb1-463f-83c9-d5518ab0dcb3-kube-api-access-kf9d2\") pod \"llmisvc-controller-manager-777c6bd7d5-kjbdq\" (UID: \"0ec9a480-9bb1-463f-83c9-d5518ab0dcb3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kjbdq" Sep 11 16:48:18.891819 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:18.891656 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ec9a480-9bb1-463f-83c9-d5518ab0dcb3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kjbdq\" (UID: \"0ec9a480-9bb1-463f-83c9-d5518ab0dcb3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kjbdq" Sep 11 16:48:18.992742 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:18.992708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ec9a480-9bb1-463f-83c9-d5518ab0dcb3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kjbdq\" (UID: \"0ec9a480-9bb1-463f-83c9-d5518ab0dcb3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kjbdq" Sep 11 16:48:18.992907 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:18.992786 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf9d2\" (UniqueName: \"kubernetes.io/projected/0ec9a480-9bb1-463f-83c9-d5518ab0dcb3-kube-api-access-kf9d2\") pod \"llmisvc-controller-manager-777c6bd7d5-kjbdq\" (UID: \"0ec9a480-9bb1-463f-83c9-d5518ab0dcb3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kjbdq" Sep 11 16:48:18.995169 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:18.995149 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ec9a480-9bb1-463f-83c9-d5518ab0dcb3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kjbdq\" (UID: \"0ec9a480-9bb1-463f-83c9-d5518ab0dcb3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kjbdq" Sep 11 16:48:19.051938 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:19.051902 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf9d2\" (UniqueName: \"kubernetes.io/projected/0ec9a480-9bb1-463f-83c9-d5518ab0dcb3-kube-api-access-kf9d2\") pod \"llmisvc-controller-manager-777c6bd7d5-kjbdq\" (UID: \"0ec9a480-9bb1-463f-83c9-d5518ab0dcb3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kjbdq" Sep 11 16:48:19.093026 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:19.092983 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00674c00-c4d9-441d-9c57-045701d9b7ff" path="/var/lib/kubelet/pods/00674c00-c4d9-441d-9c57-045701d9b7ff/volumes" Sep 11 16:48:19.328896 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:19.328801 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kjbdq" Sep 11 16:48:19.695047 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:48:19.695018 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0ec9a480_9bb1_463f_83c9_d5518ab0dcb3.slice/crio-f6efa0d7142a0c349a565cb6c599b71c1729bb3f46768deb5c89ae2e045fb83e WatchSource:0}: Error finding container f6efa0d7142a0c349a565cb6c599b71c1729bb3f46768deb5c89ae2e045fb83e: Status 404 returned error can't find the container with id f6efa0d7142a0c349a565cb6c599b71c1729bb3f46768deb5c89ae2e045fb83e Sep 11 16:48:19.695932 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:19.695908 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kjbdq"] Sep 11 16:48:19.696808 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:19.696793 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 16:48:20.603339 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:20.603302 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kjbdq" event={"ID":"0ec9a480-9bb1-463f-83c9-d5518ab0dcb3","Type":"ContainerStarted","Data":"f6efa0d7142a0c349a565cb6c599b71c1729bb3f46768deb5c89ae2e045fb83e"} Sep 11 16:48:23.617848 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:23.617811 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kjbdq" event={"ID":"0ec9a480-9bb1-463f-83c9-d5518ab0dcb3","Type":"ContainerStarted","Data":"95f61dc345a6532fec4fb8e71f3460c890fffbd603fdf58a11c14d22ffe588df"} Sep 11 16:48:23.618300 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:23.617888 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kjbdq" Sep 11 16:48:54.625014 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:54.624982 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kjbdq" Sep 11 16:48:54.646707 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:48:54.646655 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kjbdq" podStartSLOduration=33.841309712 podStartE2EDuration="36.646638698s" totalImagesPullingTime="2.805328986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:48:23.67034692 +0000 UTC m=+319.078449211" watchObservedRunningTime="2026-09-11 16:48:54.646638698 +0000 UTC m=+350.054740968" Sep 11 16:49:27.606691 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:27.604196 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-m4fx4"] Sep 11 16:49:27.608524 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:27.608491 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-m4fx4" Sep 11 16:49:27.612394 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:27.612369 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 16:49:27.612537 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:27.612404 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 16:49:27.613850 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:27.613827 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-njtfq\"" Sep 11 16:49:27.618283 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:27.618260 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-m4fx4"] Sep 11 16:49:27.668454 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:27.668423 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrxbv\" (UniqueName: \"kubernetes.io/projected/1a4356e3-3699-4b1e-9486-37d2fe270bda-kube-api-access-xrxbv\") pod \"authorino-operator-76b478bd78-m4fx4\" (UID: \"1a4356e3-3699-4b1e-9486-37d2fe270bda\") " pod="kuadrant-system/authorino-operator-76b478bd78-m4fx4" Sep 11 16:49:27.769169 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:27.769131 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrxbv\" (UniqueName: \"kubernetes.io/projected/1a4356e3-3699-4b1e-9486-37d2fe270bda-kube-api-access-xrxbv\") pod \"authorino-operator-76b478bd78-m4fx4\" (UID: \"1a4356e3-3699-4b1e-9486-37d2fe270bda\") " pod="kuadrant-system/authorino-operator-76b478bd78-m4fx4" Sep 11 16:49:27.789469 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:27.789441 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrxbv\" (UniqueName: \"kubernetes.io/projected/1a4356e3-3699-4b1e-9486-37d2fe270bda-kube-api-access-xrxbv\") pod \"authorino-operator-76b478bd78-m4fx4\" (UID: \"1a4356e3-3699-4b1e-9486-37d2fe270bda\") " pod="kuadrant-system/authorino-operator-76b478bd78-m4fx4" Sep 11 16:49:27.919813 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:27.919730 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-m4fx4" Sep 11 16:49:28.207549 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:28.207509 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-m4fx4"] Sep 11 16:49:28.210139 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:49:28.210115 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4356e3_3699_4b1e_9486_37d2fe270bda.slice/crio-8fb7d7e76186e7835a781171f174e2ada17e7b0446dbfa566117bd7315995251 WatchSource:0}: Error finding container 8fb7d7e76186e7835a781171f174e2ada17e7b0446dbfa566117bd7315995251: Status 404 returned error can't find the container with id 8fb7d7e76186e7835a781171f174e2ada17e7b0446dbfa566117bd7315995251 Sep 11 16:49:28.898836 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:28.898800 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-m4fx4" event={"ID":"1a4356e3-3699-4b1e-9486-37d2fe270bda","Type":"ContainerStarted","Data":"8fb7d7e76186e7835a781171f174e2ada17e7b0446dbfa566117bd7315995251"} Sep 11 16:49:31.916238 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:31.916202 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-m4fx4" event={"ID":"1a4356e3-3699-4b1e-9486-37d2fe270bda","Type":"ContainerStarted","Data":"9faaba8a873bf19b61f48008eb330efe60a43908d5c860c4d8741c7943de9444"} Sep 11 16:49:31.916655 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:31.916276 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-m4fx4" Sep 11 16:49:31.945144 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:31.945081 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-m4fx4" podStartSLOduration=2.173073468 podStartE2EDuration="4.945064568s" totalImagesPullingTime="2.7719911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:49:31.943199918 +0000 UTC m=+387.351302205" watchObservedRunningTime="2026-09-11 16:49:31.945064568 +0000 UTC m=+387.353166837" Sep 11 16:49:35.318085 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:35.318047 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn"] Sep 11 16:49:35.321328 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:35.321311 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" Sep 11 16:49:35.325253 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:35.325221 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ngvwn\"" Sep 11 16:49:35.340912 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:35.340883 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn"] Sep 11 16:49:35.432819 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:35.432778 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrzcn\" (UniqueName: \"kubernetes.io/projected/fbe88a88-042d-45fa-857c-760a40a8bb79-kube-api-access-zrzcn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v4gkn\" (UID: \"fbe88a88-042d-45fa-857c-760a40a8bb79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" Sep 11 16:49:35.432819 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:35.432824 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbe88a88-042d-45fa-857c-760a40a8bb79-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v4gkn\" (UID: \"fbe88a88-042d-45fa-857c-760a40a8bb79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" Sep 11 16:49:35.533613 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:35.533577 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrzcn\" (UniqueName: \"kubernetes.io/projected/fbe88a88-042d-45fa-857c-760a40a8bb79-kube-api-access-zrzcn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v4gkn\" (UID: \"fbe88a88-042d-45fa-857c-760a40a8bb79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" Sep 11 16:49:35.533613 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:35.533618 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbe88a88-042d-45fa-857c-760a40a8bb79-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v4gkn\" (UID: \"fbe88a88-042d-45fa-857c-760a40a8bb79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" Sep 11 16:49:35.533963 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:35.533946 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbe88a88-042d-45fa-857c-760a40a8bb79-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v4gkn\" (UID: \"fbe88a88-042d-45fa-857c-760a40a8bb79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" Sep 11 16:49:35.543918 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:35.543887 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrzcn\" (UniqueName: \"kubernetes.io/projected/fbe88a88-042d-45fa-857c-760a40a8bb79-kube-api-access-zrzcn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v4gkn\" (UID: \"fbe88a88-042d-45fa-857c-760a40a8bb79\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" Sep 11 16:49:35.631161 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:35.631071 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" Sep 11 16:49:35.908147 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:35.908122 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn"] Sep 11 16:49:35.910697 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:49:35.910670 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe88a88_042d_45fa_857c_760a40a8bb79.slice/crio-5fe3c5994874b2c1079b2bf174d2a6e5c4c3c176d97e38b600ec2f99214283bd WatchSource:0}: Error finding container 5fe3c5994874b2c1079b2bf174d2a6e5c4c3c176d97e38b600ec2f99214283bd: Status 404 returned error can't find the container with id 5fe3c5994874b2c1079b2bf174d2a6e5c4c3c176d97e38b600ec2f99214283bd Sep 11 16:49:35.933757 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:35.933728 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" event={"ID":"fbe88a88-042d-45fa-857c-760a40a8bb79","Type":"ContainerStarted","Data":"5fe3c5994874b2c1079b2bf174d2a6e5c4c3c176d97e38b600ec2f99214283bd"} Sep 11 16:49:39.955639 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:39.955601 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" event={"ID":"fbe88a88-042d-45fa-857c-760a40a8bb79","Type":"ContainerStarted","Data":"ced2bc2738149a5968af5500f3117a8f99cd0dd47897cb257e71ecd85302ee2d"} Sep 11 16:49:39.956007 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:39.955699 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" Sep 11 16:49:39.988792 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:39.988743 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" podStartSLOduration=1.425036071 podStartE2EDuration="4.988728415s" totalImagesPullingTime="3.563692344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:49:39.988331825 +0000 UTC m=+395.396434094" watchObservedRunningTime="2026-09-11 16:49:39.988728415 +0000 UTC m=+395.396830683" Sep 11 16:49:42.922453 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:42.922422 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-m4fx4" Sep 11 16:49:50.962567 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:50.962535 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" Sep 11 16:49:53.439298 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.439263 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn"] Sep 11 16:49:53.439718 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.439455 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" podUID="fbe88a88-042d-45fa-857c-760a40a8bb79" containerName="manager" containerID="cri-o://ced2bc2738149a5968af5500f3117a8f99cd0dd47897cb257e71ecd85302ee2d" gracePeriod=2 Sep 11 16:49:53.444931 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.444904 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn"] Sep 11 16:49:53.481716 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.481678 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6"] Sep 11 16:49:53.482129 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.482110 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbe88a88-042d-45fa-857c-760a40a8bb79" containerName="manager" Sep 11 16:49:53.482129 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.482130 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbe88a88-042d-45fa-857c-760a40a8bb79" containerName="manager" Sep 11 16:49:53.482269 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.482210 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbe88a88-042d-45fa-857c-760a40a8bb79" containerName="manager" Sep 11 16:49:53.493164 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.493132 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6" Sep 11 16:49:53.495905 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:49:53.495867 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-v4gkn\" is forbidden: User \"system:node:ip-10-0-143-165.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-165.ec2.internal' and this object" podUID="fbe88a88-042d-45fa-857c-760a40a8bb79" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" Sep 11 16:49:53.507020 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.506998 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6"] Sep 11 16:49:53.601982 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.601943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/747a6f94-727c-42c8-b724-620be4c3159e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nfjz6\" (UID: \"747a6f94-727c-42c8-b724-620be4c3159e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6" Sep 11 16:49:53.602147 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.602075 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jkz7\" (UniqueName: \"kubernetes.io/projected/747a6f94-727c-42c8-b724-620be4c3159e-kube-api-access-6jkz7\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nfjz6\" (UID: \"747a6f94-727c-42c8-b724-620be4c3159e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6" Sep 11 16:49:53.679938 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.679916 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" Sep 11 16:49:53.682751 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:49:53.682726 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-v4gkn\" is forbidden: User \"system:node:ip-10-0-143-165.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-165.ec2.internal' and this object" podUID="fbe88a88-042d-45fa-857c-760a40a8bb79" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" Sep 11 16:49:53.703212 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.703140 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/747a6f94-727c-42c8-b724-620be4c3159e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nfjz6\" (UID: \"747a6f94-727c-42c8-b724-620be4c3159e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6" Sep 11 16:49:53.703310 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.703233 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jkz7\" (UniqueName: \"kubernetes.io/projected/747a6f94-727c-42c8-b724-620be4c3159e-kube-api-access-6jkz7\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nfjz6\" (UID: \"747a6f94-727c-42c8-b724-620be4c3159e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6" Sep 11 16:49:53.703651 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.703631 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/747a6f94-727c-42c8-b724-620be4c3159e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nfjz6\" (UID: \"747a6f94-727c-42c8-b724-620be4c3159e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6" Sep 11 16:49:53.716188 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.716158 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jkz7\" (UniqueName: \"kubernetes.io/projected/747a6f94-727c-42c8-b724-620be4c3159e-kube-api-access-6jkz7\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nfjz6\" (UID: \"747a6f94-727c-42c8-b724-620be4c3159e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6" Sep 11 16:49:53.803881 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.803849 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrzcn\" (UniqueName: \"kubernetes.io/projected/fbe88a88-042d-45fa-857c-760a40a8bb79-kube-api-access-zrzcn\") pod \"fbe88a88-042d-45fa-857c-760a40a8bb79\" (UID: \"fbe88a88-042d-45fa-857c-760a40a8bb79\") " Sep 11 16:49:53.804031 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.803888 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbe88a88-042d-45fa-857c-760a40a8bb79-extensions-socket-volume\") pod \"fbe88a88-042d-45fa-857c-760a40a8bb79\" (UID: \"fbe88a88-042d-45fa-857c-760a40a8bb79\") " Sep 11 16:49:53.804459 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.804422 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe88a88-042d-45fa-857c-760a40a8bb79-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fbe88a88-042d-45fa-857c-760a40a8bb79" (UID: "fbe88a88-042d-45fa-857c-760a40a8bb79"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:49:53.806140 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.806112 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe88a88-042d-45fa-857c-760a40a8bb79-kube-api-access-zrzcn" (OuterVolumeSpecName: "kube-api-access-zrzcn") pod "fbe88a88-042d-45fa-857c-760a40a8bb79" (UID: "fbe88a88-042d-45fa-857c-760a40a8bb79"). InnerVolumeSpecName "kube-api-access-zrzcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:49:53.820281 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.820246 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6" Sep 11 16:49:53.905218 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.905189 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrzcn\" (UniqueName: \"kubernetes.io/projected/fbe88a88-042d-45fa-857c-760a40a8bb79-kube-api-access-zrzcn\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:49:53.905218 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:53.905214 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbe88a88-042d-45fa-857c-760a40a8bb79-extensions-socket-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:49:54.019318 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:54.019282 2577 generic.go:356] "Generic (PLEG): container finished" podID="fbe88a88-042d-45fa-857c-760a40a8bb79" containerID="ced2bc2738149a5968af5500f3117a8f99cd0dd47897cb257e71ecd85302ee2d" exitCode=0 Sep 11 16:49:54.019539 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:54.019343 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" Sep 11 16:49:54.019539 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:54.019382 2577 scope.go:117] "RemoveContainer" containerID="ced2bc2738149a5968af5500f3117a8f99cd0dd47897cb257e71ecd85302ee2d" Sep 11 16:49:54.022139 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:49:54.022112 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-v4gkn\" is forbidden: User \"system:node:ip-10-0-143-165.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-165.ec2.internal' and this object" podUID="fbe88a88-042d-45fa-857c-760a40a8bb79" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" Sep 11 16:49:54.028049 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:54.028033 2577 scope.go:117] "RemoveContainer" containerID="ced2bc2738149a5968af5500f3117a8f99cd0dd47897cb257e71ecd85302ee2d" Sep 11 16:49:54.028353 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:49:54.028329 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ced2bc2738149a5968af5500f3117a8f99cd0dd47897cb257e71ecd85302ee2d\": container with ID starting with ced2bc2738149a5968af5500f3117a8f99cd0dd47897cb257e71ecd85302ee2d not found: ID does not exist" containerID="ced2bc2738149a5968af5500f3117a8f99cd0dd47897cb257e71ecd85302ee2d" Sep 11 16:49:54.028449 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:54.028366 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced2bc2738149a5968af5500f3117a8f99cd0dd47897cb257e71ecd85302ee2d"} err="failed to get container status \"ced2bc2738149a5968af5500f3117a8f99cd0dd47897cb257e71ecd85302ee2d\": rpc error: code = NotFound desc = could not find container \"ced2bc2738149a5968af5500f3117a8f99cd0dd47897cb257e71ecd85302ee2d\": container with ID starting with ced2bc2738149a5968af5500f3117a8f99cd0dd47897cb257e71ecd85302ee2d not found: ID does not exist" Sep 11 16:49:54.031188 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:49:54.031164 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-v4gkn\" is forbidden: User \"system:node:ip-10-0-143-165.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-165.ec2.internal' and this object" podUID="fbe88a88-042d-45fa-857c-760a40a8bb79" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" Sep 11 16:49:54.113012 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:54.112991 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6"] Sep 11 16:49:54.114918 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:49:54.114894 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747a6f94_727c_42c8_b724_620be4c3159e.slice/crio-bc2b53c17c3c00e9b155fd2d4d024ccb2def47b4c7355082dafc81f873662991 WatchSource:0}: Error finding container bc2b53c17c3c00e9b155fd2d4d024ccb2def47b4c7355082dafc81f873662991: Status 404 returned error can't find the container with id bc2b53c17c3c00e9b155fd2d4d024ccb2def47b4c7355082dafc81f873662991 Sep 11 16:49:55.026388 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:55.026350 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6" event={"ID":"747a6f94-727c-42c8-b724-620be4c3159e","Type":"ContainerStarted","Data":"1b01a51989bf9d62458f8522d59095bdc72b0c25ca076c7f480230c7e96e1f76"} Sep 11 16:49:55.026932 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:55.026414 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6" event={"ID":"747a6f94-727c-42c8-b724-620be4c3159e","Type":"ContainerStarted","Data":"bc2b53c17c3c00e9b155fd2d4d024ccb2def47b4c7355082dafc81f873662991"} Sep 11 16:49:55.026932 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:55.026446 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6" Sep 11 16:49:55.029083 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:49:55.029053 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-v4gkn\" is forbidden: User \"system:node:ip-10-0-143-165.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-165.ec2.internal' and this object" podUID="fbe88a88-042d-45fa-857c-760a40a8bb79" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" Sep 11 16:49:55.067042 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:55.066982 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6" podStartSLOduration=2.066967648 podStartE2EDuration="2.066967648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:49:55.064274705 +0000 UTC m=+410.472376974" watchObservedRunningTime="2026-09-11 16:49:55.066967648 +0000 UTC m=+410.475069916" Sep 11 16:49:55.091803 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:49:55.091765 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-v4gkn\" is forbidden: User \"system:node:ip-10-0-143-165.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-165.ec2.internal' and this object" podUID="fbe88a88-042d-45fa-857c-760a40a8bb79" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v4gkn" Sep 11 16:49:55.093247 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:49:55.093221 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbe88a88-042d-45fa-857c-760a40a8bb79" path="/var/lib/kubelet/pods/fbe88a88-042d-45fa-857c-760a40a8bb79/volumes" Sep 11 16:50:06.035788 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:06.035756 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6" Sep 11 16:50:09.159847 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.159808 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6"] Sep 11 16:50:09.160267 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.159995 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6" podUID="747a6f94-727c-42c8-b724-620be4c3159e" containerName="manager" containerID="cri-o://1b01a51989bf9d62458f8522d59095bdc72b0c25ca076c7f480230c7e96e1f76" gracePeriod=10 Sep 11 16:50:09.402015 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.401985 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6" Sep 11 16:50:09.461489 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.461411 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8"] Sep 11 16:50:09.461762 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.461747 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="747a6f94-727c-42c8-b724-620be4c3159e" containerName="manager" Sep 11 16:50:09.461807 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.461763 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="747a6f94-727c-42c8-b724-620be4c3159e" containerName="manager" Sep 11 16:50:09.461845 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.461813 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="747a6f94-727c-42c8-b724-620be4c3159e" containerName="manager" Sep 11 16:50:09.463736 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.463720 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8" Sep 11 16:50:09.492828 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.492800 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8"] Sep 11 16:50:09.541631 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.541597 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/747a6f94-727c-42c8-b724-620be4c3159e-extensions-socket-volume\") pod \"747a6f94-727c-42c8-b724-620be4c3159e\" (UID: \"747a6f94-727c-42c8-b724-620be4c3159e\") " Sep 11 16:50:09.541794 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.541713 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jkz7\" (UniqueName: \"kubernetes.io/projected/747a6f94-727c-42c8-b724-620be4c3159e-kube-api-access-6jkz7\") pod \"747a6f94-727c-42c8-b724-620be4c3159e\" (UID: \"747a6f94-727c-42c8-b724-620be4c3159e\") " Sep 11 16:50:09.541998 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.541974 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747a6f94-727c-42c8-b724-620be4c3159e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "747a6f94-727c-42c8-b724-620be4c3159e" (UID: "747a6f94-727c-42c8-b724-620be4c3159e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:50:09.543758 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.543737 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747a6f94-727c-42c8-b724-620be4c3159e-kube-api-access-6jkz7" (OuterVolumeSpecName: "kube-api-access-6jkz7") pod "747a6f94-727c-42c8-b724-620be4c3159e" (UID: "747a6f94-727c-42c8-b724-620be4c3159e"). InnerVolumeSpecName "kube-api-access-6jkz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:50:09.643089 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.643047 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w8t8\" (UniqueName: \"kubernetes.io/projected/f499bf6d-64ff-4d7b-9872-eae31c9338ae-kube-api-access-8w8t8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7mdx8\" (UID: \"f499bf6d-64ff-4d7b-9872-eae31c9338ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8" Sep 11 16:50:09.643261 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.643100 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f499bf6d-64ff-4d7b-9872-eae31c9338ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7mdx8\" (UID: \"f499bf6d-64ff-4d7b-9872-eae31c9338ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8" Sep 11 16:50:09.643261 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.643159 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/747a6f94-727c-42c8-b724-620be4c3159e-extensions-socket-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:50:09.643261 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.643170 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jkz7\" (UniqueName: \"kubernetes.io/projected/747a6f94-727c-42c8-b724-620be4c3159e-kube-api-access-6jkz7\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:50:09.743992 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.743956 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w8t8\" (UniqueName: \"kubernetes.io/projected/f499bf6d-64ff-4d7b-9872-eae31c9338ae-kube-api-access-8w8t8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7mdx8\" (UID: \"f499bf6d-64ff-4d7b-9872-eae31c9338ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8" Sep 11 16:50:09.744163 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.744018 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f499bf6d-64ff-4d7b-9872-eae31c9338ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7mdx8\" (UID: \"f499bf6d-64ff-4d7b-9872-eae31c9338ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8" Sep 11 16:50:09.744446 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.744425 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f499bf6d-64ff-4d7b-9872-eae31c9338ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7mdx8\" (UID: \"f499bf6d-64ff-4d7b-9872-eae31c9338ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8" Sep 11 16:50:09.756437 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.756407 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w8t8\" (UniqueName: \"kubernetes.io/projected/f499bf6d-64ff-4d7b-9872-eae31c9338ae-kube-api-access-8w8t8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7mdx8\" (UID: \"f499bf6d-64ff-4d7b-9872-eae31c9338ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8" Sep 11 16:50:09.774337 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:09.774310 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8" Sep 11 16:50:10.060726 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:10.060704 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8"] Sep 11 16:50:10.062673 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:50:10.062648 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf499bf6d_64ff_4d7b_9872_eae31c9338ae.slice/crio-a598b9f4c207afca6b85a362d048dbd0abc006de2b191920bbc68287e7c2f87d WatchSource:0}: Error finding container a598b9f4c207afca6b85a362d048dbd0abc006de2b191920bbc68287e7c2f87d: Status 404 returned error can't find the container with id a598b9f4c207afca6b85a362d048dbd0abc006de2b191920bbc68287e7c2f87d Sep 11 16:50:10.097908 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:10.097879 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8" event={"ID":"f499bf6d-64ff-4d7b-9872-eae31c9338ae","Type":"ContainerStarted","Data":"a598b9f4c207afca6b85a362d048dbd0abc006de2b191920bbc68287e7c2f87d"} Sep 11 16:50:10.099180 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:10.099156 2577 generic.go:356] "Generic (PLEG): container finished" podID="747a6f94-727c-42c8-b724-620be4c3159e" containerID="1b01a51989bf9d62458f8522d59095bdc72b0c25ca076c7f480230c7e96e1f76" exitCode=0 Sep 11 16:50:10.099293 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:10.099205 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6" Sep 11 16:50:10.099293 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:10.099247 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6" event={"ID":"747a6f94-727c-42c8-b724-620be4c3159e","Type":"ContainerDied","Data":"1b01a51989bf9d62458f8522d59095bdc72b0c25ca076c7f480230c7e96e1f76"} Sep 11 16:50:10.099417 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:10.099300 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6" event={"ID":"747a6f94-727c-42c8-b724-620be4c3159e","Type":"ContainerDied","Data":"bc2b53c17c3c00e9b155fd2d4d024ccb2def47b4c7355082dafc81f873662991"} Sep 11 16:50:10.099417 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:10.099331 2577 scope.go:117] "RemoveContainer" containerID="1b01a51989bf9d62458f8522d59095bdc72b0c25ca076c7f480230c7e96e1f76" Sep 11 16:50:10.107741 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:10.107721 2577 scope.go:117] "RemoveContainer" containerID="1b01a51989bf9d62458f8522d59095bdc72b0c25ca076c7f480230c7e96e1f76" Sep 11 16:50:10.108044 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:50:10.108012 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b01a51989bf9d62458f8522d59095bdc72b0c25ca076c7f480230c7e96e1f76\": container with ID starting with 1b01a51989bf9d62458f8522d59095bdc72b0c25ca076c7f480230c7e96e1f76 not found: ID does not exist" containerID="1b01a51989bf9d62458f8522d59095bdc72b0c25ca076c7f480230c7e96e1f76" Sep 11 16:50:10.108130 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:10.108041 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b01a51989bf9d62458f8522d59095bdc72b0c25ca076c7f480230c7e96e1f76"} err="failed to get container status \"1b01a51989bf9d62458f8522d59095bdc72b0c25ca076c7f480230c7e96e1f76\": rpc error: code = NotFound desc = could not find container \"1b01a51989bf9d62458f8522d59095bdc72b0c25ca076c7f480230c7e96e1f76\": container with ID starting with 1b01a51989bf9d62458f8522d59095bdc72b0c25ca076c7f480230c7e96e1f76 not found: ID does not exist" Sep 11 16:50:10.138804 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:10.138775 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6"] Sep 11 16:50:10.164327 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:10.164295 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nfjz6"] Sep 11 16:50:11.094322 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:11.094290 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="747a6f94-727c-42c8-b724-620be4c3159e" path="/var/lib/kubelet/pods/747a6f94-727c-42c8-b724-620be4c3159e/volumes" Sep 11 16:50:11.106160 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:11.106133 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8" event={"ID":"f499bf6d-64ff-4d7b-9872-eae31c9338ae","Type":"ContainerStarted","Data":"3f0d9501101405af525f8d56317218db9630f65b572108a0cb0befb2be15fbad"} Sep 11 16:50:11.106322 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:11.106264 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8" Sep 11 16:50:22.113310 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:22.113279 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8" Sep 11 16:50:22.139159 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:50:22.139108 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8" podStartSLOduration=13.139093362 podStartE2EDuration="13.139093362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:50:11.141270017 +0000 UTC m=+426.549372348" watchObservedRunningTime="2026-09-11 16:50:22.139093362 +0000 UTC m=+437.547195635" Sep 11 16:51:16.504605 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:16.504567 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-659646d656-zs7p7"] Sep 11 16:51:16.509157 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:16.509138 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-659646d656-zs7p7" Sep 11 16:51:16.512128 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:16.512103 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 11 16:51:16.512285 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:16.512265 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 11 16:51:16.512446 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:16.512430 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-67lb8\"" Sep 11 16:51:16.512525 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:16.512491 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 11 16:51:16.521153 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:16.521126 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-659646d656-zs7p7"] Sep 11 16:51:16.686516 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:16.686465 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cfecd83-4092-4486-b5f0-7c6f48983e45-metrics-certs\") pod \"maas-controller-659646d656-zs7p7\" (UID: \"5cfecd83-4092-4486-b5f0-7c6f48983e45\") " pod="opendatahub/maas-controller-659646d656-zs7p7" Sep 11 16:51:16.686686 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:16.686522 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cfecd83-4092-4486-b5f0-7c6f48983e45-webhook-cert\") pod \"maas-controller-659646d656-zs7p7\" (UID: \"5cfecd83-4092-4486-b5f0-7c6f48983e45\") " pod="opendatahub/maas-controller-659646d656-zs7p7" Sep 11 16:51:16.686686 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:16.686650 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85r7n\" (UniqueName: \"kubernetes.io/projected/5cfecd83-4092-4486-b5f0-7c6f48983e45-kube-api-access-85r7n\") pod \"maas-controller-659646d656-zs7p7\" (UID: \"5cfecd83-4092-4486-b5f0-7c6f48983e45\") " pod="opendatahub/maas-controller-659646d656-zs7p7" Sep 11 16:51:16.787743 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:16.787641 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85r7n\" (UniqueName: \"kubernetes.io/projected/5cfecd83-4092-4486-b5f0-7c6f48983e45-kube-api-access-85r7n\") pod \"maas-controller-659646d656-zs7p7\" (UID: \"5cfecd83-4092-4486-b5f0-7c6f48983e45\") " pod="opendatahub/maas-controller-659646d656-zs7p7" Sep 11 16:51:16.787743 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:16.787703 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cfecd83-4092-4486-b5f0-7c6f48983e45-metrics-certs\") pod \"maas-controller-659646d656-zs7p7\" (UID: \"5cfecd83-4092-4486-b5f0-7c6f48983e45\") " pod="opendatahub/maas-controller-659646d656-zs7p7" Sep 11 16:51:16.787949 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:16.787752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cfecd83-4092-4486-b5f0-7c6f48983e45-webhook-cert\") pod \"maas-controller-659646d656-zs7p7\" (UID: \"5cfecd83-4092-4486-b5f0-7c6f48983e45\") " pod="opendatahub/maas-controller-659646d656-zs7p7" Sep 11 16:51:16.790248 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:16.790217 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cfecd83-4092-4486-b5f0-7c6f48983e45-webhook-cert\") pod \"maas-controller-659646d656-zs7p7\" (UID: \"5cfecd83-4092-4486-b5f0-7c6f48983e45\") " pod="opendatahub/maas-controller-659646d656-zs7p7" Sep 11 16:51:16.790378 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:16.790275 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cfecd83-4092-4486-b5f0-7c6f48983e45-metrics-certs\") pod \"maas-controller-659646d656-zs7p7\" (UID: \"5cfecd83-4092-4486-b5f0-7c6f48983e45\") " pod="opendatahub/maas-controller-659646d656-zs7p7" Sep 11 16:51:16.798348 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:16.798325 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85r7n\" (UniqueName: \"kubernetes.io/projected/5cfecd83-4092-4486-b5f0-7c6f48983e45-kube-api-access-85r7n\") pod \"maas-controller-659646d656-zs7p7\" (UID: \"5cfecd83-4092-4486-b5f0-7c6f48983e45\") " pod="opendatahub/maas-controller-659646d656-zs7p7" Sep 11 16:51:16.820318 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:16.820295 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-659646d656-zs7p7" Sep 11 16:51:17.149679 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:17.149655 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-659646d656-zs7p7"] Sep 11 16:51:17.152391 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:51:17.152361 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cfecd83_4092_4486_b5f0_7c6f48983e45.slice/crio-2d1a351c7f900a8ef1a1c6ddccf3ede82387c7778926040a543047313e9bfecb WatchSource:0}: Error finding container 2d1a351c7f900a8ef1a1c6ddccf3ede82387c7778926040a543047313e9bfecb: Status 404 returned error can't find the container with id 2d1a351c7f900a8ef1a1c6ddccf3ede82387c7778926040a543047313e9bfecb Sep 11 16:51:17.397419 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:17.397386 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-659646d656-zs7p7" event={"ID":"5cfecd83-4092-4486-b5f0-7c6f48983e45","Type":"ContainerStarted","Data":"2d1a351c7f900a8ef1a1c6ddccf3ede82387c7778926040a543047313e9bfecb"} Sep 11 16:51:20.414644 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:20.414608 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-659646d656-zs7p7" event={"ID":"5cfecd83-4092-4486-b5f0-7c6f48983e45","Type":"ContainerStarted","Data":"6b224d33a849ba8edb5b6c3bf280fb5735aa37ca5c065e803ad2ae73af38e566"} Sep 11 16:51:20.415143 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:20.414734 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-659646d656-zs7p7" Sep 11 16:51:20.441249 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:20.441199 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-659646d656-zs7p7" podStartSLOduration=2.066145744 podStartE2EDuration="4.441183532s" totalImagesPullingTime="2.375037788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:51:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:51:20.439345701 +0000 UTC m=+495.847447990" watchObservedRunningTime="2026-09-11 16:51:20.441183532 +0000 UTC m=+495.849285801" Sep 11 16:51:31.422664 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:31.422631 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-659646d656-zs7p7" Sep 11 16:51:31.464518 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:31.464468 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-659646d656-zs7p7"] Sep 11 16:51:31.468066 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:31.468031 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-659646d656-zs7p7" podUID="5cfecd83-4092-4486-b5f0-7c6f48983e45" containerName="manager" containerID="cri-o://6b224d33a849ba8edb5b6c3bf280fb5735aa37ca5c065e803ad2ae73af38e566" gracePeriod=30 Sep 11 16:51:31.712666 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:31.712640 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-659646d656-zs7p7" Sep 11 16:51:31.813670 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:31.813633 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85r7n\" (UniqueName: \"kubernetes.io/projected/5cfecd83-4092-4486-b5f0-7c6f48983e45-kube-api-access-85r7n\") pod \"5cfecd83-4092-4486-b5f0-7c6f48983e45\" (UID: \"5cfecd83-4092-4486-b5f0-7c6f48983e45\") " Sep 11 16:51:31.813836 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:31.813713 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cfecd83-4092-4486-b5f0-7c6f48983e45-metrics-certs\") pod \"5cfecd83-4092-4486-b5f0-7c6f48983e45\" (UID: \"5cfecd83-4092-4486-b5f0-7c6f48983e45\") " Sep 11 16:51:31.813836 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:31.813760 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cfecd83-4092-4486-b5f0-7c6f48983e45-webhook-cert\") pod \"5cfecd83-4092-4486-b5f0-7c6f48983e45\" (UID: \"5cfecd83-4092-4486-b5f0-7c6f48983e45\") " Sep 11 16:51:31.815951 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:31.815916 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cfecd83-4092-4486-b5f0-7c6f48983e45-kube-api-access-85r7n" (OuterVolumeSpecName: "kube-api-access-85r7n") pod "5cfecd83-4092-4486-b5f0-7c6f48983e45" (UID: "5cfecd83-4092-4486-b5f0-7c6f48983e45"). InnerVolumeSpecName "kube-api-access-85r7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:51:31.816089 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:31.816011 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cfecd83-4092-4486-b5f0-7c6f48983e45-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5cfecd83-4092-4486-b5f0-7c6f48983e45" (UID: "5cfecd83-4092-4486-b5f0-7c6f48983e45"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 16:51:31.816089 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:31.816022 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cfecd83-4092-4486-b5f0-7c6f48983e45-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5cfecd83-4092-4486-b5f0-7c6f48983e45" (UID: "5cfecd83-4092-4486-b5f0-7c6f48983e45"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 16:51:31.914783 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:31.914745 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85r7n\" (UniqueName: \"kubernetes.io/projected/5cfecd83-4092-4486-b5f0-7c6f48983e45-kube-api-access-85r7n\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:51:31.914783 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:31.914776 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cfecd83-4092-4486-b5f0-7c6f48983e45-metrics-certs\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:51:31.914783 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:31.914785 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cfecd83-4092-4486-b5f0-7c6f48983e45-webhook-cert\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:51:32.473740 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:32.473708 2577 generic.go:356] "Generic (PLEG): container finished" podID="5cfecd83-4092-4486-b5f0-7c6f48983e45" containerID="6b224d33a849ba8edb5b6c3bf280fb5735aa37ca5c065e803ad2ae73af38e566" exitCode=0 Sep 11 16:51:32.474220 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:32.473751 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-659646d656-zs7p7" event={"ID":"5cfecd83-4092-4486-b5f0-7c6f48983e45","Type":"ContainerDied","Data":"6b224d33a849ba8edb5b6c3bf280fb5735aa37ca5c065e803ad2ae73af38e566"} Sep 11 16:51:32.474220 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:32.473760 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-659646d656-zs7p7" Sep 11 16:51:32.474220 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:32.473801 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-659646d656-zs7p7" event={"ID":"5cfecd83-4092-4486-b5f0-7c6f48983e45","Type":"ContainerDied","Data":"2d1a351c7f900a8ef1a1c6ddccf3ede82387c7778926040a543047313e9bfecb"} Sep 11 16:51:32.474220 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:32.473825 2577 scope.go:117] "RemoveContainer" containerID="6b224d33a849ba8edb5b6c3bf280fb5735aa37ca5c065e803ad2ae73af38e566" Sep 11 16:51:32.482397 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:32.482380 2577 scope.go:117] "RemoveContainer" containerID="6b224d33a849ba8edb5b6c3bf280fb5735aa37ca5c065e803ad2ae73af38e566" Sep 11 16:51:32.482685 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:51:32.482664 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b224d33a849ba8edb5b6c3bf280fb5735aa37ca5c065e803ad2ae73af38e566\": container with ID starting with 6b224d33a849ba8edb5b6c3bf280fb5735aa37ca5c065e803ad2ae73af38e566 not found: ID does not exist" containerID="6b224d33a849ba8edb5b6c3bf280fb5735aa37ca5c065e803ad2ae73af38e566" Sep 11 16:51:32.482734 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:32.482696 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b224d33a849ba8edb5b6c3bf280fb5735aa37ca5c065e803ad2ae73af38e566"} err="failed to get container status \"6b224d33a849ba8edb5b6c3bf280fb5735aa37ca5c065e803ad2ae73af38e566\": rpc error: code = NotFound desc = could not find container \"6b224d33a849ba8edb5b6c3bf280fb5735aa37ca5c065e803ad2ae73af38e566\": container with ID starting with 6b224d33a849ba8edb5b6c3bf280fb5735aa37ca5c065e803ad2ae73af38e566 not found: ID does not exist" Sep 11 16:51:32.497271 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:32.497243 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-659646d656-zs7p7"] Sep 11 16:51:32.502044 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:32.502020 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-659646d656-zs7p7"] Sep 11 16:51:33.093259 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:33.093228 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cfecd83-4092-4486-b5f0-7c6f48983e45" path="/var/lib/kubelet/pods/5cfecd83-4092-4486-b5f0-7c6f48983e45/volumes" Sep 11 16:51:55.565726 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:55.565687 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-5zjwh"] Sep 11 16:51:55.566202 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:55.566039 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cfecd83-4092-4486-b5f0-7c6f48983e45" containerName="manager" Sep 11 16:51:55.566202 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:55.566049 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cfecd83-4092-4486-b5f0-7c6f48983e45" containerName="manager" Sep 11 16:51:55.566202 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:55.566127 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cfecd83-4092-4486-b5f0-7c6f48983e45" containerName="manager" Sep 11 16:51:55.574207 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:55.574184 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5zjwh" Sep 11 16:51:55.578304 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:55.578282 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nsckj\"" Sep 11 16:51:55.578304 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:55.578278 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 11 16:51:55.586579 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:55.586557 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-5zjwh"] Sep 11 16:51:55.705484 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:55.705447 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c091003-cbc1-4289-959f-0779b52e49f5-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-5zjwh\" (UID: \"4c091003-cbc1-4289-959f-0779b52e49f5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5zjwh" Sep 11 16:51:55.705672 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:55.705489 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzms2\" (UniqueName: \"kubernetes.io/projected/4c091003-cbc1-4289-959f-0779b52e49f5-kube-api-access-qzms2\") pod \"payload-pre-processing-59fbc47bf8-5zjwh\" (UID: \"4c091003-cbc1-4289-959f-0779b52e49f5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5zjwh" Sep 11 16:51:55.806818 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:55.806778 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c091003-cbc1-4289-959f-0779b52e49f5-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-5zjwh\" (UID: \"4c091003-cbc1-4289-959f-0779b52e49f5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5zjwh" Sep 11 16:51:55.806818 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:55.806823 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzms2\" (UniqueName: \"kubernetes.io/projected/4c091003-cbc1-4289-959f-0779b52e49f5-kube-api-access-qzms2\") pod \"payload-pre-processing-59fbc47bf8-5zjwh\" (UID: \"4c091003-cbc1-4289-959f-0779b52e49f5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5zjwh" Sep 11 16:51:55.807520 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:55.807473 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c091003-cbc1-4289-959f-0779b52e49f5-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-5zjwh\" (UID: \"4c091003-cbc1-4289-959f-0779b52e49f5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5zjwh" Sep 11 16:51:55.815928 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:55.815864 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzms2\" (UniqueName: \"kubernetes.io/projected/4c091003-cbc1-4289-959f-0779b52e49f5-kube-api-access-qzms2\") pod \"payload-pre-processing-59fbc47bf8-5zjwh\" (UID: \"4c091003-cbc1-4289-959f-0779b52e49f5\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5zjwh" Sep 11 16:51:55.884583 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:55.884543 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5zjwh" Sep 11 16:51:56.159344 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:56.159273 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-5zjwh"] Sep 11 16:51:56.161790 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:51:56.161751 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c091003_cbc1_4289_959f_0779b52e49f5.slice/crio-d54cb8da3f09a6172cb0598d5a93d31e5fdf7f53f640c03ee6ee3eed15c03a5f WatchSource:0}: Error finding container d54cb8da3f09a6172cb0598d5a93d31e5fdf7f53f640c03ee6ee3eed15c03a5f: Status 404 returned error can't find the container with id d54cb8da3f09a6172cb0598d5a93d31e5fdf7f53f640c03ee6ee3eed15c03a5f Sep 11 16:51:56.587077 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:56.587038 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5zjwh" event={"ID":"4c091003-cbc1-4289-959f-0779b52e49f5","Type":"ContainerStarted","Data":"d54cb8da3f09a6172cb0598d5a93d31e5fdf7f53f640c03ee6ee3eed15c03a5f"} Sep 11 16:51:58.597391 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:58.597352 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5zjwh" event={"ID":"4c091003-cbc1-4289-959f-0779b52e49f5","Type":"ContainerStarted","Data":"879b39f543e49a5e4bec503fa93c00e43b12ddbe290d4a9fcf047e656d2491bf"} Sep 11 16:51:58.597789 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:58.597484 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5zjwh" Sep 11 16:51:58.624678 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:51:58.624627 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5zjwh" podStartSLOduration=1.3408004199999999 podStartE2EDuration="3.624611237s" totalImagesPullingTime="2.283810817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:51:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:51:58.623895041 +0000 UTC m=+534.031997338" watchObservedRunningTime="2026-09-11 16:51:58.624611237 +0000 UTC m=+534.032713505" Sep 11 16:52:09.603617 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:09.603586 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-5zjwh" Sep 11 16:52:10.857973 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:10.857933 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr"] Sep 11 16:52:10.862604 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:10.862587 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:10.867992 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:10.867970 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 11 16:52:10.869365 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:10.869332 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vkjjm\"" Sep 11 16:52:10.869519 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:10.869485 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 16:52:10.869573 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:10.869526 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 16:52:10.881701 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:10.881672 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr"] Sep 11 16:52:10.940129 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:10.940096 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373b1503-48c1-4514-8742-91212afde6d3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr\" (UID: \"373b1503-48c1-4514-8742-91212afde6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:10.940129 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:10.940134 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373b1503-48c1-4514-8742-91212afde6d3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr\" (UID: \"373b1503-48c1-4514-8742-91212afde6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:10.940386 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:10.940206 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373b1503-48c1-4514-8742-91212afde6d3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr\" (UID: \"373b1503-48c1-4514-8742-91212afde6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:10.940386 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:10.940230 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373b1503-48c1-4514-8742-91212afde6d3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr\" (UID: \"373b1503-48c1-4514-8742-91212afde6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:10.940386 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:10.940254 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqs9s\" (UniqueName: \"kubernetes.io/projected/373b1503-48c1-4514-8742-91212afde6d3-kube-api-access-pqs9s\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr\" (UID: \"373b1503-48c1-4514-8742-91212afde6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:10.940386 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:10.940329 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373b1503-48c1-4514-8742-91212afde6d3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr\" (UID: \"373b1503-48c1-4514-8742-91212afde6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:11.041216 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:11.041179 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373b1503-48c1-4514-8742-91212afde6d3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr\" (UID: \"373b1503-48c1-4514-8742-91212afde6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:11.041216 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:11.041218 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373b1503-48c1-4514-8742-91212afde6d3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr\" (UID: \"373b1503-48c1-4514-8742-91212afde6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:11.041456 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:11.041248 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373b1503-48c1-4514-8742-91212afde6d3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr\" (UID: \"373b1503-48c1-4514-8742-91212afde6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:11.041456 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:11.041265 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373b1503-48c1-4514-8742-91212afde6d3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr\" (UID: \"373b1503-48c1-4514-8742-91212afde6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:11.041456 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:11.041284 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqs9s\" (UniqueName: \"kubernetes.io/projected/373b1503-48c1-4514-8742-91212afde6d3-kube-api-access-pqs9s\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr\" (UID: \"373b1503-48c1-4514-8742-91212afde6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:11.041456 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:11.041332 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373b1503-48c1-4514-8742-91212afde6d3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr\" (UID: \"373b1503-48c1-4514-8742-91212afde6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:11.041767 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:11.041738 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373b1503-48c1-4514-8742-91212afde6d3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr\" (UID: \"373b1503-48c1-4514-8742-91212afde6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:11.041877 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:11.041777 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373b1503-48c1-4514-8742-91212afde6d3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr\" (UID: \"373b1503-48c1-4514-8742-91212afde6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:11.041877 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:11.041796 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373b1503-48c1-4514-8742-91212afde6d3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr\" (UID: \"373b1503-48c1-4514-8742-91212afde6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:11.043452 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:11.043422 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373b1503-48c1-4514-8742-91212afde6d3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr\" (UID: \"373b1503-48c1-4514-8742-91212afde6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:11.043784 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:11.043764 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373b1503-48c1-4514-8742-91212afde6d3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr\" (UID: \"373b1503-48c1-4514-8742-91212afde6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:11.056990 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:11.056970 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqs9s\" (UniqueName: \"kubernetes.io/projected/373b1503-48c1-4514-8742-91212afde6d3-kube-api-access-pqs9s\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr\" (UID: \"373b1503-48c1-4514-8742-91212afde6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:11.172364 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:11.172273 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:11.453611 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:11.453578 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr"] Sep 11 16:52:11.456086 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:52:11.456053 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373b1503_48c1_4514_8742_91212afde6d3.slice/crio-ded65d9b549581f36826e88f08c3bf22e85965e14741619f296085b6382d954d WatchSource:0}: Error finding container ded65d9b549581f36826e88f08c3bf22e85965e14741619f296085b6382d954d: Status 404 returned error can't find the container with id ded65d9b549581f36826e88f08c3bf22e85965e14741619f296085b6382d954d Sep 11 16:52:11.656907 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:11.656876 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" event={"ID":"373b1503-48c1-4514-8742-91212afde6d3","Type":"ContainerStarted","Data":"ded65d9b549581f36826e88f08c3bf22e85965e14741619f296085b6382d954d"} Sep 11 16:52:12.663767 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:12.663729 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" event={"ID":"373b1503-48c1-4514-8742-91212afde6d3","Type":"ContainerStarted","Data":"3755881402d12c77bbcc959b6febf9a67500c21cff3624bcb9f029783c10e56d"} Sep 11 16:52:12.685484 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:12.685432 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" podStartSLOduration=1.672660095 podStartE2EDuration="2.685414428s" totalImagesPullingTime="1.012754333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:52:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:52:12.683423822 +0000 UTC m=+548.091526110" watchObservedRunningTime="2026-09-11 16:52:12.685414428 +0000 UTC m=+548.093516696" Sep 11 16:52:21.172772 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:21.172722 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:21.173151 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:21.172809 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:21.185160 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:21.185133 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:21.716249 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:21.716222 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr" Sep 11 16:52:51.633610 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.633573 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8"] Sep 11 16:52:51.641951 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.641922 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:51.645346 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.645316 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 11 16:52:51.654703 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.654678 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8"] Sep 11 16:52:51.792619 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.792581 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b2c580-995b-42f1-9d43-097a77f12226-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8\" (UID: \"94b2c580-995b-42f1-9d43-097a77f12226\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:51.792619 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.792623 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b2c580-995b-42f1-9d43-097a77f12226-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8\" (UID: \"94b2c580-995b-42f1-9d43-097a77f12226\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:51.792865 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.792679 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr477\" (UniqueName: \"kubernetes.io/projected/94b2c580-995b-42f1-9d43-097a77f12226-kube-api-access-jr477\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8\" (UID: \"94b2c580-995b-42f1-9d43-097a77f12226\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:51.792865 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.792728 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b2c580-995b-42f1-9d43-097a77f12226-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8\" (UID: \"94b2c580-995b-42f1-9d43-097a77f12226\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:51.792865 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.792773 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b2c580-995b-42f1-9d43-097a77f12226-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8\" (UID: \"94b2c580-995b-42f1-9d43-097a77f12226\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:51.792865 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.792808 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b2c580-995b-42f1-9d43-097a77f12226-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8\" (UID: \"94b2c580-995b-42f1-9d43-097a77f12226\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:51.893791 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.893708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr477\" (UniqueName: \"kubernetes.io/projected/94b2c580-995b-42f1-9d43-097a77f12226-kube-api-access-jr477\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8\" (UID: \"94b2c580-995b-42f1-9d43-097a77f12226\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:51.893791 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.893754 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b2c580-995b-42f1-9d43-097a77f12226-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8\" (UID: \"94b2c580-995b-42f1-9d43-097a77f12226\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:51.893791 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.893789 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b2c580-995b-42f1-9d43-097a77f12226-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8\" (UID: \"94b2c580-995b-42f1-9d43-097a77f12226\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:51.894064 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.893819 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b2c580-995b-42f1-9d43-097a77f12226-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8\" (UID: \"94b2c580-995b-42f1-9d43-097a77f12226\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:51.894064 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.893846 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b2c580-995b-42f1-9d43-097a77f12226-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8\" (UID: \"94b2c580-995b-42f1-9d43-097a77f12226\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:51.894064 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.893883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b2c580-995b-42f1-9d43-097a77f12226-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8\" (UID: \"94b2c580-995b-42f1-9d43-097a77f12226\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:51.894251 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.894230 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b2c580-995b-42f1-9d43-097a77f12226-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8\" (UID: \"94b2c580-995b-42f1-9d43-097a77f12226\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:51.894323 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.894259 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b2c580-995b-42f1-9d43-097a77f12226-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8\" (UID: \"94b2c580-995b-42f1-9d43-097a77f12226\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:51.894323 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.894302 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b2c580-995b-42f1-9d43-097a77f12226-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8\" (UID: \"94b2c580-995b-42f1-9d43-097a77f12226\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:51.896107 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.896078 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b2c580-995b-42f1-9d43-097a77f12226-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8\" (UID: \"94b2c580-995b-42f1-9d43-097a77f12226\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:51.896363 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.896343 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b2c580-995b-42f1-9d43-097a77f12226-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8\" (UID: \"94b2c580-995b-42f1-9d43-097a77f12226\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:51.909680 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.909658 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr477\" (UniqueName: \"kubernetes.io/projected/94b2c580-995b-42f1-9d43-097a77f12226-kube-api-access-jr477\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8\" (UID: \"94b2c580-995b-42f1-9d43-097a77f12226\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:51.953363 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:51.953323 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:52:52.229315 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:52.229290 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8"] Sep 11 16:52:52.232189 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:52:52.232132 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b2c580_995b_42f1_9d43_097a77f12226.slice/crio-71e8da6f8cd46a631f9125a8ba8ff3b4b661e06178eaf5a3c13d3e0668dca067 WatchSource:0}: Error finding container 71e8da6f8cd46a631f9125a8ba8ff3b4b661e06178eaf5a3c13d3e0668dca067: Status 404 returned error can't find the container with id 71e8da6f8cd46a631f9125a8ba8ff3b4b661e06178eaf5a3c13d3e0668dca067 Sep 11 16:52:52.849982 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:52.849896 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" event={"ID":"94b2c580-995b-42f1-9d43-097a77f12226","Type":"ContainerStarted","Data":"3ebfa9eb97d54042e862bffb6ad933323fcdb8a0c2bccc8fc786eb025587ebfd"} Sep 11 16:52:52.849982 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:52.849950 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" event={"ID":"94b2c580-995b-42f1-9d43-097a77f12226","Type":"ContainerStarted","Data":"71e8da6f8cd46a631f9125a8ba8ff3b4b661e06178eaf5a3c13d3e0668dca067"} Sep 11 16:52:52.878347 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:52:52.878288 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" podStartSLOduration=1.585446299 podStartE2EDuration="1.878273735s" totalImagesPullingTime="292.827436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:52:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:52:52.877329399 +0000 UTC m=+588.285431697" watchObservedRunningTime="2026-09-11 16:52:52.878273735 +0000 UTC m=+588.286376004" Sep 11 16:53:01.954214 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:01.954174 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:53:01.954714 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:01.954238 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:53:01.967220 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:01.967189 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:53:02.909426 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:02.909399 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8" Sep 11 16:53:05.007464 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:05.007434 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 16:53:05.007931 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:05.007686 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 16:53:08.197690 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.197653 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt"] Sep 11 16:53:08.201032 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.201013 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.204834 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.204809 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 11 16:53:08.221625 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.221598 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt"] Sep 11 16:53:08.345074 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.345039 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b4884b3-83d6-45ae-85c8-eed414110527-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt\" (UID: \"9b4884b3-83d6-45ae-85c8-eed414110527\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.345074 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.345085 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b4884b3-83d6-45ae-85c8-eed414110527-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt\" (UID: \"9b4884b3-83d6-45ae-85c8-eed414110527\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.345338 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.345106 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4884b3-83d6-45ae-85c8-eed414110527-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt\" (UID: \"9b4884b3-83d6-45ae-85c8-eed414110527\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.345338 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.345230 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fglmt\" (UniqueName: \"kubernetes.io/projected/9b4884b3-83d6-45ae-85c8-eed414110527-kube-api-access-fglmt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt\" (UID: \"9b4884b3-83d6-45ae-85c8-eed414110527\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.345338 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.345268 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b4884b3-83d6-45ae-85c8-eed414110527-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt\" (UID: \"9b4884b3-83d6-45ae-85c8-eed414110527\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.345448 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.345341 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b4884b3-83d6-45ae-85c8-eed414110527-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt\" (UID: \"9b4884b3-83d6-45ae-85c8-eed414110527\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.446566 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.446526 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b4884b3-83d6-45ae-85c8-eed414110527-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt\" (UID: \"9b4884b3-83d6-45ae-85c8-eed414110527\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.446783 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.446599 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b4884b3-83d6-45ae-85c8-eed414110527-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt\" (UID: \"9b4884b3-83d6-45ae-85c8-eed414110527\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.446783 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.446633 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b4884b3-83d6-45ae-85c8-eed414110527-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt\" (UID: \"9b4884b3-83d6-45ae-85c8-eed414110527\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.446783 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.446652 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4884b3-83d6-45ae-85c8-eed414110527-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt\" (UID: \"9b4884b3-83d6-45ae-85c8-eed414110527\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.446783 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.446690 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fglmt\" (UniqueName: \"kubernetes.io/projected/9b4884b3-83d6-45ae-85c8-eed414110527-kube-api-access-fglmt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt\" (UID: \"9b4884b3-83d6-45ae-85c8-eed414110527\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.446783 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.446707 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b4884b3-83d6-45ae-85c8-eed414110527-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt\" (UID: \"9b4884b3-83d6-45ae-85c8-eed414110527\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.447050 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.447030 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b4884b3-83d6-45ae-85c8-eed414110527-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt\" (UID: \"9b4884b3-83d6-45ae-85c8-eed414110527\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.447099 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.447053 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b4884b3-83d6-45ae-85c8-eed414110527-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt\" (UID: \"9b4884b3-83d6-45ae-85c8-eed414110527\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.447099 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.447063 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b4884b3-83d6-45ae-85c8-eed414110527-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt\" (UID: \"9b4884b3-83d6-45ae-85c8-eed414110527\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.449142 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.449081 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b4884b3-83d6-45ae-85c8-eed414110527-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt\" (UID: \"9b4884b3-83d6-45ae-85c8-eed414110527\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.449283 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.449259 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4884b3-83d6-45ae-85c8-eed414110527-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt\" (UID: \"9b4884b3-83d6-45ae-85c8-eed414110527\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.456092 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.456065 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fglmt\" (UniqueName: \"kubernetes.io/projected/9b4884b3-83d6-45ae-85c8-eed414110527-kube-api-access-fglmt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt\" (UID: \"9b4884b3-83d6-45ae-85c8-eed414110527\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.510814 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.510772 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:08.789890 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.789863 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt"] Sep 11 16:53:08.792013 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:53:08.791984 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4884b3_83d6_45ae_85c8_eed414110527.slice/crio-49353a4ecaf417a0df56227c43291c196d3fff54af5cc839913e2fac99314d7e WatchSource:0}: Error finding container 49353a4ecaf417a0df56227c43291c196d3fff54af5cc839913e2fac99314d7e: Status 404 returned error can't find the container with id 49353a4ecaf417a0df56227c43291c196d3fff54af5cc839913e2fac99314d7e Sep 11 16:53:08.932609 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:08.932571 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" event={"ID":"9b4884b3-83d6-45ae-85c8-eed414110527","Type":"ContainerStarted","Data":"49353a4ecaf417a0df56227c43291c196d3fff54af5cc839913e2fac99314d7e"} Sep 11 16:53:09.939189 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:09.939151 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" event={"ID":"9b4884b3-83d6-45ae-85c8-eed414110527","Type":"ContainerStarted","Data":"1d9ef7c9696f924b055e59052cfd5bcbc98fa6a400292b3d245632ceb808f44a"} Sep 11 16:53:09.963113 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:09.963057 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" podStartSLOduration=1.720780592 podStartE2EDuration="1.963039267s" totalImagesPullingTime="242.258675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:53:09.961967175 +0000 UTC m=+605.370069463" watchObservedRunningTime="2026-09-11 16:53:09.963039267 +0000 UTC m=+605.371141572" Sep 11 16:53:18.511986 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:18.511939 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:18.514455 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:18.512007 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:18.526541 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:18.526491 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:53:18.997129 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:53:18.997097 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt" Sep 11 16:55:37.407048 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.407010 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc"] Sep 11 16:55:37.410479 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.410460 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc" Sep 11 16:55:37.413796 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.413774 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-2dc95c-dockercfg-cw9cf\"" Sep 11 16:55:37.413923 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.413829 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-2dc95c\"" Sep 11 16:55:37.420894 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.420865 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc"] Sep 11 16:55:37.529354 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.529321 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a899ee3-cc24-4b0c-865f-972feea82a70-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc\" (UID: \"2a899ee3-cc24-4b0c-865f-972feea82a70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc" Sep 11 16:55:37.529539 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.529383 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svtrv\" (UniqueName: \"kubernetes.io/projected/2a899ee3-cc24-4b0c-865f-972feea82a70-kube-api-access-svtrv\") pod \"payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc\" (UID: \"2a899ee3-cc24-4b0c-865f-972feea82a70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc" Sep 11 16:55:37.576644 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.576612 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz"] Sep 11 16:55:37.580002 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.579979 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz" Sep 11 16:55:37.601006 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.600976 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz"] Sep 11 16:55:37.630143 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.630100 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a899ee3-cc24-4b0c-865f-972feea82a70-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc\" (UID: \"2a899ee3-cc24-4b0c-865f-972feea82a70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc" Sep 11 16:55:37.630330 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.630182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svtrv\" (UniqueName: \"kubernetes.io/projected/2a899ee3-cc24-4b0c-865f-972feea82a70-kube-api-access-svtrv\") pod \"payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc\" (UID: \"2a899ee3-cc24-4b0c-865f-972feea82a70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc" Sep 11 16:55:37.630847 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.630826 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a899ee3-cc24-4b0c-865f-972feea82a70-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc\" (UID: \"2a899ee3-cc24-4b0c-865f-972feea82a70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc" Sep 11 16:55:37.638691 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.638665 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svtrv\" (UniqueName: \"kubernetes.io/projected/2a899ee3-cc24-4b0c-865f-972feea82a70-kube-api-access-svtrv\") pod \"payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc\" (UID: \"2a899ee3-cc24-4b0c-865f-972feea82a70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc" Sep 11 16:55:37.721959 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.721882 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc" Sep 11 16:55:37.731375 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.731337 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ae08bd5-de0f-44e9-ac94-8f6761648502-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz\" (UID: \"9ae08bd5-de0f-44e9-ac94-8f6761648502\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz" Sep 11 16:55:37.731532 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.731480 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kqpn\" (UniqueName: \"kubernetes.io/projected/9ae08bd5-de0f-44e9-ac94-8f6761648502-kube-api-access-8kqpn\") pod \"payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz\" (UID: \"9ae08bd5-de0f-44e9-ac94-8f6761648502\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz" Sep 11 16:55:37.832741 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.832699 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kqpn\" (UniqueName: \"kubernetes.io/projected/9ae08bd5-de0f-44e9-ac94-8f6761648502-kube-api-access-8kqpn\") pod \"payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz\" (UID: \"9ae08bd5-de0f-44e9-ac94-8f6761648502\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz" Sep 11 16:55:37.832929 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.832777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ae08bd5-de0f-44e9-ac94-8f6761648502-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz\" (UID: \"9ae08bd5-de0f-44e9-ac94-8f6761648502\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz" Sep 11 16:55:37.833727 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.833695 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ae08bd5-de0f-44e9-ac94-8f6761648502-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz\" (UID: \"9ae08bd5-de0f-44e9-ac94-8f6761648502\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz" Sep 11 16:55:37.840528 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.840487 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kqpn\" (UniqueName: \"kubernetes.io/projected/9ae08bd5-de0f-44e9-ac94-8f6761648502-kube-api-access-8kqpn\") pod \"payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz\" (UID: \"9ae08bd5-de0f-44e9-ac94-8f6761648502\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz" Sep 11 16:55:37.890253 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.890222 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz" Sep 11 16:55:37.999772 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:37.999738 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc"] Sep 11 16:55:38.002357 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:55:38.002330 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a899ee3_cc24_4b0c_865f_972feea82a70.slice/crio-95a8c81aca8e0e395fd03e7a70aa2628bb346d9680ad433d8f75e97cc2efca13 WatchSource:0}: Error finding container 95a8c81aca8e0e395fd03e7a70aa2628bb346d9680ad433d8f75e97cc2efca13: Status 404 returned error can't find the container with id 95a8c81aca8e0e395fd03e7a70aa2628bb346d9680ad433d8f75e97cc2efca13 Sep 11 16:55:38.169119 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.169078 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz"] Sep 11 16:55:38.172181 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:55:38.172154 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae08bd5_de0f_44e9_ac94_8f6761648502.slice/crio-d5f3a78a63b4366db0d815309d51470e01925860265495277b4e19481ddb162a WatchSource:0}: Error finding container d5f3a78a63b4366db0d815309d51470e01925860265495277b4e19481ddb162a: Status 404 returned error can't find the container with id d5f3a78a63b4366db0d815309d51470e01925860265495277b4e19481ddb162a Sep 11 16:55:38.601977 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.601939 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468"] Sep 11 16:55:38.605441 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.605415 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468" Sep 11 16:55:38.608353 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.608332 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4268e3-dockercfg-65qsx\"" Sep 11 16:55:38.608518 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.608434 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4268e3\"" Sep 11 16:55:38.615557 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.615353 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468"] Sep 11 16:55:38.617889 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.617861 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4"] Sep 11 16:55:38.621856 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.621830 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4" Sep 11 16:55:38.651397 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.651363 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc" event={"ID":"2a899ee3-cc24-4b0c-865f-972feea82a70","Type":"ContainerStarted","Data":"4ac14724347c11e448aafc7a4afd6820c8921a6dfb31f82296491418208d4002"} Sep 11 16:55:38.651691 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.651674 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4"] Sep 11 16:55:38.651810 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.651797 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc" event={"ID":"2a899ee3-cc24-4b0c-865f-972feea82a70","Type":"ContainerStarted","Data":"95a8c81aca8e0e395fd03e7a70aa2628bb346d9680ad433d8f75e97cc2efca13"} Sep 11 16:55:38.652915 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.652845 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc" Sep 11 16:55:38.655148 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.655122 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz" event={"ID":"9ae08bd5-de0f-44e9-ac94-8f6761648502","Type":"ContainerStarted","Data":"cb4974c7885f618e7232d21c3d8f958f406bf49b40485368e210a26f293b03c5"} Sep 11 16:55:38.655262 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.655167 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz" event={"ID":"9ae08bd5-de0f-44e9-ac94-8f6761648502","Type":"ContainerStarted","Data":"d5f3a78a63b4366db0d815309d51470e01925860265495277b4e19481ddb162a"} Sep 11 16:55:38.655319 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.655298 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz" Sep 11 16:55:38.685328 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.685271 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc" podStartSLOduration=1.685254773 podStartE2EDuration="1.685254773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:55:38.684542923 +0000 UTC m=+754.092645215" watchObservedRunningTime="2026-09-11 16:55:38.685254773 +0000 UTC m=+754.093357070" Sep 11 16:55:38.710049 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.709996 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz" podStartSLOduration=1.7099782270000001 podStartE2EDuration="1.709978227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:55:38.708085706 +0000 UTC m=+754.116188005" watchObservedRunningTime="2026-09-11 16:55:38.709978227 +0000 UTC m=+754.118080495" Sep 11 16:55:38.742288 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.742235 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468\" (UID: \"df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468" Sep 11 16:55:38.742488 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.742298 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znk2f\" (UniqueName: \"kubernetes.io/projected/df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d-kube-api-access-znk2f\") pod \"payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468\" (UID: \"df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468" Sep 11 16:55:38.742488 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.742441 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5lnd\" (UniqueName: \"kubernetes.io/projected/1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa-kube-api-access-b5lnd\") pod \"payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4\" (UID: \"1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4" Sep 11 16:55:38.742488 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.742477 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4\" (UID: \"1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4" Sep 11 16:55:38.843536 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.843478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468\" (UID: \"df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468" Sep 11 16:55:38.843735 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.843570 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znk2f\" (UniqueName: \"kubernetes.io/projected/df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d-kube-api-access-znk2f\") pod \"payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468\" (UID: \"df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468" Sep 11 16:55:38.843926 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.843900 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lnd\" (UniqueName: \"kubernetes.io/projected/1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa-kube-api-access-b5lnd\") pod \"payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4\" (UID: \"1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4" Sep 11 16:55:38.844063 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.844039 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4\" (UID: \"1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4" Sep 11 16:55:38.844434 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.844412 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468\" (UID: \"df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468" Sep 11 16:55:38.844738 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.844721 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4\" (UID: \"1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4" Sep 11 16:55:38.857298 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.857232 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5lnd\" (UniqueName: \"kubernetes.io/projected/1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa-kube-api-access-b5lnd\") pod \"payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4\" (UID: \"1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4" Sep 11 16:55:38.861126 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.861103 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znk2f\" (UniqueName: \"kubernetes.io/projected/df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d-kube-api-access-znk2f\") pod \"payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468\" (UID: \"df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468" Sep 11 16:55:38.919099 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.919046 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468" Sep 11 16:55:38.956951 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:38.956922 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4" Sep 11 16:55:39.212770 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:39.212742 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468"] Sep 11 16:55:39.214998 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:55:39.214966 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3ca3c5_0f50_48b6_8e7b_a4c1ba9f001d.slice/crio-8c375b8505650947abf79735fd4c13b0ca54efa099e4d43b212d8249558178a2 WatchSource:0}: Error finding container 8c375b8505650947abf79735fd4c13b0ca54efa099e4d43b212d8249558178a2: Status 404 returned error can't find the container with id 8c375b8505650947abf79735fd4c13b0ca54efa099e4d43b212d8249558178a2 Sep 11 16:55:39.261370 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:39.261338 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4"] Sep 11 16:55:39.264660 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:55:39.264635 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5cfb02_37a1_4351_bf1c_d5b9a1e452aa.slice/crio-9167d8119e3560e6219c52818696557414fb36d1306f3c0ac0e6c7c12b030453 WatchSource:0}: Error finding container 9167d8119e3560e6219c52818696557414fb36d1306f3c0ac0e6c7c12b030453: Status 404 returned error can't find the container with id 9167d8119e3560e6219c52818696557414fb36d1306f3c0ac0e6c7c12b030453 Sep 11 16:55:39.661558 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:39.661522 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4" event={"ID":"1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa","Type":"ContainerStarted","Data":"ac06863ec3e26cfbe4d874deb4a537a022618489388bbe876417834a8dfd8bf5"} Sep 11 16:55:39.662017 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:39.661781 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4" event={"ID":"1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa","Type":"ContainerStarted","Data":"9167d8119e3560e6219c52818696557414fb36d1306f3c0ac0e6c7c12b030453"} Sep 11 16:55:39.663328 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:39.663299 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468" event={"ID":"df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d","Type":"ContainerStarted","Data":"ef6ed3b984b86c498fc60c42dccf2de27f7d3a5502e1f82779746e0a514d72e5"} Sep 11 16:55:39.663526 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:39.663509 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468" Sep 11 16:55:39.663609 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:39.663559 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468" event={"ID":"df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d","Type":"ContainerStarted","Data":"8c375b8505650947abf79735fd4c13b0ca54efa099e4d43b212d8249558178a2"} Sep 11 16:55:39.682938 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:39.682884 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4" podStartSLOduration=1.682862616 podStartE2EDuration="1.682862616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:55:39.681327559 +0000 UTC m=+755.089429871" watchObservedRunningTime="2026-09-11 16:55:39.682862616 +0000 UTC m=+755.090964885" Sep 11 16:55:39.700058 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:39.700008 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468" podStartSLOduration=1.6999945840000001 podStartE2EDuration="1.699994584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:55:39.699596521 +0000 UTC m=+755.107698818" watchObservedRunningTime="2026-09-11 16:55:39.699994584 +0000 UTC m=+755.108096853" Sep 11 16:55:39.755510 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:39.755469 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc"] Sep 11 16:55:39.758798 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:39.758782 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc" Sep 11 16:55:39.762547 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:39.762525 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-52b0af\"" Sep 11 16:55:39.762672 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:39.762585 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-52b0af-dockercfg-2mwck\"" Sep 11 16:55:39.772487 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:39.772456 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc"] Sep 11 16:55:39.955258 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:39.955160 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e122d49-2235-456c-a734-757161c1d060-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc\" (UID: \"9e122d49-2235-456c-a734-757161c1d060\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc" Sep 11 16:55:39.955258 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:39.955246 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch9sj\" (UniqueName: \"kubernetes.io/projected/9e122d49-2235-456c-a734-757161c1d060-kube-api-access-ch9sj\") pod \"payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc\" (UID: \"9e122d49-2235-456c-a734-757161c1d060\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc" Sep 11 16:55:40.056275 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:40.056239 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch9sj\" (UniqueName: \"kubernetes.io/projected/9e122d49-2235-456c-a734-757161c1d060-kube-api-access-ch9sj\") pod \"payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc\" (UID: \"9e122d49-2235-456c-a734-757161c1d060\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc" Sep 11 16:55:40.056441 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:40.056342 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e122d49-2235-456c-a734-757161c1d060-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc\" (UID: \"9e122d49-2235-456c-a734-757161c1d060\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc" Sep 11 16:55:40.057104 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:40.057083 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e122d49-2235-456c-a734-757161c1d060-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc\" (UID: \"9e122d49-2235-456c-a734-757161c1d060\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc" Sep 11 16:55:40.065377 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:40.065346 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch9sj\" (UniqueName: \"kubernetes.io/projected/9e122d49-2235-456c-a734-757161c1d060-kube-api-access-ch9sj\") pod \"payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc\" (UID: \"9e122d49-2235-456c-a734-757161c1d060\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc" Sep 11 16:55:40.070053 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:40.070031 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc" Sep 11 16:55:40.363206 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:40.363180 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc"] Sep 11 16:55:40.366296 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:55:40.366256 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e122d49_2235_456c_a734_757161c1d060.slice/crio-581b61fa1df2f880ea530be6f71e6020d319bb0e5d9464e417c71ce0b80504a7 WatchSource:0}: Error finding container 581b61fa1df2f880ea530be6f71e6020d319bb0e5d9464e417c71ce0b80504a7: Status 404 returned error can't find the container with id 581b61fa1df2f880ea530be6f71e6020d319bb0e5d9464e417c71ce0b80504a7 Sep 11 16:55:40.670685 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:40.670590 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc" event={"ID":"9e122d49-2235-456c-a734-757161c1d060","Type":"ContainerStarted","Data":"66ad23eb1ebf4d9531786979ee2d2d92c6024f645e12924275b4ab5250aff43e"} Sep 11 16:55:40.670685 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:40.670667 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc" Sep 11 16:55:40.671180 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:40.670693 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc" event={"ID":"9e122d49-2235-456c-a734-757161c1d060","Type":"ContainerStarted","Data":"581b61fa1df2f880ea530be6f71e6020d319bb0e5d9464e417c71ce0b80504a7"} Sep 11 16:55:40.671180 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:40.671059 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4" Sep 11 16:55:40.704439 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:40.704392 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc" podStartSLOduration=1.704372823 podStartE2EDuration="1.704372823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:55:40.702593448 +0000 UTC m=+756.110695737" watchObservedRunningTime="2026-09-11 16:55:40.704372823 +0000 UTC m=+756.112475521" Sep 11 16:55:48.910576 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:48.910542 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg"] Sep 11 16:55:48.914198 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:48.914182 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg" Sep 11 16:55:48.917069 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:48.917044 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-8b1a16-dockercfg-9mwfn\"" Sep 11 16:55:48.917069 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:48.917064 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-8b1a16\"" Sep 11 16:55:48.928325 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:48.928299 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg"] Sep 11 16:55:48.943331 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:48.943306 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jshzh\" (UniqueName: \"kubernetes.io/projected/cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c-kube-api-access-jshzh\") pod \"payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg\" (UID: \"cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c\") " pod="openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg" Sep 11 16:55:48.943449 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:48.943337 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c-plugins-config-volume\") pod \"payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg\" (UID: \"cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c\") " pod="openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg" Sep 11 16:55:48.964790 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:48.964766 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km"] Sep 11 16:55:48.968148 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:48.968132 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km" Sep 11 16:55:48.979657 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:48.979631 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km"] Sep 11 16:55:49.044727 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.044694 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb4f96a5-8b50-465e-ac45-583fe52ffc26-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km\" (UID: \"eb4f96a5-8b50-465e-ac45-583fe52ffc26\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km" Sep 11 16:55:49.044879 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.044810 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jshzh\" (UniqueName: \"kubernetes.io/projected/cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c-kube-api-access-jshzh\") pod \"payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg\" (UID: \"cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c\") " pod="openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg" Sep 11 16:55:49.044879 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.044834 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c-plugins-config-volume\") pod \"payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg\" (UID: \"cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c\") " pod="openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg" Sep 11 16:55:49.044879 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.044857 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjsw6\" (UniqueName: \"kubernetes.io/projected/eb4f96a5-8b50-465e-ac45-583fe52ffc26-kube-api-access-zjsw6\") pod \"payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km\" (UID: \"eb4f96a5-8b50-465e-ac45-583fe52ffc26\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km" Sep 11 16:55:49.045524 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.045480 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c-plugins-config-volume\") pod \"payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg\" (UID: \"cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c\") " pod="openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg" Sep 11 16:55:49.055997 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.055968 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jshzh\" (UniqueName: \"kubernetes.io/projected/cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c-kube-api-access-jshzh\") pod \"payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg\" (UID: \"cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c\") " pod="openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg" Sep 11 16:55:49.146308 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.146274 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjsw6\" (UniqueName: \"kubernetes.io/projected/eb4f96a5-8b50-465e-ac45-583fe52ffc26-kube-api-access-zjsw6\") pod \"payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km\" (UID: \"eb4f96a5-8b50-465e-ac45-583fe52ffc26\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km" Sep 11 16:55:49.146486 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.146332 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb4f96a5-8b50-465e-ac45-583fe52ffc26-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km\" (UID: \"eb4f96a5-8b50-465e-ac45-583fe52ffc26\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km" Sep 11 16:55:49.147197 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.147177 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb4f96a5-8b50-465e-ac45-583fe52ffc26-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km\" (UID: \"eb4f96a5-8b50-465e-ac45-583fe52ffc26\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km" Sep 11 16:55:49.155223 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.155198 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjsw6\" (UniqueName: \"kubernetes.io/projected/eb4f96a5-8b50-465e-ac45-583fe52ffc26-kube-api-access-zjsw6\") pod \"payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km\" (UID: \"eb4f96a5-8b50-465e-ac45-583fe52ffc26\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km" Sep 11 16:55:49.226678 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.226598 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg" Sep 11 16:55:49.277652 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.277622 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km" Sep 11 16:55:49.498368 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.498338 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg"] Sep 11 16:55:49.501682 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:55:49.501651 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde4ea5e_64cd_4bad_a527_b5e5d5f0de5c.slice/crio-c4cd7f914fdd368c4d090a71ffff32a75c159811a68223b15f5e806b4d3b3d1b WatchSource:0}: Error finding container c4cd7f914fdd368c4d090a71ffff32a75c159811a68223b15f5e806b4d3b3d1b: Status 404 returned error can't find the container with id c4cd7f914fdd368c4d090a71ffff32a75c159811a68223b15f5e806b4d3b3d1b Sep 11 16:55:49.560589 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.560560 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km"] Sep 11 16:55:49.562393 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:55:49.562364 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4f96a5_8b50_465e_ac45_583fe52ffc26.slice/crio-7f2675a971fa0b5929d06bd2697ed4f3f708d01cd131699dad81a73b5108b777 WatchSource:0}: Error finding container 7f2675a971fa0b5929d06bd2697ed4f3f708d01cd131699dad81a73b5108b777: Status 404 returned error can't find the container with id 7f2675a971fa0b5929d06bd2697ed4f3f708d01cd131699dad81a73b5108b777 Sep 11 16:55:49.665184 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.665143 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz" Sep 11 16:55:49.720387 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.720313 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km" event={"ID":"eb4f96a5-8b50-465e-ac45-583fe52ffc26","Type":"ContainerStarted","Data":"8b6758b1b250f9c57cfc3b7bdd8abff525ffd8a120e7068da7b9ddce0207f62e"} Sep 11 16:55:49.720387 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.720374 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km" event={"ID":"eb4f96a5-8b50-465e-ac45-583fe52ffc26","Type":"ContainerStarted","Data":"7f2675a971fa0b5929d06bd2697ed4f3f708d01cd131699dad81a73b5108b777"} Sep 11 16:55:49.722191 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.722161 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg" event={"ID":"cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c","Type":"ContainerStarted","Data":"49251404b3ba66f20a1cf8dfbf46559c4e2c107399b3d1a9e22320592a361193"} Sep 11 16:55:49.722323 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.722210 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg" event={"ID":"cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c","Type":"ContainerStarted","Data":"c4cd7f914fdd368c4d090a71ffff32a75c159811a68223b15f5e806b4d3b3d1b"} Sep 11 16:55:49.722323 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.722269 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg" Sep 11 16:55:49.740466 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.740414 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km" podStartSLOduration=1.7403992179999999 podStartE2EDuration="1.740399218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:55:49.739668516 +0000 UTC m=+765.147770807" watchObservedRunningTime="2026-09-11 16:55:49.740399218 +0000 UTC m=+765.148501487" Sep 11 16:55:49.761218 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:49.761112 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg" podStartSLOduration=1.761092046 podStartE2EDuration="1.761092046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:55:49.760408997 +0000 UTC m=+765.168511335" watchObservedRunningTime="2026-09-11 16:55:49.761092046 +0000 UTC m=+765.169194317" Sep 11 16:55:50.205278 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:50.205197 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq"] Sep 11 16:55:50.209017 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:50.208994 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq" Sep 11 16:55:50.211938 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:50.211906 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ab1ade68-dockercfg-hnm5b\"" Sep 11 16:55:50.212080 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:50.211954 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ab1ade68\"" Sep 11 16:55:50.223247 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:50.223217 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq"] Sep 11 16:55:50.258304 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:50.258265 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgs76\" (UniqueName: \"kubernetes.io/projected/eb8e75e6-5344-4584-aaec-1ae505b8b852-kube-api-access-pgs76\") pod \"payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq\" (UID: \"eb8e75e6-5344-4584-aaec-1ae505b8b852\") " pod="openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq" Sep 11 16:55:50.258471 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:50.258329 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8e75e6-5344-4584-aaec-1ae505b8b852-plugins-config-volume\") pod \"payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq\" (UID: \"eb8e75e6-5344-4584-aaec-1ae505b8b852\") " pod="openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq" Sep 11 16:55:50.359757 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:50.359723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgs76\" (UniqueName: \"kubernetes.io/projected/eb8e75e6-5344-4584-aaec-1ae505b8b852-kube-api-access-pgs76\") pod \"payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq\" (UID: \"eb8e75e6-5344-4584-aaec-1ae505b8b852\") " pod="openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq" Sep 11 16:55:50.359757 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:50.359762 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8e75e6-5344-4584-aaec-1ae505b8b852-plugins-config-volume\") pod \"payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq\" (UID: \"eb8e75e6-5344-4584-aaec-1ae505b8b852\") " pod="openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq" Sep 11 16:55:50.360508 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:50.360475 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8e75e6-5344-4584-aaec-1ae505b8b852-plugins-config-volume\") pod \"payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq\" (UID: \"eb8e75e6-5344-4584-aaec-1ae505b8b852\") " pod="openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq" Sep 11 16:55:50.374012 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:50.373980 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgs76\" (UniqueName: \"kubernetes.io/projected/eb8e75e6-5344-4584-aaec-1ae505b8b852-kube-api-access-pgs76\") pod \"payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq\" (UID: \"eb8e75e6-5344-4584-aaec-1ae505b8b852\") " pod="openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq" Sep 11 16:55:50.524589 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:50.524546 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq" Sep 11 16:55:50.672850 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:50.672819 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468" Sep 11 16:55:50.672986 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:50.672896 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc" Sep 11 16:55:50.729667 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:50.729637 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km" Sep 11 16:55:50.805914 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:50.805835 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq"] Sep 11 16:55:50.809992 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:55:50.809963 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8e75e6_5344_4584_aaec_1ae505b8b852.slice/crio-00f1ee9ccd13213599771e26ee3495fa1847dc3746ecaa4dd698b63d9e51d25a WatchSource:0}: Error finding container 00f1ee9ccd13213599771e26ee3495fa1847dc3746ecaa4dd698b63d9e51d25a: Status 404 returned error can't find the container with id 00f1ee9ccd13213599771e26ee3495fa1847dc3746ecaa4dd698b63d9e51d25a Sep 11 16:55:51.677353 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:51.677323 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4" Sep 11 16:55:51.677832 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:51.677617 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc" Sep 11 16:55:51.737539 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:51.737479 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq" event={"ID":"eb8e75e6-5344-4584-aaec-1ae505b8b852","Type":"ContainerStarted","Data":"21e6b8107cf060dce9d8674fbe85b5437837cb7147afe10df6a12046a2d43ede"} Sep 11 16:55:51.737726 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:51.737568 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq" event={"ID":"eb8e75e6-5344-4584-aaec-1ae505b8b852","Type":"ContainerStarted","Data":"00f1ee9ccd13213599771e26ee3495fa1847dc3746ecaa4dd698b63d9e51d25a"} Sep 11 16:55:51.758526 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:51.758436 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq" podStartSLOduration=1.758414376 podStartE2EDuration="1.758414376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:55:51.757336079 +0000 UTC m=+767.165438349" watchObservedRunningTime="2026-09-11 16:55:51.758414376 +0000 UTC m=+767.166516644" Sep 11 16:55:52.742756 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:52.742727 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq" Sep 11 16:55:59.196707 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.196674 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7"] Sep 11 16:55:59.201301 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.201284 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7" Sep 11 16:55:59.203863 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.203843 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-32d680\"" Sep 11 16:55:59.203964 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.203934 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-32d680-dockercfg-pd49v\"" Sep 11 16:55:59.212764 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.212741 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7"] Sep 11 16:55:59.223150 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.223119 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf"] Sep 11 16:55:59.226694 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.226673 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf" Sep 11 16:55:59.236463 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.236418 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf"] Sep 11 16:55:59.243893 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.243861 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5e888-ed40-44f9-b440-7f53f3df906b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7\" (UID: \"6ab5e888-ed40-44f9-b440-7f53f3df906b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7" Sep 11 16:55:59.244070 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.244047 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25kqr\" (UniqueName: \"kubernetes.io/projected/6ab5e888-ed40-44f9-b440-7f53f3df906b-kube-api-access-25kqr\") pod \"payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7\" (UID: \"6ab5e888-ed40-44f9-b440-7f53f3df906b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7" Sep 11 16:55:59.344948 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.344916 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25kqr\" (UniqueName: \"kubernetes.io/projected/6ab5e888-ed40-44f9-b440-7f53f3df906b-kube-api-access-25kqr\") pod \"payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7\" (UID: \"6ab5e888-ed40-44f9-b440-7f53f3df906b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7" Sep 11 16:55:59.345133 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.344957 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzvwg\" (UniqueName: \"kubernetes.io/projected/5696d4b7-12a5-4508-aa80-7d6d4e1d9f68-kube-api-access-nzvwg\") pod \"payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf\" (UID: \"5696d4b7-12a5-4508-aa80-7d6d4e1d9f68\") " pod="openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf" Sep 11 16:55:59.345133 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.345009 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5e888-ed40-44f9-b440-7f53f3df906b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7\" (UID: \"6ab5e888-ed40-44f9-b440-7f53f3df906b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7" Sep 11 16:55:59.345264 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.345145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5696d4b7-12a5-4508-aa80-7d6d4e1d9f68-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf\" (UID: \"5696d4b7-12a5-4508-aa80-7d6d4e1d9f68\") " pod="openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf" Sep 11 16:55:59.345647 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.345630 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5e888-ed40-44f9-b440-7f53f3df906b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7\" (UID: \"6ab5e888-ed40-44f9-b440-7f53f3df906b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7" Sep 11 16:55:59.356145 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.356122 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25kqr\" (UniqueName: \"kubernetes.io/projected/6ab5e888-ed40-44f9-b440-7f53f3df906b-kube-api-access-25kqr\") pod \"payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7\" (UID: \"6ab5e888-ed40-44f9-b440-7f53f3df906b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7" Sep 11 16:55:59.446606 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.446570 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzvwg\" (UniqueName: \"kubernetes.io/projected/5696d4b7-12a5-4508-aa80-7d6d4e1d9f68-kube-api-access-nzvwg\") pod \"payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf\" (UID: \"5696d4b7-12a5-4508-aa80-7d6d4e1d9f68\") " pod="openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf" Sep 11 16:55:59.446819 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.446645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5696d4b7-12a5-4508-aa80-7d6d4e1d9f68-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf\" (UID: \"5696d4b7-12a5-4508-aa80-7d6d4e1d9f68\") " pod="openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf" Sep 11 16:55:59.447381 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.447354 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5696d4b7-12a5-4508-aa80-7d6d4e1d9f68-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf\" (UID: \"5696d4b7-12a5-4508-aa80-7d6d4e1d9f68\") " pod="openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf" Sep 11 16:55:59.455945 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.455921 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzvwg\" (UniqueName: \"kubernetes.io/projected/5696d4b7-12a5-4508-aa80-7d6d4e1d9f68-kube-api-access-nzvwg\") pod \"payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf\" (UID: \"5696d4b7-12a5-4508-aa80-7d6d4e1d9f68\") " pod="openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf" Sep 11 16:55:59.512690 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.512664 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7" Sep 11 16:55:59.539675 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.539638 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf" Sep 11 16:55:59.790606 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.790578 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7"] Sep 11 16:55:59.791623 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:55:59.791601 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab5e888_ed40_44f9_b440_7f53f3df906b.slice/crio-74c9ddd67123a5c2c3d01064717efcbd20b21fc62f07e93b8b980a73eb9500e1 WatchSource:0}: Error finding container 74c9ddd67123a5c2c3d01064717efcbd20b21fc62f07e93b8b980a73eb9500e1: Status 404 returned error can't find the container with id 74c9ddd67123a5c2c3d01064717efcbd20b21fc62f07e93b8b980a73eb9500e1 Sep 11 16:55:59.824405 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:55:59.824378 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf"] Sep 11 16:55:59.826851 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:55:59.826826 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5696d4b7_12a5_4508_aa80_7d6d4e1d9f68.slice/crio-a17dc4b18f7a94f37dde6fa0b6f37cd8d13aabc5cc634942ce1eb42fa7783f0c WatchSource:0}: Error finding container a17dc4b18f7a94f37dde6fa0b6f37cd8d13aabc5cc634942ce1eb42fa7783f0c: Status 404 returned error can't find the container with id a17dc4b18f7a94f37dde6fa0b6f37cd8d13aabc5cc634942ce1eb42fa7783f0c Sep 11 16:56:00.731378 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:00.731338 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg" Sep 11 16:56:00.785759 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:00.785722 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7" event={"ID":"6ab5e888-ed40-44f9-b440-7f53f3df906b","Type":"ContainerStarted","Data":"470cafe5b4e3d96b0badec3d653ebe83d7d46ff6877e39e9c5e2499e21b50fa3"} Sep 11 16:56:00.785948 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:00.785788 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7" event={"ID":"6ab5e888-ed40-44f9-b440-7f53f3df906b","Type":"ContainerStarted","Data":"74c9ddd67123a5c2c3d01064717efcbd20b21fc62f07e93b8b980a73eb9500e1"} Sep 11 16:56:00.785948 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:00.785865 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7" Sep 11 16:56:00.787636 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:00.787611 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf" event={"ID":"5696d4b7-12a5-4508-aa80-7d6d4e1d9f68","Type":"ContainerStarted","Data":"b0b9efd163fb10f1f038cc11d0603541c4041ba2d0d3ab216225b02349e57654"} Sep 11 16:56:00.787763 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:00.787648 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf" event={"ID":"5696d4b7-12a5-4508-aa80-7d6d4e1d9f68","Type":"ContainerStarted","Data":"a17dc4b18f7a94f37dde6fa0b6f37cd8d13aabc5cc634942ce1eb42fa7783f0c"} Sep 11 16:56:00.787821 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:00.787806 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf" Sep 11 16:56:00.804435 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:00.804393 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7" podStartSLOduration=1.804378586 podStartE2EDuration="1.804378586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:56:00.802561851 +0000 UTC m=+776.210664147" watchObservedRunningTime="2026-09-11 16:56:00.804378586 +0000 UTC m=+776.212480855" Sep 11 16:56:00.821084 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:00.821021 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf" podStartSLOduration=1.821001042 podStartE2EDuration="1.821001042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:56:00.81949229 +0000 UTC m=+776.227594589" watchObservedRunningTime="2026-09-11 16:56:00.821001042 +0000 UTC m=+776.229103311" Sep 11 16:56:01.740043 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:01.740012 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km" Sep 11 16:56:03.751302 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:03.751266 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq" Sep 11 16:56:11.794834 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:11.794804 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf" Sep 11 16:56:11.795409 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:11.794889 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7" Sep 11 16:56:27.935093 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:27.935013 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq"] Sep 11 16:56:27.935651 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:27.935219 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq" podUID="eb8e75e6-5344-4584-aaec-1ae505b8b852" containerName="payload-processing" containerID="cri-o://21e6b8107cf060dce9d8674fbe85b5437837cb7147afe10df6a12046a2d43ede" gracePeriod=30 Sep 11 16:56:28.190678 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:28.190616 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq" Sep 11 16:56:28.224592 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:28.224554 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgs76\" (UniqueName: \"kubernetes.io/projected/eb8e75e6-5344-4584-aaec-1ae505b8b852-kube-api-access-pgs76\") pod \"eb8e75e6-5344-4584-aaec-1ae505b8b852\" (UID: \"eb8e75e6-5344-4584-aaec-1ae505b8b852\") " Sep 11 16:56:28.224863 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:28.224625 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8e75e6-5344-4584-aaec-1ae505b8b852-plugins-config-volume\") pod \"eb8e75e6-5344-4584-aaec-1ae505b8b852\" (UID: \"eb8e75e6-5344-4584-aaec-1ae505b8b852\") " Sep 11 16:56:28.225713 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:28.225633 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8e75e6-5344-4584-aaec-1ae505b8b852-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb8e75e6-5344-4584-aaec-1ae505b8b852" (UID: "eb8e75e6-5344-4584-aaec-1ae505b8b852"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:56:28.227778 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:28.227750 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8e75e6-5344-4584-aaec-1ae505b8b852-kube-api-access-pgs76" (OuterVolumeSpecName: "kube-api-access-pgs76") pod "eb8e75e6-5344-4584-aaec-1ae505b8b852" (UID: "eb8e75e6-5344-4584-aaec-1ae505b8b852"). InnerVolumeSpecName "kube-api-access-pgs76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:56:28.325593 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:28.325562 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgs76\" (UniqueName: \"kubernetes.io/projected/eb8e75e6-5344-4584-aaec-1ae505b8b852-kube-api-access-pgs76\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:56:28.325593 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:28.325589 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8e75e6-5344-4584-aaec-1ae505b8b852-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:56:28.940762 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:28.940728 2577 generic.go:356] "Generic (PLEG): container finished" podID="eb8e75e6-5344-4584-aaec-1ae505b8b852" containerID="21e6b8107cf060dce9d8674fbe85b5437837cb7147afe10df6a12046a2d43ede" exitCode=0 Sep 11 16:56:28.940762 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:28.940768 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq" Sep 11 16:56:28.941241 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:28.940812 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq" event={"ID":"eb8e75e6-5344-4584-aaec-1ae505b8b852","Type":"ContainerDied","Data":"21e6b8107cf060dce9d8674fbe85b5437837cb7147afe10df6a12046a2d43ede"} Sep 11 16:56:28.941241 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:28.940870 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq" event={"ID":"eb8e75e6-5344-4584-aaec-1ae505b8b852","Type":"ContainerDied","Data":"00f1ee9ccd13213599771e26ee3495fa1847dc3746ecaa4dd698b63d9e51d25a"} Sep 11 16:56:28.941241 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:28.940899 2577 scope.go:117] "RemoveContainer" containerID="21e6b8107cf060dce9d8674fbe85b5437837cb7147afe10df6a12046a2d43ede" Sep 11 16:56:28.950186 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:28.950164 2577 scope.go:117] "RemoveContainer" containerID="21e6b8107cf060dce9d8674fbe85b5437837cb7147afe10df6a12046a2d43ede" Sep 11 16:56:28.950448 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:56:28.950418 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e6b8107cf060dce9d8674fbe85b5437837cb7147afe10df6a12046a2d43ede\": container with ID starting with 21e6b8107cf060dce9d8674fbe85b5437837cb7147afe10df6a12046a2d43ede not found: ID does not exist" containerID="21e6b8107cf060dce9d8674fbe85b5437837cb7147afe10df6a12046a2d43ede" Sep 11 16:56:28.950533 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:28.950456 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e6b8107cf060dce9d8674fbe85b5437837cb7147afe10df6a12046a2d43ede"} err="failed to get container status \"21e6b8107cf060dce9d8674fbe85b5437837cb7147afe10df6a12046a2d43ede\": rpc error: code = NotFound desc = could not find container \"21e6b8107cf060dce9d8674fbe85b5437837cb7147afe10df6a12046a2d43ede\": container with ID starting with 21e6b8107cf060dce9d8674fbe85b5437837cb7147afe10df6a12046a2d43ede not found: ID does not exist" Sep 11 16:56:28.963328 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:28.963303 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq"] Sep 11 16:56:28.967990 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:28.967964 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ab1ade68-5cf97ccd79-zj2dq"] Sep 11 16:56:29.094363 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:29.094326 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb8e75e6-5344-4584-aaec-1ae505b8b852" path="/var/lib/kubelet/pods/eb8e75e6-5344-4584-aaec-1ae505b8b852/volumes" Sep 11 16:56:35.189839 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.189804 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj"] Sep 11 16:56:35.190312 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.190208 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb8e75e6-5344-4584-aaec-1ae505b8b852" containerName="payload-processing" Sep 11 16:56:35.190312 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.190219 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb8e75e6-5344-4584-aaec-1ae505b8b852" containerName="payload-processing" Sep 11 16:56:35.190312 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.190278 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb8e75e6-5344-4584-aaec-1ae505b8b852" containerName="payload-processing" Sep 11 16:56:35.193397 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.193381 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj" Sep 11 16:56:35.196389 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.196366 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-5969a028\"" Sep 11 16:56:35.196539 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.196444 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-5969a028-dockercfg-r2kjf\"" Sep 11 16:56:35.202479 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.202454 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj"] Sep 11 16:56:35.283731 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.283695 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6zpq\" (UniqueName: \"kubernetes.io/projected/34c104aa-4003-4d56-bd1a-8d74f48142ee-kube-api-access-m6zpq\") pod \"payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj\" (UID: \"34c104aa-4003-4d56-bd1a-8d74f48142ee\") " pod="openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj" Sep 11 16:56:35.283909 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.283805 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c104aa-4003-4d56-bd1a-8d74f48142ee-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj\" (UID: \"34c104aa-4003-4d56-bd1a-8d74f48142ee\") " pod="openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj" Sep 11 16:56:35.384951 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.384910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c104aa-4003-4d56-bd1a-8d74f48142ee-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj\" (UID: \"34c104aa-4003-4d56-bd1a-8d74f48142ee\") " pod="openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj" Sep 11 16:56:35.385152 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.385013 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6zpq\" (UniqueName: \"kubernetes.io/projected/34c104aa-4003-4d56-bd1a-8d74f48142ee-kube-api-access-m6zpq\") pod \"payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj\" (UID: \"34c104aa-4003-4d56-bd1a-8d74f48142ee\") " pod="openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj" Sep 11 16:56:35.385631 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.385608 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c104aa-4003-4d56-bd1a-8d74f48142ee-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj\" (UID: \"34c104aa-4003-4d56-bd1a-8d74f48142ee\") " pod="openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj" Sep 11 16:56:35.394406 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.394381 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6zpq\" (UniqueName: \"kubernetes.io/projected/34c104aa-4003-4d56-bd1a-8d74f48142ee-kube-api-access-m6zpq\") pod \"payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj\" (UID: \"34c104aa-4003-4d56-bd1a-8d74f48142ee\") " pod="openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj" Sep 11 16:56:35.505230 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.505198 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj" Sep 11 16:56:35.783399 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.783374 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj"] Sep 11 16:56:35.784791 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:56:35.784762 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c104aa_4003_4d56_bd1a_8d74f48142ee.slice/crio-933f94dbe2b5aac9d3c94068cb59d5004c7152fb0d00396b0b3d2cb764817e29 WatchSource:0}: Error finding container 933f94dbe2b5aac9d3c94068cb59d5004c7152fb0d00396b0b3d2cb764817e29: Status 404 returned error can't find the container with id 933f94dbe2b5aac9d3c94068cb59d5004c7152fb0d00396b0b3d2cb764817e29 Sep 11 16:56:35.977563 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.977489 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj" event={"ID":"34c104aa-4003-4d56-bd1a-8d74f48142ee","Type":"ContainerStarted","Data":"6545cdb9a489f0dc62c4fa0ac81576966d57a5d5ffe2f42324c1cbd11a68e0c4"} Sep 11 16:56:35.977751 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.977576 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj" event={"ID":"34c104aa-4003-4d56-bd1a-8d74f48142ee","Type":"ContainerStarted","Data":"933f94dbe2b5aac9d3c94068cb59d5004c7152fb0d00396b0b3d2cb764817e29"} Sep 11 16:56:35.977751 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.977707 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj" Sep 11 16:56:35.996368 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:35.996298 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj" podStartSLOduration=0.996282158 podStartE2EDuration="996.282158ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:56:35.996175492 +0000 UTC m=+811.404277772" watchObservedRunningTime="2026-09-11 16:56:35.996282158 +0000 UTC m=+811.404384427" Sep 11 16:56:46.985203 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:56:46.985167 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj" Sep 11 16:57:03.132211 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:03.132181 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj"] Sep 11 16:57:03.133181 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:03.132566 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj" podUID="34c104aa-4003-4d56-bd1a-8d74f48142ee" containerName="payload-processing" containerID="cri-o://6545cdb9a489f0dc62c4fa0ac81576966d57a5d5ffe2f42324c1cbd11a68e0c4" gracePeriod=30 Sep 11 16:57:03.387649 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:03.387589 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj" Sep 11 16:57:03.561610 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:03.561577 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6zpq\" (UniqueName: \"kubernetes.io/projected/34c104aa-4003-4d56-bd1a-8d74f48142ee-kube-api-access-m6zpq\") pod \"34c104aa-4003-4d56-bd1a-8d74f48142ee\" (UID: \"34c104aa-4003-4d56-bd1a-8d74f48142ee\") " Sep 11 16:57:03.561803 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:03.561679 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c104aa-4003-4d56-bd1a-8d74f48142ee-plugins-config-volume\") pod \"34c104aa-4003-4d56-bd1a-8d74f48142ee\" (UID: \"34c104aa-4003-4d56-bd1a-8d74f48142ee\") " Sep 11 16:57:03.562055 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:03.562032 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c104aa-4003-4d56-bd1a-8d74f48142ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "34c104aa-4003-4d56-bd1a-8d74f48142ee" (UID: "34c104aa-4003-4d56-bd1a-8d74f48142ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:57:03.563822 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:03.563791 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c104aa-4003-4d56-bd1a-8d74f48142ee-kube-api-access-m6zpq" (OuterVolumeSpecName: "kube-api-access-m6zpq") pod "34c104aa-4003-4d56-bd1a-8d74f48142ee" (UID: "34c104aa-4003-4d56-bd1a-8d74f48142ee"). InnerVolumeSpecName "kube-api-access-m6zpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:57:03.663169 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:03.663085 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c104aa-4003-4d56-bd1a-8d74f48142ee-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:57:03.663169 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:03.663115 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6zpq\" (UniqueName: \"kubernetes.io/projected/34c104aa-4003-4d56-bd1a-8d74f48142ee-kube-api-access-m6zpq\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:57:04.123869 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:04.123830 2577 generic.go:356] "Generic (PLEG): container finished" podID="34c104aa-4003-4d56-bd1a-8d74f48142ee" containerID="6545cdb9a489f0dc62c4fa0ac81576966d57a5d5ffe2f42324c1cbd11a68e0c4" exitCode=0 Sep 11 16:57:04.123869 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:04.123874 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj" Sep 11 16:57:04.124131 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:04.123914 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj" event={"ID":"34c104aa-4003-4d56-bd1a-8d74f48142ee","Type":"ContainerDied","Data":"6545cdb9a489f0dc62c4fa0ac81576966d57a5d5ffe2f42324c1cbd11a68e0c4"} Sep 11 16:57:04.124131 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:04.123972 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj" event={"ID":"34c104aa-4003-4d56-bd1a-8d74f48142ee","Type":"ContainerDied","Data":"933f94dbe2b5aac9d3c94068cb59d5004c7152fb0d00396b0b3d2cb764817e29"} Sep 11 16:57:04.124131 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:04.123995 2577 scope.go:117] "RemoveContainer" containerID="6545cdb9a489f0dc62c4fa0ac81576966d57a5d5ffe2f42324c1cbd11a68e0c4" Sep 11 16:57:04.134100 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:04.133970 2577 scope.go:117] "RemoveContainer" containerID="6545cdb9a489f0dc62c4fa0ac81576966d57a5d5ffe2f42324c1cbd11a68e0c4" Sep 11 16:57:04.134356 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:57:04.134277 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6545cdb9a489f0dc62c4fa0ac81576966d57a5d5ffe2f42324c1cbd11a68e0c4\": container with ID starting with 6545cdb9a489f0dc62c4fa0ac81576966d57a5d5ffe2f42324c1cbd11a68e0c4 not found: ID does not exist" containerID="6545cdb9a489f0dc62c4fa0ac81576966d57a5d5ffe2f42324c1cbd11a68e0c4" Sep 11 16:57:04.134356 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:04.134306 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6545cdb9a489f0dc62c4fa0ac81576966d57a5d5ffe2f42324c1cbd11a68e0c4"} err="failed to get container status \"6545cdb9a489f0dc62c4fa0ac81576966d57a5d5ffe2f42324c1cbd11a68e0c4\": rpc error: code = NotFound desc = could not find container \"6545cdb9a489f0dc62c4fa0ac81576966d57a5d5ffe2f42324c1cbd11a68e0c4\": container with ID starting with 6545cdb9a489f0dc62c4fa0ac81576966d57a5d5ffe2f42324c1cbd11a68e0c4 not found: ID does not exist" Sep 11 16:57:04.145899 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:04.145869 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj"] Sep 11 16:57:04.150052 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:04.150027 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5969a028-59c4cf6f6b-9x5kj"] Sep 11 16:57:05.094118 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:05.094085 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34c104aa-4003-4d56-bd1a-8d74f48142ee" path="/var/lib/kubelet/pods/34c104aa-4003-4d56-bd1a-8d74f48142ee/volumes" Sep 11 16:57:50.063908 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:50.063872 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676"] Sep 11 16:57:50.064346 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:50.064263 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34c104aa-4003-4d56-bd1a-8d74f48142ee" containerName="payload-processing" Sep 11 16:57:50.064346 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:50.064275 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="34c104aa-4003-4d56-bd1a-8d74f48142ee" containerName="payload-processing" Sep 11 16:57:50.064346 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:50.064345 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="34c104aa-4003-4d56-bd1a-8d74f48142ee" containerName="payload-processing" Sep 11 16:57:50.067351 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:50.067335 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676" Sep 11 16:57:50.070334 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:50.070315 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-9abacf77-dockercfg-b7d48\"" Sep 11 16:57:50.070415 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:50.070367 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-9abacf77\"" Sep 11 16:57:50.078010 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:50.077988 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676"] Sep 11 16:57:50.096262 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:50.096226 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4fvh\" (UniqueName: \"kubernetes.io/projected/eef5202e-212b-499e-a630-e8c13d9ac823-kube-api-access-l4fvh\") pod \"payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676\" (UID: \"eef5202e-212b-499e-a630-e8c13d9ac823\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676" Sep 11 16:57:50.096427 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:50.096281 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef5202e-212b-499e-a630-e8c13d9ac823-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676\" (UID: \"eef5202e-212b-499e-a630-e8c13d9ac823\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676" Sep 11 16:57:50.196961 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:50.196918 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4fvh\" (UniqueName: \"kubernetes.io/projected/eef5202e-212b-499e-a630-e8c13d9ac823-kube-api-access-l4fvh\") pod \"payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676\" (UID: \"eef5202e-212b-499e-a630-e8c13d9ac823\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676" Sep 11 16:57:50.197204 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:50.197008 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef5202e-212b-499e-a630-e8c13d9ac823-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676\" (UID: \"eef5202e-212b-499e-a630-e8c13d9ac823\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676" Sep 11 16:57:50.197774 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:50.197750 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef5202e-212b-499e-a630-e8c13d9ac823-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676\" (UID: \"eef5202e-212b-499e-a630-e8c13d9ac823\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676" Sep 11 16:57:50.206011 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:50.205980 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4fvh\" (UniqueName: \"kubernetes.io/projected/eef5202e-212b-499e-a630-e8c13d9ac823-kube-api-access-l4fvh\") pod \"payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676\" (UID: \"eef5202e-212b-499e-a630-e8c13d9ac823\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676" Sep 11 16:57:50.378697 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:50.378613 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676" Sep 11 16:57:50.653766 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:50.653731 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676"] Sep 11 16:57:50.656225 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:57:50.656193 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef5202e_212b_499e_a630_e8c13d9ac823.slice/crio-7ec5ebd639c5a7a88465fbc68e2606fe730026ef36e23653e34bcd7f0866cb4d WatchSource:0}: Error finding container 7ec5ebd639c5a7a88465fbc68e2606fe730026ef36e23653e34bcd7f0866cb4d: Status 404 returned error can't find the container with id 7ec5ebd639c5a7a88465fbc68e2606fe730026ef36e23653e34bcd7f0866cb4d Sep 11 16:57:51.369579 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:51.369543 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676" event={"ID":"eef5202e-212b-499e-a630-e8c13d9ac823","Type":"ContainerStarted","Data":"82412adf5f3d9c613fa024a7a032aa7d8d5b1fb415a2c556b269193752adf9fb"} Sep 11 16:57:51.370052 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:51.369597 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676" event={"ID":"eef5202e-212b-499e-a630-e8c13d9ac823","Type":"ContainerStarted","Data":"7ec5ebd639c5a7a88465fbc68e2606fe730026ef36e23653e34bcd7f0866cb4d"} Sep 11 16:57:51.370052 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:51.369705 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676" Sep 11 16:57:51.388555 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:57:51.388490 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676" podStartSLOduration=1.388474778 podStartE2EDuration="1.388474778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:57:51.38761814 +0000 UTC m=+886.795720430" watchObservedRunningTime="2026-09-11 16:57:51.388474778 +0000 UTC m=+886.796577047" Sep 11 16:58:02.377731 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:02.377646 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676" Sep 11 16:58:05.040180 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:05.040152 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 16:58:05.041637 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:05.041615 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 16:58:07.887392 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:07.887353 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8b1a16-52hwd"] Sep 11 16:58:07.891105 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:07.891088 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8b1a16-52hwd" Sep 11 16:58:07.894107 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:07.894084 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pc4qb\"" Sep 11 16:58:07.894249 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:07.894166 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 16:58:07.900641 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:07.900613 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8b1a16-52hwd"] Sep 11 16:58:07.960259 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:07.960223 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a6f4d82-0b45-4595-a3d5-659e3a83635b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-8b1a16-52hwd\" (UID: \"9a6f4d82-0b45-4595-a3d5-659e3a83635b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8b1a16-52hwd" Sep 11 16:58:08.061489 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:08.061440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a6f4d82-0b45-4595-a3d5-659e3a83635b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-8b1a16-52hwd\" (UID: \"9a6f4d82-0b45-4595-a3d5-659e3a83635b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8b1a16-52hwd" Sep 11 16:58:08.062143 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:08.062120 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a6f4d82-0b45-4595-a3d5-659e3a83635b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-8b1a16-52hwd\" (UID: \"9a6f4d82-0b45-4595-a3d5-659e3a83635b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8b1a16-52hwd" Sep 11 16:58:08.202669 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:08.202580 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8b1a16-52hwd" Sep 11 16:58:08.482199 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:08.482174 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8b1a16-52hwd"] Sep 11 16:58:08.484367 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:58:08.484337 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6f4d82_0b45_4595_a3d5_659e3a83635b.slice/crio-8d691114d71e73f72a8af0ced316a1b4a24be9b26ce8b9667f0826711038f963 WatchSource:0}: Error finding container 8d691114d71e73f72a8af0ced316a1b4a24be9b26ce8b9667f0826711038f963: Status 404 returned error can't find the container with id 8d691114d71e73f72a8af0ced316a1b4a24be9b26ce8b9667f0826711038f963 Sep 11 16:58:08.486401 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:08.486383 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 16:58:09.470018 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:09.469985 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8b1a16-52hwd" event={"ID":"9a6f4d82-0b45-4595-a3d5-659e3a83635b","Type":"ContainerStarted","Data":"8d691114d71e73f72a8af0ced316a1b4a24be9b26ce8b9667f0826711038f963"} Sep 11 16:58:26.227272 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:26.227233 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676"] Sep 11 16:58:26.229841 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:26.227540 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676" podUID="eef5202e-212b-499e-a630-e8c13d9ac823" containerName="payload-pre-processing" containerID="cri-o://82412adf5f3d9c613fa024a7a032aa7d8d5b1fb415a2c556b269193752adf9fb" gracePeriod=30 Sep 11 16:58:26.497237 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:26.497206 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676" Sep 11 16:58:26.558716 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:26.558685 2577 generic.go:356] "Generic (PLEG): container finished" podID="eef5202e-212b-499e-a630-e8c13d9ac823" containerID="82412adf5f3d9c613fa024a7a032aa7d8d5b1fb415a2c556b269193752adf9fb" exitCode=0 Sep 11 16:58:26.558948 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:26.558729 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676" Sep 11 16:58:26.558948 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:26.558767 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676" event={"ID":"eef5202e-212b-499e-a630-e8c13d9ac823","Type":"ContainerDied","Data":"82412adf5f3d9c613fa024a7a032aa7d8d5b1fb415a2c556b269193752adf9fb"} Sep 11 16:58:26.558948 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:26.558832 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676" event={"ID":"eef5202e-212b-499e-a630-e8c13d9ac823","Type":"ContainerDied","Data":"7ec5ebd639c5a7a88465fbc68e2606fe730026ef36e23653e34bcd7f0866cb4d"} Sep 11 16:58:26.558948 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:26.558862 2577 scope.go:117] "RemoveContainer" containerID="82412adf5f3d9c613fa024a7a032aa7d8d5b1fb415a2c556b269193752adf9fb" Sep 11 16:58:26.567735 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:26.567717 2577 scope.go:117] "RemoveContainer" containerID="82412adf5f3d9c613fa024a7a032aa7d8d5b1fb415a2c556b269193752adf9fb" Sep 11 16:58:26.568032 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:58:26.568013 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82412adf5f3d9c613fa024a7a032aa7d8d5b1fb415a2c556b269193752adf9fb\": container with ID starting with 82412adf5f3d9c613fa024a7a032aa7d8d5b1fb415a2c556b269193752adf9fb not found: ID does not exist" containerID="82412adf5f3d9c613fa024a7a032aa7d8d5b1fb415a2c556b269193752adf9fb" Sep 11 16:58:26.568091 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:26.568042 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82412adf5f3d9c613fa024a7a032aa7d8d5b1fb415a2c556b269193752adf9fb"} err="failed to get container status \"82412adf5f3d9c613fa024a7a032aa7d8d5b1fb415a2c556b269193752adf9fb\": rpc error: code = NotFound desc = could not find container \"82412adf5f3d9c613fa024a7a032aa7d8d5b1fb415a2c556b269193752adf9fb\": container with ID starting with 82412adf5f3d9c613fa024a7a032aa7d8d5b1fb415a2c556b269193752adf9fb not found: ID does not exist" Sep 11 16:58:26.639750 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:26.639714 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef5202e-212b-499e-a630-e8c13d9ac823-plugins-config-volume\") pod \"eef5202e-212b-499e-a630-e8c13d9ac823\" (UID: \"eef5202e-212b-499e-a630-e8c13d9ac823\") " Sep 11 16:58:26.639920 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:26.639764 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4fvh\" (UniqueName: \"kubernetes.io/projected/eef5202e-212b-499e-a630-e8c13d9ac823-kube-api-access-l4fvh\") pod \"eef5202e-212b-499e-a630-e8c13d9ac823\" (UID: \"eef5202e-212b-499e-a630-e8c13d9ac823\") " Sep 11 16:58:26.640186 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:26.640160 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef5202e-212b-499e-a630-e8c13d9ac823-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eef5202e-212b-499e-a630-e8c13d9ac823" (UID: "eef5202e-212b-499e-a630-e8c13d9ac823"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:58:26.641977 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:26.641951 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef5202e-212b-499e-a630-e8c13d9ac823-kube-api-access-l4fvh" (OuterVolumeSpecName: "kube-api-access-l4fvh") pod "eef5202e-212b-499e-a630-e8c13d9ac823" (UID: "eef5202e-212b-499e-a630-e8c13d9ac823"). InnerVolumeSpecName "kube-api-access-l4fvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:58:26.740771 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:26.740736 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef5202e-212b-499e-a630-e8c13d9ac823-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:58:26.740771 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:26.740766 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4fvh\" (UniqueName: \"kubernetes.io/projected/eef5202e-212b-499e-a630-e8c13d9ac823-kube-api-access-l4fvh\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:58:26.882410 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:26.882375 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676"] Sep 11 16:58:26.887562 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:26.887532 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9abacf77-b8cf54698-q2676"] Sep 11 16:58:27.094998 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:27.094962 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eef5202e-212b-499e-a630-e8c13d9ac823" path="/var/lib/kubelet/pods/eef5202e-212b-499e-a630-e8c13d9ac823/volumes" Sep 11 16:58:33.595415 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:33.595327 2577 generic.go:356] "Generic (PLEG): container finished" podID="9a6f4d82-0b45-4595-a3d5-659e3a83635b" containerID="a3bb7fe2c80d50b8ab23566b47c0675025a653bc41429c6871432fd4209cd031" exitCode=0 Sep 11 16:58:33.595813 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:33.595415 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8b1a16-52hwd" event={"ID":"9a6f4d82-0b45-4595-a3d5-659e3a83635b","Type":"ContainerDied","Data":"a3bb7fe2c80d50b8ab23566b47c0675025a653bc41429c6871432fd4209cd031"} Sep 11 16:58:34.747181 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:34.747151 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8b1a16-52hwd" Sep 11 16:58:34.814019 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:34.813986 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a6f4d82-0b45-4595-a3d5-659e3a83635b-maas-api-service-ca\") pod \"9a6f4d82-0b45-4595-a3d5-659e3a83635b\" (UID: \"9a6f4d82-0b45-4595-a3d5-659e3a83635b\") " Sep 11 16:58:34.814363 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:34.814337 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6f4d82-0b45-4595-a3d5-659e3a83635b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a6f4d82-0b45-4595-a3d5-659e3a83635b" (UID: "9a6f4d82-0b45-4595-a3d5-659e3a83635b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:58:34.914781 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:34.914690 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a6f4d82-0b45-4595-a3d5-659e3a83635b-maas-api-service-ca\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:58:35.608099 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:35.608067 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8b1a16-52hwd" Sep 11 16:58:35.608099 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:35.608078 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8b1a16-52hwd" event={"ID":"9a6f4d82-0b45-4595-a3d5-659e3a83635b","Type":"ContainerDied","Data":"8d691114d71e73f72a8af0ced316a1b4a24be9b26ce8b9667f0826711038f963"} Sep 11 16:58:35.608320 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:35.608132 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d691114d71e73f72a8af0ced316a1b4a24be9b26ce8b9667f0826711038f963" Sep 11 16:58:38.823962 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:38.823918 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg"] Sep 11 16:58:38.824420 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:38.824168 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg" podUID="cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c" containerName="payload-processing" containerID="cri-o://49251404b3ba66f20a1cf8dfbf46559c4e2c107399b3d1a9e22320592a361193" gracePeriod=30 Sep 11 16:58:38.832627 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:38.832599 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km"] Sep 11 16:58:38.832820 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:38.832795 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km" podUID="eb4f96a5-8b50-465e-ac45-583fe52ffc26" containerName="payload-pre-processing" containerID="cri-o://8b6758b1b250f9c57cfc3b7bdd8abff525ffd8a120e7068da7b9ddce0207f62e" gracePeriod=30 Sep 11 16:58:39.012917 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.012846 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8b1a16-52hwd"] Sep 11 16:58:39.015373 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.015347 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-8b1a16-52hwd"] Sep 11 16:58:39.096956 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.096877 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a6f4d82-0b45-4595-a3d5-659e3a83635b" path="/var/lib/kubelet/pods/9a6f4d82-0b45-4595-a3d5-659e3a83635b/volumes" Sep 11 16:58:39.116285 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.116266 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg" Sep 11 16:58:39.128475 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.128445 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km" Sep 11 16:58:39.157523 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.152237 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c-plugins-config-volume\") pod \"cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c\" (UID: \"cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c\") " Sep 11 16:58:39.157523 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.152350 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjsw6\" (UniqueName: \"kubernetes.io/projected/eb4f96a5-8b50-465e-ac45-583fe52ffc26-kube-api-access-zjsw6\") pod \"eb4f96a5-8b50-465e-ac45-583fe52ffc26\" (UID: \"eb4f96a5-8b50-465e-ac45-583fe52ffc26\") " Sep 11 16:58:39.157523 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.152437 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jshzh\" (UniqueName: \"kubernetes.io/projected/cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c-kube-api-access-jshzh\") pod \"cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c\" (UID: \"cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c\") " Sep 11 16:58:39.157523 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.152476 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb4f96a5-8b50-465e-ac45-583fe52ffc26-plugins-config-volume\") pod \"eb4f96a5-8b50-465e-ac45-583fe52ffc26\" (UID: \"eb4f96a5-8b50-465e-ac45-583fe52ffc26\") " Sep 11 16:58:39.157523 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.152814 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c" (UID: "cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:58:39.157523 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.152923 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:58:39.158020 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.157721 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb4f96a5-8b50-465e-ac45-583fe52ffc26-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb4f96a5-8b50-465e-ac45-583fe52ffc26" (UID: "eb4f96a5-8b50-465e-ac45-583fe52ffc26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:58:39.158776 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.158734 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4f96a5-8b50-465e-ac45-583fe52ffc26-kube-api-access-zjsw6" (OuterVolumeSpecName: "kube-api-access-zjsw6") pod "eb4f96a5-8b50-465e-ac45-583fe52ffc26" (UID: "eb4f96a5-8b50-465e-ac45-583fe52ffc26"). InnerVolumeSpecName "kube-api-access-zjsw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:58:39.160316 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.160276 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c-kube-api-access-jshzh" (OuterVolumeSpecName: "kube-api-access-jshzh") pod "cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c" (UID: "cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c"). InnerVolumeSpecName "kube-api-access-jshzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:58:39.254025 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.253986 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjsw6\" (UniqueName: \"kubernetes.io/projected/eb4f96a5-8b50-465e-ac45-583fe52ffc26-kube-api-access-zjsw6\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:58:39.254025 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.254023 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jshzh\" (UniqueName: \"kubernetes.io/projected/cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c-kube-api-access-jshzh\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:58:39.254224 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.254039 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb4f96a5-8b50-465e-ac45-583fe52ffc26-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 16:58:39.629127 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.629090 2577 generic.go:356] "Generic (PLEG): container finished" podID="eb4f96a5-8b50-465e-ac45-583fe52ffc26" containerID="8b6758b1b250f9c57cfc3b7bdd8abff525ffd8a120e7068da7b9ddce0207f62e" exitCode=0 Sep 11 16:58:39.629326 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.629136 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km" Sep 11 16:58:39.629326 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.629177 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km" event={"ID":"eb4f96a5-8b50-465e-ac45-583fe52ffc26","Type":"ContainerDied","Data":"8b6758b1b250f9c57cfc3b7bdd8abff525ffd8a120e7068da7b9ddce0207f62e"} Sep 11 16:58:39.629326 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.629237 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km" event={"ID":"eb4f96a5-8b50-465e-ac45-583fe52ffc26","Type":"ContainerDied","Data":"7f2675a971fa0b5929d06bd2697ed4f3f708d01cd131699dad81a73b5108b777"} Sep 11 16:58:39.629326 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.629269 2577 scope.go:117] "RemoveContainer" containerID="8b6758b1b250f9c57cfc3b7bdd8abff525ffd8a120e7068da7b9ddce0207f62e" Sep 11 16:58:39.630787 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.630765 2577 generic.go:356] "Generic (PLEG): container finished" podID="cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c" containerID="49251404b3ba66f20a1cf8dfbf46559c4e2c107399b3d1a9e22320592a361193" exitCode=0 Sep 11 16:58:39.630898 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.630804 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg" Sep 11 16:58:39.630898 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.630864 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg" event={"ID":"cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c","Type":"ContainerDied","Data":"49251404b3ba66f20a1cf8dfbf46559c4e2c107399b3d1a9e22320592a361193"} Sep 11 16:58:39.631005 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.630905 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg" event={"ID":"cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c","Type":"ContainerDied","Data":"c4cd7f914fdd368c4d090a71ffff32a75c159811a68223b15f5e806b4d3b3d1b"} Sep 11 16:58:39.639885 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.639733 2577 scope.go:117] "RemoveContainer" containerID="8b6758b1b250f9c57cfc3b7bdd8abff525ffd8a120e7068da7b9ddce0207f62e" Sep 11 16:58:39.640038 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:58:39.640020 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b6758b1b250f9c57cfc3b7bdd8abff525ffd8a120e7068da7b9ddce0207f62e\": container with ID starting with 8b6758b1b250f9c57cfc3b7bdd8abff525ffd8a120e7068da7b9ddce0207f62e not found: ID does not exist" containerID="8b6758b1b250f9c57cfc3b7bdd8abff525ffd8a120e7068da7b9ddce0207f62e" Sep 11 16:58:39.640110 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.640050 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6758b1b250f9c57cfc3b7bdd8abff525ffd8a120e7068da7b9ddce0207f62e"} err="failed to get container status \"8b6758b1b250f9c57cfc3b7bdd8abff525ffd8a120e7068da7b9ddce0207f62e\": rpc error: code = NotFound desc = could not find container \"8b6758b1b250f9c57cfc3b7bdd8abff525ffd8a120e7068da7b9ddce0207f62e\": container with ID starting with 8b6758b1b250f9c57cfc3b7bdd8abff525ffd8a120e7068da7b9ddce0207f62e not found: ID does not exist" Sep 11 16:58:39.640110 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.640076 2577 scope.go:117] "RemoveContainer" containerID="49251404b3ba66f20a1cf8dfbf46559c4e2c107399b3d1a9e22320592a361193" Sep 11 16:58:39.648077 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.648059 2577 scope.go:117] "RemoveContainer" containerID="49251404b3ba66f20a1cf8dfbf46559c4e2c107399b3d1a9e22320592a361193" Sep 11 16:58:39.648334 ip-10-0-143-165 kubenswrapper[2577]: E0911 16:58:39.648314 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49251404b3ba66f20a1cf8dfbf46559c4e2c107399b3d1a9e22320592a361193\": container with ID starting with 49251404b3ba66f20a1cf8dfbf46559c4e2c107399b3d1a9e22320592a361193 not found: ID does not exist" containerID="49251404b3ba66f20a1cf8dfbf46559c4e2c107399b3d1a9e22320592a361193" Sep 11 16:58:39.648385 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.648340 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49251404b3ba66f20a1cf8dfbf46559c4e2c107399b3d1a9e22320592a361193"} err="failed to get container status \"49251404b3ba66f20a1cf8dfbf46559c4e2c107399b3d1a9e22320592a361193\": rpc error: code = NotFound desc = could not find container \"49251404b3ba66f20a1cf8dfbf46559c4e2c107399b3d1a9e22320592a361193\": container with ID starting with 49251404b3ba66f20a1cf8dfbf46559c4e2c107399b3d1a9e22320592a361193 not found: ID does not exist" Sep 11 16:58:39.653289 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.653268 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km"] Sep 11 16:58:39.657658 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.657639 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-8b1a16-d5b9458f4-nd6km"] Sep 11 16:58:39.669137 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.669113 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg"] Sep 11 16:58:39.671163 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:39.671145 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-8b1a16-b95999f9-b9rdg"] Sep 11 16:58:41.094436 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:41.094390 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c" path="/var/lib/kubelet/pods/cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c/volumes" Sep 11 16:58:41.094963 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:41.094943 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb4f96a5-8b50-465e-ac45-583fe52ffc26" path="/var/lib/kubelet/pods/eb4f96a5-8b50-465e-ac45-583fe52ffc26/volumes" Sep 11 16:58:54.825733 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.825700 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn"] Sep 11 16:58:54.826098 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.826086 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef5202e-212b-499e-a630-e8c13d9ac823" containerName="payload-pre-processing" Sep 11 16:58:54.826098 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.826098 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef5202e-212b-499e-a630-e8c13d9ac823" containerName="payload-pre-processing" Sep 11 16:58:54.826162 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.826108 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb4f96a5-8b50-465e-ac45-583fe52ffc26" containerName="payload-pre-processing" Sep 11 16:58:54.826162 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.826114 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb4f96a5-8b50-465e-ac45-583fe52ffc26" containerName="payload-pre-processing" Sep 11 16:58:54.826162 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.826127 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c" containerName="payload-processing" Sep 11 16:58:54.826162 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.826132 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c" containerName="payload-processing" Sep 11 16:58:54.826162 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.826141 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a6f4d82-0b45-4595-a3d5-659e3a83635b" containerName="revoke-keys" Sep 11 16:58:54.826162 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.826146 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a6f4d82-0b45-4595-a3d5-659e3a83635b" containerName="revoke-keys" Sep 11 16:58:54.826344 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.826206 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a6f4d82-0b45-4595-a3d5-659e3a83635b" containerName="revoke-keys" Sep 11 16:58:54.826344 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.826213 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef5202e-212b-499e-a630-e8c13d9ac823" containerName="payload-pre-processing" Sep 11 16:58:54.826344 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.826221 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde4ea5e-64cd-4bad-a527-b5e5d5f0de5c" containerName="payload-processing" Sep 11 16:58:54.826344 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.826226 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb4f96a5-8b50-465e-ac45-583fe52ffc26" containerName="payload-pre-processing" Sep 11 16:58:54.831032 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.831006 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn" Sep 11 16:58:54.833649 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.833628 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-2cceea\"" Sep 11 16:58:54.833782 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.833733 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-2cceea-dockercfg-szfkh\"" Sep 11 16:58:54.839694 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.839669 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn"] Sep 11 16:58:54.897397 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.897362 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwsfk\" (UniqueName: \"kubernetes.io/projected/3496d171-a6e6-4469-bc3d-5ee9ade44f17-kube-api-access-mwsfk\") pod \"payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn\" (UID: \"3496d171-a6e6-4469-bc3d-5ee9ade44f17\") " pod="openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn" Sep 11 16:58:54.897563 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.897465 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3496d171-a6e6-4469-bc3d-5ee9ade44f17-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn\" (UID: \"3496d171-a6e6-4469-bc3d-5ee9ade44f17\") " pod="openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn" Sep 11 16:58:54.997982 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.997939 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwsfk\" (UniqueName: \"kubernetes.io/projected/3496d171-a6e6-4469-bc3d-5ee9ade44f17-kube-api-access-mwsfk\") pod \"payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn\" (UID: \"3496d171-a6e6-4469-bc3d-5ee9ade44f17\") " pod="openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn" Sep 11 16:58:54.998196 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.998045 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3496d171-a6e6-4469-bc3d-5ee9ade44f17-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn\" (UID: \"3496d171-a6e6-4469-bc3d-5ee9ade44f17\") " pod="openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn" Sep 11 16:58:54.998864 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:54.998845 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3496d171-a6e6-4469-bc3d-5ee9ade44f17-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn\" (UID: \"3496d171-a6e6-4469-bc3d-5ee9ade44f17\") " pod="openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn" Sep 11 16:58:55.006841 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:55.006812 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwsfk\" (UniqueName: \"kubernetes.io/projected/3496d171-a6e6-4469-bc3d-5ee9ade44f17-kube-api-access-mwsfk\") pod \"payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn\" (UID: \"3496d171-a6e6-4469-bc3d-5ee9ade44f17\") " pod="openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn" Sep 11 16:58:55.141899 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:55.141814 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn" Sep 11 16:58:55.425425 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:55.425397 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn"] Sep 11 16:58:55.427693 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:58:55.427662 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3496d171_a6e6_4469_bc3d_5ee9ade44f17.slice/crio-82d2e1189fff760b7b7fc6325b62e6c12767b607c820af43efed260effb2d342 WatchSource:0}: Error finding container 82d2e1189fff760b7b7fc6325b62e6c12767b607c820af43efed260effb2d342: Status 404 returned error can't find the container with id 82d2e1189fff760b7b7fc6325b62e6c12767b607c820af43efed260effb2d342 Sep 11 16:58:55.713374 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:55.713278 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn" event={"ID":"3496d171-a6e6-4469-bc3d-5ee9ade44f17","Type":"ContainerStarted","Data":"cca6136c34befcd3b0744db29251977f7ab4887011b7d09321734a8f9089294b"} Sep 11 16:58:55.713374 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:55.713343 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn" event={"ID":"3496d171-a6e6-4469-bc3d-5ee9ade44f17","Type":"ContainerStarted","Data":"82d2e1189fff760b7b7fc6325b62e6c12767b607c820af43efed260effb2d342"} Sep 11 16:58:55.713610 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:55.713382 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn" Sep 11 16:58:55.738300 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:58:55.738242 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn" podStartSLOduration=1.738228244 podStartE2EDuration="1.738228244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:58:55.737944764 +0000 UTC m=+951.146047053" watchObservedRunningTime="2026-09-11 16:58:55.738228244 +0000 UTC m=+951.146330523" Sep 11 16:59:06.721326 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:06.721294 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn" Sep 11 16:59:40.072805 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.072770 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669"] Sep 11 16:59:40.076187 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.076164 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.080437 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.080414 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-32d680\"/\"kube-root-ca.crt\"" Sep 11 16:59:40.080585 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.080433 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-32d680\"/\"openshift-service-ca.crt\"" Sep 11 16:59:40.080585 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.080568 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-32d680\"/\"test-model-32d680-kserve-self-signed-certs\"" Sep 11 16:59:40.080704 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.080664 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-32d680\"/\"default-dockercfg-227n8\"" Sep 11 16:59:40.086794 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.086773 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669"] Sep 11 16:59:40.130553 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.130479 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-tmp-dir\") pod \"test-model-32d680-kserve-7b68cf6899-ck669\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.130553 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.130555 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fa03b19-98f8-44e6-b696-d29613305934-tls-certs\") pod \"test-model-32d680-kserve-7b68cf6899-ck669\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.130788 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.130596 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tg58\" (UniqueName: \"kubernetes.io/projected/2fa03b19-98f8-44e6-b696-d29613305934-kube-api-access-4tg58\") pod \"test-model-32d680-kserve-7b68cf6899-ck669\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.130788 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.130632 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-model-cache\") pod \"test-model-32d680-kserve-7b68cf6899-ck669\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.130788 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.130669 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-home\") pod \"test-model-32d680-kserve-7b68cf6899-ck669\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.130788 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.130733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-dshm\") pod \"test-model-32d680-kserve-7b68cf6899-ck669\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.231204 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.231169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tg58\" (UniqueName: \"kubernetes.io/projected/2fa03b19-98f8-44e6-b696-d29613305934-kube-api-access-4tg58\") pod \"test-model-32d680-kserve-7b68cf6899-ck669\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.231381 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.231221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-model-cache\") pod \"test-model-32d680-kserve-7b68cf6899-ck669\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.231381 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.231275 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-home\") pod \"test-model-32d680-kserve-7b68cf6899-ck669\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.231457 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.231408 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-dshm\") pod \"test-model-32d680-kserve-7b68cf6899-ck669\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.231542 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.231475 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-tmp-dir\") pod \"test-model-32d680-kserve-7b68cf6899-ck669\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.231542 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.231522 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fa03b19-98f8-44e6-b696-d29613305934-tls-certs\") pod \"test-model-32d680-kserve-7b68cf6899-ck669\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.231735 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.231716 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-home\") pod \"test-model-32d680-kserve-7b68cf6899-ck669\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.231809 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.231774 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-model-cache\") pod \"test-model-32d680-kserve-7b68cf6899-ck669\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.231866 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.231827 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-tmp-dir\") pod \"test-model-32d680-kserve-7b68cf6899-ck669\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.233728 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.233703 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-dshm\") pod \"test-model-32d680-kserve-7b68cf6899-ck669\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.234011 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.233992 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fa03b19-98f8-44e6-b696-d29613305934-tls-certs\") pod \"test-model-32d680-kserve-7b68cf6899-ck669\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.239723 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.239697 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tg58\" (UniqueName: \"kubernetes.io/projected/2fa03b19-98f8-44e6-b696-d29613305934-kube-api-access-4tg58\") pod \"test-model-32d680-kserve-7b68cf6899-ck669\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.386562 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.386439 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:40.664710 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.664684 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669"] Sep 11 16:59:40.666613 ip-10-0-143-165 kubenswrapper[2577]: W0911 16:59:40.666585 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fa03b19_98f8_44e6_b696_d29613305934.slice/crio-5c7146807a5d9c98238b01100760b7baad58cf5875cc57b77ad123e17e4bda11 WatchSource:0}: Error finding container 5c7146807a5d9c98238b01100760b7baad58cf5875cc57b77ad123e17e4bda11: Status 404 returned error can't find the container with id 5c7146807a5d9c98238b01100760b7baad58cf5875cc57b77ad123e17e4bda11 Sep 11 16:59:40.965804 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.965707 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" event={"ID":"2fa03b19-98f8-44e6-b696-d29613305934","Type":"ContainerStarted","Data":"32bdb74eca1464bfa8c891459ce4cc494486b8d7ea75e63b377a61f906aa4df3"} Sep 11 16:59:40.965804 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.965749 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" event={"ID":"2fa03b19-98f8-44e6-b696-d29613305934","Type":"ContainerStarted","Data":"5c7146807a5d9c98238b01100760b7baad58cf5875cc57b77ad123e17e4bda11"} Sep 11 16:59:40.985953 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:40.985901 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" podStartSLOduration=0.985885199 podStartE2EDuration="985.885199ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:59:40.983779485 +0000 UTC m=+996.391881775" watchObservedRunningTime="2026-09-11 16:59:40.985885199 +0000 UTC m=+996.393987468" Sep 11 16:59:50.386905 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:50.386866 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:50.387418 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:50.386955 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:50.399730 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:50.399698 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 16:59:51.028106 ip-10-0-143-165 kubenswrapper[2577]: I0911 16:59:51.028070 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 17:00:00.160131 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:00.160099 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819100-6g6h5"] Sep 11 17:00:00.167200 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:00.167176 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32d680-29819100-46zn2"] Sep 11 17:00:00.167353 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:00.167291 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819100-6g6h5" Sep 11 17:00:00.170324 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:00.170304 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2cceea-29819100-6g7hk"] Sep 11 17:00:00.170459 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:00.170439 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32d680-29819100-46zn2" Sep 11 17:00:00.170791 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:00.170775 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pc4qb\"" Sep 11 17:00:00.173695 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:00.173646 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2cceea-29819100-6g7hk" Sep 11 17:00:00.176359 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:00.176336 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819100-6g6h5"] Sep 11 17:00:00.183886 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:00.183863 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819100-6g6h5" Sep 11 17:00:00.185114 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:00.185093 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32d680-29819100-46zn2"] Sep 11 17:00:00.193174 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:00.193154 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32d680-29819100-46zn2" Sep 11 17:00:00.197850 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:00.197413 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2cceea-29819100-6g7hk"] Sep 11 17:00:00.235380 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:00.235357 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2cceea-29819100-6g7hk" Sep 11 17:00:00.480917 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:00.480889 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819100-6g6h5"] Sep 11 17:00:00.482687 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:00:00.482657 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac48fc33_c994_4f55_99cd_9cd5d96352a0.slice/crio-92781433fc446a6670e7178070054df87dd841974fe627fba16a50eb8577e7bc WatchSource:0}: Error finding container 92781433fc446a6670e7178070054df87dd841974fe627fba16a50eb8577e7bc: Status 404 returned error can't find the container with id 92781433fc446a6670e7178070054df87dd841974fe627fba16a50eb8577e7bc Sep 11 17:00:00.488758 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:00.488663 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32d680-29819100-46zn2"] Sep 11 17:00:00.491260 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:00:00.491237 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeba4068_9144_4fca_b299_92f0d5b63ac2.slice/crio-c79790a1526ae51c4bb682ee365d1a182c76a62f7ba0123498df994876faad24 WatchSource:0}: Error finding container c79790a1526ae51c4bb682ee365d1a182c76a62f7ba0123498df994876faad24: Status 404 returned error can't find the container with id c79790a1526ae51c4bb682ee365d1a182c76a62f7ba0123498df994876faad24 Sep 11 17:00:00.529057 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:00.529033 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2cceea-29819100-6g7hk"] Sep 11 17:00:00.531241 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:00:00.531219 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76369e3_0326_4873_aab9_a48291ea079a.slice/crio-2e21778eaa5970f8a979b661a1b83e57b7c68386a5619fcca4269cb2d422ec39 WatchSource:0}: Error finding container 2e21778eaa5970f8a979b661a1b83e57b7c68386a5619fcca4269cb2d422ec39: Status 404 returned error can't find the container with id 2e21778eaa5970f8a979b661a1b83e57b7c68386a5619fcca4269cb2d422ec39 Sep 11 17:00:01.071132 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:01.071037 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819100-6g6h5" event={"ID":"ac48fc33-c994-4f55-99cd-9cd5d96352a0","Type":"ContainerStarted","Data":"1c3baac24afa76fd1ea2026935219d9c48eaeecd3cab15c03284609a95279d17"} Sep 11 17:00:01.071132 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:01.071094 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819100-6g6h5" event={"ID":"ac48fc33-c994-4f55-99cd-9cd5d96352a0","Type":"ContainerStarted","Data":"92781433fc446a6670e7178070054df87dd841974fe627fba16a50eb8577e7bc"} Sep 11 17:00:01.072646 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:01.072619 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2cceea-29819100-6g7hk" event={"ID":"b76369e3-0326-4873-aab9-a48291ea079a","Type":"ContainerStarted","Data":"d64001afd40dc4d36ee1585cadf3a681eb842e94777703c921662bc1eeb28ab7"} Sep 11 17:00:01.072787 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:01.072671 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2cceea-29819100-6g7hk" event={"ID":"b76369e3-0326-4873-aab9-a48291ea079a","Type":"ContainerStarted","Data":"2e21778eaa5970f8a979b661a1b83e57b7c68386a5619fcca4269cb2d422ec39"} Sep 11 17:00:01.074043 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:01.074016 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32d680-29819100-46zn2" event={"ID":"eeba4068-9144-4fca-b299-92f0d5b63ac2","Type":"ContainerStarted","Data":"9c0df5e499f74c8c7d7c4daeaeaac3ae18da72b980d4a70b70ac54d3d37837ee"} Sep 11 17:00:01.074184 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:01.074064 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32d680-29819100-46zn2" event={"ID":"eeba4068-9144-4fca-b299-92f0d5b63ac2","Type":"ContainerStarted","Data":"c79790a1526ae51c4bb682ee365d1a182c76a62f7ba0123498df994876faad24"} Sep 11 17:00:01.089565 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:01.089516 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819100-6g6h5" podStartSLOduration=1.089483197 podStartE2EDuration="1.089483197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:00:01.088704269 +0000 UTC m=+1016.496806538" watchObservedRunningTime="2026-09-11 17:00:01.089483197 +0000 UTC m=+1016.497585465" Sep 11 17:00:01.106515 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:01.106454 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32d680-29819100-46zn2" podStartSLOduration=1.106441268 podStartE2EDuration="1.106441268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:00:01.104588942 +0000 UTC m=+1016.512691233" watchObservedRunningTime="2026-09-11 17:00:01.106441268 +0000 UTC m=+1016.514543537" Sep 11 17:00:01.122206 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:01.122141 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2cceea-29819100-6g7hk" podStartSLOduration=1.1221207 podStartE2EDuration="1.1221207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:00:01.121873813 +0000 UTC m=+1016.529976109" watchObservedRunningTime="2026-09-11 17:00:01.1221207 +0000 UTC m=+1016.530222972" Sep 11 17:00:02.080643 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:02.080605 2577 generic.go:356] "Generic (PLEG): container finished" podID="b76369e3-0326-4873-aab9-a48291ea079a" containerID="d64001afd40dc4d36ee1585cadf3a681eb842e94777703c921662bc1eeb28ab7" exitCode=0 Sep 11 17:00:02.081057 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:02.080696 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2cceea-29819100-6g7hk" event={"ID":"b76369e3-0326-4873-aab9-a48291ea079a","Type":"ContainerDied","Data":"d64001afd40dc4d36ee1585cadf3a681eb842e94777703c921662bc1eeb28ab7"} Sep 11 17:00:02.082314 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:02.082292 2577 generic.go:356] "Generic (PLEG): container finished" podID="eeba4068-9144-4fca-b299-92f0d5b63ac2" containerID="9c0df5e499f74c8c7d7c4daeaeaac3ae18da72b980d4a70b70ac54d3d37837ee" exitCode=0 Sep 11 17:00:02.082443 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:02.082424 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32d680-29819100-46zn2" event={"ID":"eeba4068-9144-4fca-b299-92f0d5b63ac2","Type":"ContainerDied","Data":"9c0df5e499f74c8c7d7c4daeaeaac3ae18da72b980d4a70b70ac54d3d37837ee"} Sep 11 17:00:02.083975 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:02.083955 2577 generic.go:356] "Generic (PLEG): container finished" podID="ac48fc33-c994-4f55-99cd-9cd5d96352a0" containerID="1c3baac24afa76fd1ea2026935219d9c48eaeecd3cab15c03284609a95279d17" exitCode=0 Sep 11 17:00:02.084064 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:02.083985 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819100-6g6h5" event={"ID":"ac48fc33-c994-4f55-99cd-9cd5d96352a0","Type":"ContainerDied","Data":"1c3baac24afa76fd1ea2026935219d9c48eaeecd3cab15c03284609a95279d17"} Sep 11 17:00:03.237169 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:03.237135 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32d680-29819100-46zn2" Sep 11 17:00:03.259193 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:03.258342 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819100-6g6h5" Sep 11 17:00:03.286661 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:03.286630 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2cceea-29819100-6g7hk" Sep 11 17:00:04.095902 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:04.095864 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819100-6g6h5" Sep 11 17:00:04.096103 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:04.095917 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819100-6g6h5" event={"ID":"ac48fc33-c994-4f55-99cd-9cd5d96352a0","Type":"ContainerDied","Data":"92781433fc446a6670e7178070054df87dd841974fe627fba16a50eb8577e7bc"} Sep 11 17:00:04.096103 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:04.095969 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92781433fc446a6670e7178070054df87dd841974fe627fba16a50eb8577e7bc" Sep 11 17:00:04.097489 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:04.097449 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2cceea-29819100-6g7hk" Sep 11 17:00:04.097675 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:04.097443 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2cceea-29819100-6g7hk" event={"ID":"b76369e3-0326-4873-aab9-a48291ea079a","Type":"ContainerDied","Data":"2e21778eaa5970f8a979b661a1b83e57b7c68386a5619fcca4269cb2d422ec39"} Sep 11 17:00:04.097675 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:04.097564 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e21778eaa5970f8a979b661a1b83e57b7c68386a5619fcca4269cb2d422ec39" Sep 11 17:00:04.099009 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:04.098985 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32d680-29819100-46zn2" event={"ID":"eeba4068-9144-4fca-b299-92f0d5b63ac2","Type":"ContainerDied","Data":"c79790a1526ae51c4bb682ee365d1a182c76a62f7ba0123498df994876faad24"} Sep 11 17:00:04.099138 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:04.099021 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c79790a1526ae51c4bb682ee365d1a182c76a62f7ba0123498df994876faad24" Sep 11 17:00:04.099138 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:04.099025 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32d680-29819100-46zn2" Sep 11 17:00:16.008774 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.008737 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp"] Sep 11 17:00:16.009162 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.009148 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac48fc33-c994-4f55-99cd-9cd5d96352a0" containerName="cleanup" Sep 11 17:00:16.009162 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.009160 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac48fc33-c994-4f55-99cd-9cd5d96352a0" containerName="cleanup" Sep 11 17:00:16.009247 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.009180 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeba4068-9144-4fca-b299-92f0d5b63ac2" containerName="cleanup" Sep 11 17:00:16.009247 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.009185 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeba4068-9144-4fca-b299-92f0d5b63ac2" containerName="cleanup" Sep 11 17:00:16.009247 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.009192 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76369e3-0326-4873-aab9-a48291ea079a" containerName="cleanup" Sep 11 17:00:16.009247 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.009197 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76369e3-0326-4873-aab9-a48291ea079a" containerName="cleanup" Sep 11 17:00:16.009406 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.009264 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeba4068-9144-4fca-b299-92f0d5b63ac2" containerName="cleanup" Sep 11 17:00:16.009406 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.009271 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76369e3-0326-4873-aab9-a48291ea079a" containerName="cleanup" Sep 11 17:00:16.009406 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.009278 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac48fc33-c994-4f55-99cd-9cd5d96352a0" containerName="cleanup" Sep 11 17:00:16.012406 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.012389 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp" Sep 11 17:00:16.015329 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.015306 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-31e12f72\"" Sep 11 17:00:16.025254 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.025230 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp"] Sep 11 17:00:16.083630 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.083595 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/938cf641-5a9b-4772-b16d-6991b27ea5b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp\" (UID: \"938cf641-5a9b-4772-b16d-6991b27ea5b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp" Sep 11 17:00:16.083804 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.083643 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmjx4\" (UniqueName: \"kubernetes.io/projected/938cf641-5a9b-4772-b16d-6991b27ea5b4-kube-api-access-bmjx4\") pod \"payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp\" (UID: \"938cf641-5a9b-4772-b16d-6991b27ea5b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp" Sep 11 17:00:16.185865 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.185087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmjx4\" (UniqueName: \"kubernetes.io/projected/938cf641-5a9b-4772-b16d-6991b27ea5b4-kube-api-access-bmjx4\") pod \"payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp\" (UID: \"938cf641-5a9b-4772-b16d-6991b27ea5b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp" Sep 11 17:00:16.185865 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.185280 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/938cf641-5a9b-4772-b16d-6991b27ea5b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp\" (UID: \"938cf641-5a9b-4772-b16d-6991b27ea5b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp" Sep 11 17:00:16.186351 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.186325 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/938cf641-5a9b-4772-b16d-6991b27ea5b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp\" (UID: \"938cf641-5a9b-4772-b16d-6991b27ea5b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp" Sep 11 17:00:16.200365 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.200344 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmjx4\" (UniqueName: \"kubernetes.io/projected/938cf641-5a9b-4772-b16d-6991b27ea5b4-kube-api-access-bmjx4\") pod \"payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp\" (UID: \"938cf641-5a9b-4772-b16d-6991b27ea5b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp" Sep 11 17:00:16.324868 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.324776 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp" Sep 11 17:00:16.602752 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:16.602720 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp"] Sep 11 17:00:16.605277 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:00:16.605250 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938cf641_5a9b_4772_b16d_6991b27ea5b4.slice/crio-3ad002328a693ef1964274a698b2f0f404db57e581a36118eb81667fd4093ebd WatchSource:0}: Error finding container 3ad002328a693ef1964274a698b2f0f404db57e581a36118eb81667fd4093ebd: Status 404 returned error can't find the container with id 3ad002328a693ef1964274a698b2f0f404db57e581a36118eb81667fd4093ebd Sep 11 17:00:17.169586 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:17.169543 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp" event={"ID":"938cf641-5a9b-4772-b16d-6991b27ea5b4","Type":"ContainerStarted","Data":"b4bd1dd66bf3382095742f544cf745c484884f7e3a7b479a6d2a618cec007aca"} Sep 11 17:00:17.169991 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:17.169706 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp" Sep 11 17:00:17.169991 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:17.169748 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp" event={"ID":"938cf641-5a9b-4772-b16d-6991b27ea5b4","Type":"ContainerStarted","Data":"3ad002328a693ef1964274a698b2f0f404db57e581a36118eb81667fd4093ebd"} Sep 11 17:00:17.202289 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:17.202237 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp" podStartSLOduration=2.202219694 podStartE2EDuration="2.202219694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:00:17.201872697 +0000 UTC m=+1032.609974987" watchObservedRunningTime="2026-09-11 17:00:17.202219694 +0000 UTC m=+1032.610321963" Sep 11 17:00:26.778856 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.778816 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz"] Sep 11 17:00:26.782506 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.782480 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:26.787037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.787012 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-2cceea\"/\"default-dockercfg-rrfb6\"" Sep 11 17:00:26.787037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.787012 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-2cceea\"/\"kube-root-ca.crt\"" Sep 11 17:00:26.787400 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.787377 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-2cceea\"/\"explicit-model-2cceea-kserve-self-signed-certs\"" Sep 11 17:00:26.787547 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.787466 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-2cceea\"/\"openshift-service-ca.crt\"" Sep 11 17:00:26.793096 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.793067 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz"] Sep 11 17:00:26.893751 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.893716 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7942g\" (UniqueName: \"kubernetes.io/projected/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-kube-api-access-7942g\") pod \"explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:26.893944 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.893768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-model-cache\") pod \"explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:26.893944 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.893792 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-tls-certs\") pod \"explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:26.893944 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.893845 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-dshm\") pod \"explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:26.893944 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.893881 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-tmp-dir\") pod \"explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:26.894109 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.893964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-home\") pod \"explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:26.994998 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.994961 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-tmp-dir\") pod \"explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:26.995195 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.995030 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-home\") pod \"explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:26.995195 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.995067 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7942g\" (UniqueName: \"kubernetes.io/projected/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-kube-api-access-7942g\") pod \"explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:26.995195 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.995092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-model-cache\") pod \"explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:26.995195 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.995107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-tls-certs\") pod \"explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:26.995195 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.995147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-dshm\") pod \"explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:26.995477 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.995454 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-tmp-dir\") pod \"explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:26.995554 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.995486 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-model-cache\") pod \"explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:26.995617 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.995583 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-home\") pod \"explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:26.997423 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.997402 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-dshm\") pod \"explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:26.997711 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:26.997693 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-tls-certs\") pod \"explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:27.004484 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:27.004456 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7942g\" (UniqueName: \"kubernetes.io/projected/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-kube-api-access-7942g\") pod \"explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:27.092591 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:27.092484 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:27.372618 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:27.372581 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz"] Sep 11 17:00:27.375004 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:00:27.374970 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ff83b32_1ad6_4e90_b268_9f5d81c985fb.slice/crio-606140a2d8d89d1d2cfecb4ba912c36d36676cf8a7926a1f057de1c476f9dec6 WatchSource:0}: Error finding container 606140a2d8d89d1d2cfecb4ba912c36d36676cf8a7926a1f057de1c476f9dec6: Status 404 returned error can't find the container with id 606140a2d8d89d1d2cfecb4ba912c36d36676cf8a7926a1f057de1c476f9dec6 Sep 11 17:00:28.177795 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:28.177763 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp" Sep 11 17:00:28.228539 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:28.228487 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" event={"ID":"5ff83b32-1ad6-4e90-b268-9f5d81c985fb","Type":"ContainerStarted","Data":"85204fe19f12930a03e4fd7851d8f2dc6f489df07af966c191c52a07aff53e42"} Sep 11 17:00:28.228697 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:28.228545 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" event={"ID":"5ff83b32-1ad6-4e90-b268-9f5d81c985fb","Type":"ContainerStarted","Data":"606140a2d8d89d1d2cfecb4ba912c36d36676cf8a7926a1f057de1c476f9dec6"} Sep 11 17:00:28.249563 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:28.249492 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" podStartSLOduration=2.249467815 podStartE2EDuration="2.249467815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:00:28.248346435 +0000 UTC m=+1043.656448785" watchObservedRunningTime="2026-09-11 17:00:28.249467815 +0000 UTC m=+1043.657570084" Sep 11 17:00:37.094359 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:37.094320 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:37.094359 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:37.094364 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:37.105813 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:37.105784 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:37.285041 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:37.285014 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:00:49.139640 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:49.139598 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31e12f72-jqrml"] Sep 11 17:00:49.143384 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:49.143365 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31e12f72-jqrml" Sep 11 17:00:49.146252 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:49.146227 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 17:00:49.146367 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:49.146261 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pc4qb\"" Sep 11 17:00:49.152410 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:49.152381 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31e12f72-jqrml"] Sep 11 17:00:49.200211 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:49.200162 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d446655b-6023-4296-b079-7f103f99d120-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-31e12f72-jqrml\" (UID: \"d446655b-6023-4296-b079-7f103f99d120\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31e12f72-jqrml" Sep 11 17:00:49.300922 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:49.300880 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d446655b-6023-4296-b079-7f103f99d120-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-31e12f72-jqrml\" (UID: \"d446655b-6023-4296-b079-7f103f99d120\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31e12f72-jqrml" Sep 11 17:00:49.301532 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:49.301472 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d446655b-6023-4296-b079-7f103f99d120-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-31e12f72-jqrml\" (UID: \"d446655b-6023-4296-b079-7f103f99d120\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31e12f72-jqrml" Sep 11 17:00:49.455917 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:49.455819 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31e12f72-jqrml" Sep 11 17:00:49.726737 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:49.726696 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31e12f72-jqrml"] Sep 11 17:00:49.727604 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:00:49.727564 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd446655b_6023_4296_b079_7f103f99d120.slice/crio-adff1d7ec12ecc3307a41406281645d1de67fc4b1774ce25771e0c12b462293a WatchSource:0}: Error finding container adff1d7ec12ecc3307a41406281645d1de67fc4b1774ce25771e0c12b462293a: Status 404 returned error can't find the container with id adff1d7ec12ecc3307a41406281645d1de67fc4b1774ce25771e0c12b462293a Sep 11 17:00:50.345544 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:50.345515 2577 generic.go:356] "Generic (PLEG): container finished" podID="d446655b-6023-4296-b079-7f103f99d120" containerID="5ea3e3a15c0ab4206361ed32f3509a19ea23a21d91f7c8d816db3b6b1a5dd772" exitCode=0 Sep 11 17:00:50.345876 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:50.345604 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31e12f72-jqrml" event={"ID":"d446655b-6023-4296-b079-7f103f99d120","Type":"ContainerDied","Data":"5ea3e3a15c0ab4206361ed32f3509a19ea23a21d91f7c8d816db3b6b1a5dd772"} Sep 11 17:00:50.345876 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:50.345653 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31e12f72-jqrml" event={"ID":"d446655b-6023-4296-b079-7f103f99d120","Type":"ContainerStarted","Data":"adff1d7ec12ecc3307a41406281645d1de67fc4b1774ce25771e0c12b462293a"} Sep 11 17:00:50.566990 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:50.566948 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz"] Sep 11 17:00:51.479047 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:51.479019 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31e12f72-jqrml" Sep 11 17:00:51.522773 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:51.522730 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d446655b-6023-4296-b079-7f103f99d120-maas-api-service-ca\") pod \"d446655b-6023-4296-b079-7f103f99d120\" (UID: \"d446655b-6023-4296-b079-7f103f99d120\") " Sep 11 17:00:51.523177 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:51.523149 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d446655b-6023-4296-b079-7f103f99d120-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d446655b-6023-4296-b079-7f103f99d120" (UID: "d446655b-6023-4296-b079-7f103f99d120"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:00:51.623892 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:51.623803 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d446655b-6023-4296-b079-7f103f99d120-maas-api-service-ca\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:00:52.357595 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:52.357562 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31e12f72-jqrml" Sep 11 17:00:52.357780 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:52.357562 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31e12f72-jqrml" event={"ID":"d446655b-6023-4296-b079-7f103f99d120","Type":"ContainerDied","Data":"adff1d7ec12ecc3307a41406281645d1de67fc4b1774ce25771e0c12b462293a"} Sep 11 17:00:52.357780 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:52.357691 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adff1d7ec12ecc3307a41406281645d1de67fc4b1774ce25771e0c12b462293a" Sep 11 17:00:59.941593 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:59.941517 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp"] Sep 11 17:00:59.942091 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:00:59.941758 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp" podUID="938cf641-5a9b-4772-b16d-6991b27ea5b4" containerName="payload-pre-processing" containerID="cri-o://b4bd1dd66bf3382095742f544cf745c484884f7e3a7b479a6d2a618cec007aca" gracePeriod=30 Sep 11 17:01:00.121523 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:00.121474 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31e12f72-jqrml"] Sep 11 17:01:00.125223 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:00.125201 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31e12f72-jqrml"] Sep 11 17:01:00.214703 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:00.214676 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp" Sep 11 17:01:00.316329 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:00.316293 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/938cf641-5a9b-4772-b16d-6991b27ea5b4-plugins-config-volume\") pod \"938cf641-5a9b-4772-b16d-6991b27ea5b4\" (UID: \"938cf641-5a9b-4772-b16d-6991b27ea5b4\") " Sep 11 17:01:00.316513 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:00.316359 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmjx4\" (UniqueName: \"kubernetes.io/projected/938cf641-5a9b-4772-b16d-6991b27ea5b4-kube-api-access-bmjx4\") pod \"938cf641-5a9b-4772-b16d-6991b27ea5b4\" (UID: \"938cf641-5a9b-4772-b16d-6991b27ea5b4\") " Sep 11 17:01:00.316788 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:00.316759 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938cf641-5a9b-4772-b16d-6991b27ea5b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "938cf641-5a9b-4772-b16d-6991b27ea5b4" (UID: "938cf641-5a9b-4772-b16d-6991b27ea5b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:01:00.318646 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:00.318612 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938cf641-5a9b-4772-b16d-6991b27ea5b4-kube-api-access-bmjx4" (OuterVolumeSpecName: "kube-api-access-bmjx4") pod "938cf641-5a9b-4772-b16d-6991b27ea5b4" (UID: "938cf641-5a9b-4772-b16d-6991b27ea5b4"). InnerVolumeSpecName "kube-api-access-bmjx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:01:00.404200 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:00.404166 2577 generic.go:356] "Generic (PLEG): container finished" podID="938cf641-5a9b-4772-b16d-6991b27ea5b4" containerID="b4bd1dd66bf3382095742f544cf745c484884f7e3a7b479a6d2a618cec007aca" exitCode=0 Sep 11 17:01:00.404373 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:00.404216 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp" Sep 11 17:01:00.404373 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:00.404217 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp" event={"ID":"938cf641-5a9b-4772-b16d-6991b27ea5b4","Type":"ContainerDied","Data":"b4bd1dd66bf3382095742f544cf745c484884f7e3a7b479a6d2a618cec007aca"} Sep 11 17:01:00.404373 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:00.404347 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp" event={"ID":"938cf641-5a9b-4772-b16d-6991b27ea5b4","Type":"ContainerDied","Data":"3ad002328a693ef1964274a698b2f0f404db57e581a36118eb81667fd4093ebd"} Sep 11 17:01:00.404539 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:00.404383 2577 scope.go:117] "RemoveContainer" containerID="b4bd1dd66bf3382095742f544cf745c484884f7e3a7b479a6d2a618cec007aca" Sep 11 17:01:00.413879 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:00.413856 2577 scope.go:117] "RemoveContainer" containerID="b4bd1dd66bf3382095742f544cf745c484884f7e3a7b479a6d2a618cec007aca" Sep 11 17:01:00.414151 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:01:00.414133 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4bd1dd66bf3382095742f544cf745c484884f7e3a7b479a6d2a618cec007aca\": container with ID starting with b4bd1dd66bf3382095742f544cf745c484884f7e3a7b479a6d2a618cec007aca not found: ID does not exist" containerID="b4bd1dd66bf3382095742f544cf745c484884f7e3a7b479a6d2a618cec007aca" Sep 11 17:01:00.414196 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:00.414161 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4bd1dd66bf3382095742f544cf745c484884f7e3a7b479a6d2a618cec007aca"} err="failed to get container status \"b4bd1dd66bf3382095742f544cf745c484884f7e3a7b479a6d2a618cec007aca\": rpc error: code = NotFound desc = could not find container \"b4bd1dd66bf3382095742f544cf745c484884f7e3a7b479a6d2a618cec007aca\": container with ID starting with b4bd1dd66bf3382095742f544cf745c484884f7e3a7b479a6d2a618cec007aca not found: ID does not exist" Sep 11 17:01:00.417140 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:00.417120 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/938cf641-5a9b-4772-b16d-6991b27ea5b4-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:01:00.417227 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:00.417142 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmjx4\" (UniqueName: \"kubernetes.io/projected/938cf641-5a9b-4772-b16d-6991b27ea5b4-kube-api-access-bmjx4\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:01:00.426181 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:00.426154 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp"] Sep 11 17:01:00.432052 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:00.432028 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-31e12f72-5d7bd7bfdb-dvbcp"] Sep 11 17:01:01.095765 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:01.095733 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="938cf641-5a9b-4772-b16d-6991b27ea5b4" path="/var/lib/kubelet/pods/938cf641-5a9b-4772-b16d-6991b27ea5b4/volumes" Sep 11 17:01:01.096178 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:01.096084 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d446655b-6023-4296-b079-7f103f99d120" path="/var/lib/kubelet/pods/d446655b-6023-4296-b079-7f103f99d120/volumes" Sep 11 17:01:01.156338 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:01.156301 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2cceea-29819100-6g7hk"] Sep 11 17:01:01.159329 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:01.159303 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-2cceea-29819100-6g7hk"] Sep 11 17:01:01.184079 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:01.184038 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn"] Sep 11 17:01:01.184370 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:01.184338 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn" podUID="3496d171-a6e6-4469-bc3d-5ee9ade44f17" containerName="payload-processing" containerID="cri-o://cca6136c34befcd3b0744db29251977f7ab4887011b7d09321734a8f9089294b" gracePeriod=30 Sep 11 17:01:01.421147 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:01.421104 2577 generic.go:356] "Generic (PLEG): container finished" podID="3496d171-a6e6-4469-bc3d-5ee9ade44f17" containerID="cca6136c34befcd3b0744db29251977f7ab4887011b7d09321734a8f9089294b" exitCode=0 Sep 11 17:01:01.421343 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:01.421282 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn" event={"ID":"3496d171-a6e6-4469-bc3d-5ee9ade44f17","Type":"ContainerDied","Data":"cca6136c34befcd3b0744db29251977f7ab4887011b7d09321734a8f9089294b"} Sep 11 17:01:01.465294 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:01.465256 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn" Sep 11 17:01:01.631303 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:01.631224 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwsfk\" (UniqueName: \"kubernetes.io/projected/3496d171-a6e6-4469-bc3d-5ee9ade44f17-kube-api-access-mwsfk\") pod \"3496d171-a6e6-4469-bc3d-5ee9ade44f17\" (UID: \"3496d171-a6e6-4469-bc3d-5ee9ade44f17\") " Sep 11 17:01:01.631303 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:01.631294 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3496d171-a6e6-4469-bc3d-5ee9ade44f17-plugins-config-volume\") pod \"3496d171-a6e6-4469-bc3d-5ee9ade44f17\" (UID: \"3496d171-a6e6-4469-bc3d-5ee9ade44f17\") " Sep 11 17:01:01.631709 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:01.631683 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3496d171-a6e6-4469-bc3d-5ee9ade44f17-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3496d171-a6e6-4469-bc3d-5ee9ade44f17" (UID: "3496d171-a6e6-4469-bc3d-5ee9ade44f17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:01:01.633443 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:01.633413 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3496d171-a6e6-4469-bc3d-5ee9ade44f17-kube-api-access-mwsfk" (OuterVolumeSpecName: "kube-api-access-mwsfk") pod "3496d171-a6e6-4469-bc3d-5ee9ade44f17" (UID: "3496d171-a6e6-4469-bc3d-5ee9ade44f17"). InnerVolumeSpecName "kube-api-access-mwsfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:01:01.732694 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:01.732655 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3496d171-a6e6-4469-bc3d-5ee9ade44f17-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:01:01.732694 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:01.732688 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwsfk\" (UniqueName: \"kubernetes.io/projected/3496d171-a6e6-4469-bc3d-5ee9ade44f17-kube-api-access-mwsfk\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:01:02.431470 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:02.431427 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn" event={"ID":"3496d171-a6e6-4469-bc3d-5ee9ade44f17","Type":"ContainerDied","Data":"82d2e1189fff760b7b7fc6325b62e6c12767b607c820af43efed260effb2d342"} Sep 11 17:01:02.431470 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:02.431463 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn" Sep 11 17:01:02.431980 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:02.431541 2577 scope.go:117] "RemoveContainer" containerID="cca6136c34befcd3b0744db29251977f7ab4887011b7d09321734a8f9089294b" Sep 11 17:01:02.453675 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:02.453641 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn"] Sep 11 17:01:02.460407 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:02.460380 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2cceea-7bd575c75c-289zn"] Sep 11 17:01:03.095164 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:03.095123 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3496d171-a6e6-4469-bc3d-5ee9ade44f17" path="/var/lib/kubelet/pods/3496d171-a6e6-4469-bc3d-5ee9ade44f17/volumes" Sep 11 17:01:03.095521 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:03.095480 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b76369e3-0326-4873-aab9-a48291ea079a" path="/var/lib/kubelet/pods/b76369e3-0326-4873-aab9-a48291ea079a/volumes" Sep 11 17:01:05.586327 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:05.586286 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" podUID="5ff83b32-1ad6-4e90-b268-9f5d81c985fb" containerName="main" containerID="cri-o://85204fe19f12930a03e4fd7851d8f2dc6f489df07af966c191c52a07aff53e42" gracePeriod=45 Sep 11 17:01:05.850423 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:05.850364 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:01:05.973407 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:05.973369 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7942g\" (UniqueName: \"kubernetes.io/projected/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-kube-api-access-7942g\") pod \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " Sep 11 17:01:05.973407 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:05.973409 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-dshm\") pod \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " Sep 11 17:01:05.973690 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:05.973427 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-tmp-dir\") pod \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " Sep 11 17:01:05.973690 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:05.973459 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-tls-certs\") pod \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " Sep 11 17:01:05.973690 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:05.973549 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-model-cache\") pod \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " Sep 11 17:01:05.973690 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:05.973622 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-home\") pod \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\" (UID: \"5ff83b32-1ad6-4e90-b268-9f5d81c985fb\") " Sep 11 17:01:05.973896 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:05.973767 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5ff83b32-1ad6-4e90-b268-9f5d81c985fb" (UID: "5ff83b32-1ad6-4e90-b268-9f5d81c985fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:01:05.973896 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:05.973810 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "5ff83b32-1ad6-4e90-b268-9f5d81c985fb" (UID: "5ff83b32-1ad6-4e90-b268-9f5d81c985fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:01:05.973896 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:05.973875 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-home" (OuterVolumeSpecName: "home") pod "5ff83b32-1ad6-4e90-b268-9f5d81c985fb" (UID: "5ff83b32-1ad6-4e90-b268-9f5d81c985fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:01:05.974046 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:05.974022 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-home\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:01:05.974046 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:05.974048 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-tmp-dir\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:01:05.974194 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:05.974062 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-model-cache\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:01:05.975738 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:05.975714 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-dshm" (OuterVolumeSpecName: "dshm") pod "5ff83b32-1ad6-4e90-b268-9f5d81c985fb" (UID: "5ff83b32-1ad6-4e90-b268-9f5d81c985fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:01:05.976080 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:05.976060 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5ff83b32-1ad6-4e90-b268-9f5d81c985fb" (UID: "5ff83b32-1ad6-4e90-b268-9f5d81c985fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 17:01:05.976138 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:05.976064 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-kube-api-access-7942g" (OuterVolumeSpecName: "kube-api-access-7942g") pod "5ff83b32-1ad6-4e90-b268-9f5d81c985fb" (UID: "5ff83b32-1ad6-4e90-b268-9f5d81c985fb"). InnerVolumeSpecName "kube-api-access-7942g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:01:06.074919 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:06.074878 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7942g\" (UniqueName: \"kubernetes.io/projected/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-kube-api-access-7942g\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:01:06.074919 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:06.074914 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-dshm\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:01:06.074919 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:06.074925 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff83b32-1ad6-4e90-b268-9f5d81c985fb-tls-certs\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:01:06.454772 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:06.454738 2577 generic.go:356] "Generic (PLEG): container finished" podID="5ff83b32-1ad6-4e90-b268-9f5d81c985fb" containerID="85204fe19f12930a03e4fd7851d8f2dc6f489df07af966c191c52a07aff53e42" exitCode=0 Sep 11 17:01:06.454952 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:06.454781 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" Sep 11 17:01:06.454952 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:06.454795 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" event={"ID":"5ff83b32-1ad6-4e90-b268-9f5d81c985fb","Type":"ContainerDied","Data":"85204fe19f12930a03e4fd7851d8f2dc6f489df07af966c191c52a07aff53e42"} Sep 11 17:01:06.454952 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:06.454826 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz" event={"ID":"5ff83b32-1ad6-4e90-b268-9f5d81c985fb","Type":"ContainerDied","Data":"606140a2d8d89d1d2cfecb4ba912c36d36676cf8a7926a1f057de1c476f9dec6"} Sep 11 17:01:06.454952 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:06.454843 2577 scope.go:117] "RemoveContainer" containerID="85204fe19f12930a03e4fd7851d8f2dc6f489df07af966c191c52a07aff53e42" Sep 11 17:01:06.464414 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:06.464393 2577 scope.go:117] "RemoveContainer" containerID="85204fe19f12930a03e4fd7851d8f2dc6f489df07af966c191c52a07aff53e42" Sep 11 17:01:06.464750 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:01:06.464731 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85204fe19f12930a03e4fd7851d8f2dc6f489df07af966c191c52a07aff53e42\": container with ID starting with 85204fe19f12930a03e4fd7851d8f2dc6f489df07af966c191c52a07aff53e42 not found: ID does not exist" containerID="85204fe19f12930a03e4fd7851d8f2dc6f489df07af966c191c52a07aff53e42" Sep 11 17:01:06.464816 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:06.464758 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85204fe19f12930a03e4fd7851d8f2dc6f489df07af966c191c52a07aff53e42"} err="failed to get container status \"85204fe19f12930a03e4fd7851d8f2dc6f489df07af966c191c52a07aff53e42\": rpc error: code = NotFound desc = could not find container \"85204fe19f12930a03e4fd7851d8f2dc6f489df07af966c191c52a07aff53e42\": container with ID starting with 85204fe19f12930a03e4fd7851d8f2dc6f489df07af966c191c52a07aff53e42 not found: ID does not exist" Sep 11 17:01:06.476166 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:06.476129 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz"] Sep 11 17:01:06.478396 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:06.478368 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-2cceea/explicit-model-2cceea-kserve-6b9ccffc5f-bhjkz"] Sep 11 17:01:07.094632 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:07.094595 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ff83b32-1ad6-4e90-b268-9f5d81c985fb" path="/var/lib/kubelet/pods/5ff83b32-1ad6-4e90-b268-9f5d81c985fb/volumes" Sep 11 17:01:28.866671 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:28.866639 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk"] Sep 11 17:01:28.867074 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:28.867021 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938cf641-5a9b-4772-b16d-6991b27ea5b4" containerName="payload-pre-processing" Sep 11 17:01:28.867074 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:28.867032 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="938cf641-5a9b-4772-b16d-6991b27ea5b4" containerName="payload-pre-processing" Sep 11 17:01:28.867074 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:28.867041 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d446655b-6023-4296-b079-7f103f99d120" containerName="revoke-keys" Sep 11 17:01:28.867074 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:28.867047 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d446655b-6023-4296-b079-7f103f99d120" containerName="revoke-keys" Sep 11 17:01:28.867074 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:28.867054 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3496d171-a6e6-4469-bc3d-5ee9ade44f17" containerName="payload-processing" Sep 11 17:01:28.867074 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:28.867060 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3496d171-a6e6-4469-bc3d-5ee9ade44f17" containerName="payload-processing" Sep 11 17:01:28.867074 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:28.867075 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff83b32-1ad6-4e90-b268-9f5d81c985fb" containerName="main" Sep 11 17:01:28.867296 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:28.867081 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff83b32-1ad6-4e90-b268-9f5d81c985fb" containerName="main" Sep 11 17:01:28.867296 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:28.867145 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ff83b32-1ad6-4e90-b268-9f5d81c985fb" containerName="main" Sep 11 17:01:28.867296 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:28.867153 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3496d171-a6e6-4469-bc3d-5ee9ade44f17" containerName="payload-processing" Sep 11 17:01:28.867296 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:28.867159 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d446655b-6023-4296-b079-7f103f99d120" containerName="revoke-keys" Sep 11 17:01:28.867296 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:28.867167 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="938cf641-5a9b-4772-b16d-6991b27ea5b4" containerName="payload-pre-processing" Sep 11 17:01:28.871881 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:28.871860 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk" Sep 11 17:01:28.874442 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:28.874419 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-6b6653\"" Sep 11 17:01:28.874615 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:28.874482 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-6b6653-dockercfg-r6snq\"" Sep 11 17:01:28.881462 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:28.881436 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk"] Sep 11 17:01:28.974607 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:28.974567 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5qcv\" (UniqueName: \"kubernetes.io/projected/ebb32df2-192e-44f9-9eb0-dab17559e266-kube-api-access-t5qcv\") pod \"payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk\" (UID: \"ebb32df2-192e-44f9-9eb0-dab17559e266\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk" Sep 11 17:01:28.974766 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:28.974664 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb32df2-192e-44f9-9eb0-dab17559e266-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk\" (UID: \"ebb32df2-192e-44f9-9eb0-dab17559e266\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk" Sep 11 17:01:29.076066 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:29.076021 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5qcv\" (UniqueName: \"kubernetes.io/projected/ebb32df2-192e-44f9-9eb0-dab17559e266-kube-api-access-t5qcv\") pod \"payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk\" (UID: \"ebb32df2-192e-44f9-9eb0-dab17559e266\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk" Sep 11 17:01:29.076270 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:29.076086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb32df2-192e-44f9-9eb0-dab17559e266-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk\" (UID: \"ebb32df2-192e-44f9-9eb0-dab17559e266\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk" Sep 11 17:01:29.076822 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:29.076801 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb32df2-192e-44f9-9eb0-dab17559e266-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk\" (UID: \"ebb32df2-192e-44f9-9eb0-dab17559e266\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk" Sep 11 17:01:29.086947 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:29.086917 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5qcv\" (UniqueName: \"kubernetes.io/projected/ebb32df2-192e-44f9-9eb0-dab17559e266-kube-api-access-t5qcv\") pod \"payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk\" (UID: \"ebb32df2-192e-44f9-9eb0-dab17559e266\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk" Sep 11 17:01:29.183533 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:29.183419 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk" Sep 11 17:01:29.457476 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:29.457446 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk"] Sep 11 17:01:29.459710 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:01:29.459684 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb32df2_192e_44f9_9eb0_dab17559e266.slice/crio-988cc7033b26df57e31e790f4432b10a88f9d2937155b4ceb55a0cadcad5d60f WatchSource:0}: Error finding container 988cc7033b26df57e31e790f4432b10a88f9d2937155b4ceb55a0cadcad5d60f: Status 404 returned error can't find the container with id 988cc7033b26df57e31e790f4432b10a88f9d2937155b4ceb55a0cadcad5d60f Sep 11 17:01:29.582819 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:29.582773 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk" event={"ID":"ebb32df2-192e-44f9-9eb0-dab17559e266","Type":"ContainerStarted","Data":"7f483a149fea48cd7b2693254d424bcd2c31135ff7b7074a3d968d9103dd690e"} Sep 11 17:01:29.582976 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:29.582846 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk" Sep 11 17:01:29.582976 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:29.582871 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk" event={"ID":"ebb32df2-192e-44f9-9eb0-dab17559e266","Type":"ContainerStarted","Data":"988cc7033b26df57e31e790f4432b10a88f9d2937155b4ceb55a0cadcad5d60f"} Sep 11 17:01:29.600795 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:29.600743 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk" podStartSLOduration=1.600726464 podStartE2EDuration="1.600726464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:01:29.600291322 +0000 UTC m=+1105.008393604" watchObservedRunningTime="2026-09-11 17:01:29.600726464 +0000 UTC m=+1105.008828733" Sep 11 17:01:38.161912 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.161881 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh"] Sep 11 17:01:38.164466 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.164448 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.167287 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.167266 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-52b0af\"/\"default-dockercfg-c2dgz\"" Sep 11 17:01:38.167393 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.167291 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-52b0af\"/\"openshift-service-ca.crt\"" Sep 11 17:01:38.168831 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.168815 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-52b0af\"/\"kube-root-ca.crt\"" Sep 11 17:01:38.168898 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.168854 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-52b0af\"/\"distinct-w0-52b0af-kserve-self-signed-certs\"" Sep 11 17:01:38.174403 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.174382 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh"] Sep 11 17:01:38.260481 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.260447 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-home\") pod \"distinct-w0-52b0af-kserve-5c7c795b49-6lqfh\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.260683 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.260519 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a78ab3-31cd-4480-89fb-3507de99bb39-tls-certs\") pod \"distinct-w0-52b0af-kserve-5c7c795b49-6lqfh\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.260683 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.260553 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-tmp-dir\") pod \"distinct-w0-52b0af-kserve-5c7c795b49-6lqfh\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.260683 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.260574 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skqpt\" (UniqueName: \"kubernetes.io/projected/55a78ab3-31cd-4480-89fb-3507de99bb39-kube-api-access-skqpt\") pod \"distinct-w0-52b0af-kserve-5c7c795b49-6lqfh\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.260683 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.260626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-dshm\") pod \"distinct-w0-52b0af-kserve-5c7c795b49-6lqfh\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.260683 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.260650 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-model-cache\") pod \"distinct-w0-52b0af-kserve-5c7c795b49-6lqfh\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.361903 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.361869 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-home\") pod \"distinct-w0-52b0af-kserve-5c7c795b49-6lqfh\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.362082 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.361915 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a78ab3-31cd-4480-89fb-3507de99bb39-tls-certs\") pod \"distinct-w0-52b0af-kserve-5c7c795b49-6lqfh\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.362082 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.362037 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-tmp-dir\") pod \"distinct-w0-52b0af-kserve-5c7c795b49-6lqfh\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.362082 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.362073 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skqpt\" (UniqueName: \"kubernetes.io/projected/55a78ab3-31cd-4480-89fb-3507de99bb39-kube-api-access-skqpt\") pod \"distinct-w0-52b0af-kserve-5c7c795b49-6lqfh\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.362265 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.362119 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-dshm\") pod \"distinct-w0-52b0af-kserve-5c7c795b49-6lqfh\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.362265 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.362136 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-model-cache\") pod \"distinct-w0-52b0af-kserve-5c7c795b49-6lqfh\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.362375 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.362285 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-home\") pod \"distinct-w0-52b0af-kserve-5c7c795b49-6lqfh\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.362431 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.362378 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-tmp-dir\") pod \"distinct-w0-52b0af-kserve-5c7c795b49-6lqfh\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.362539 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.362523 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-model-cache\") pod \"distinct-w0-52b0af-kserve-5c7c795b49-6lqfh\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.364258 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.364229 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-dshm\") pod \"distinct-w0-52b0af-kserve-5c7c795b49-6lqfh\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.364528 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.364487 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a78ab3-31cd-4480-89fb-3507de99bb39-tls-certs\") pod \"distinct-w0-52b0af-kserve-5c7c795b49-6lqfh\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.370349 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.370332 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skqpt\" (UniqueName: \"kubernetes.io/projected/55a78ab3-31cd-4480-89fb-3507de99bb39-kube-api-access-skqpt\") pod \"distinct-w0-52b0af-kserve-5c7c795b49-6lqfh\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.474554 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.474433 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:38.749620 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:38.749596 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh"] Sep 11 17:01:38.751391 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:01:38.751364 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55a78ab3_31cd_4480_89fb_3507de99bb39.slice/crio-2a6dd2ab31707baf75876ea310e59c32a9f272749398b3f4a8504de490c08807 WatchSource:0}: Error finding container 2a6dd2ab31707baf75876ea310e59c32a9f272749398b3f4a8504de490c08807: Status 404 returned error can't find the container with id 2a6dd2ab31707baf75876ea310e59c32a9f272749398b3f4a8504de490c08807 Sep 11 17:01:39.636173 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:39.636132 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" event={"ID":"55a78ab3-31cd-4480-89fb-3507de99bb39","Type":"ContainerStarted","Data":"acd57c73f5f3b94b0ccb653a3faa3b3fce0fe51cb9a7fcfb9198296fd5b68966"} Sep 11 17:01:39.636560 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:39.636183 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" event={"ID":"55a78ab3-31cd-4480-89fb-3507de99bb39","Type":"ContainerStarted","Data":"2a6dd2ab31707baf75876ea310e59c32a9f272749398b3f4a8504de490c08807"} Sep 11 17:01:39.654622 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:39.654571 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" podStartSLOduration=1.654556279 podStartE2EDuration="1.654556279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:01:39.653541733 +0000 UTC m=+1115.061644003" watchObservedRunningTime="2026-09-11 17:01:39.654556279 +0000 UTC m=+1115.062658547" Sep 11 17:01:40.590648 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:40.590617 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk" Sep 11 17:01:48.475285 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:48.475248 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:48.475808 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:48.475299 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:48.488195 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:48.488166 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:48.693611 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:48.693577 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:01:56.966406 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:56.966367 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j"] Sep 11 17:01:56.969237 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:56.969221 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j" Sep 11 17:01:56.971978 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:56.971954 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-bff65f\"" Sep 11 17:01:56.972093 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:56.971995 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-bff65f-dockercfg-snn4s\"" Sep 11 17:01:56.980838 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:56.980809 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j"] Sep 11 17:01:56.998849 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:56.998815 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk"] Sep 11 17:01:57.001569 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.001547 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk" Sep 11 17:01:57.013915 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.013889 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk"] Sep 11 17:01:57.140056 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.140016 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgggq\" (UniqueName: \"kubernetes.io/projected/63232f3e-a2aa-4a72-a06c-4b4d9290d03c-kube-api-access-rgggq\") pod \"payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j\" (UID: \"63232f3e-a2aa-4a72-a06c-4b4d9290d03c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j" Sep 11 17:01:57.140214 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.140067 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63232f3e-a2aa-4a72-a06c-4b4d9290d03c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j\" (UID: \"63232f3e-a2aa-4a72-a06c-4b4d9290d03c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j" Sep 11 17:01:57.140214 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.140100 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893ef306-3682-46f8-9824-7f79ee6ad39a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk\" (UID: \"893ef306-3682-46f8-9824-7f79ee6ad39a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk" Sep 11 17:01:57.140214 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.140199 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w9rh\" (UniqueName: \"kubernetes.io/projected/893ef306-3682-46f8-9824-7f79ee6ad39a-kube-api-access-2w9rh\") pod \"payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk\" (UID: \"893ef306-3682-46f8-9824-7f79ee6ad39a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk" Sep 11 17:01:57.241347 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.241312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgggq\" (UniqueName: \"kubernetes.io/projected/63232f3e-a2aa-4a72-a06c-4b4d9290d03c-kube-api-access-rgggq\") pod \"payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j\" (UID: \"63232f3e-a2aa-4a72-a06c-4b4d9290d03c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j" Sep 11 17:01:57.241557 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.241365 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63232f3e-a2aa-4a72-a06c-4b4d9290d03c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j\" (UID: \"63232f3e-a2aa-4a72-a06c-4b4d9290d03c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j" Sep 11 17:01:57.241557 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.241479 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893ef306-3682-46f8-9824-7f79ee6ad39a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk\" (UID: \"893ef306-3682-46f8-9824-7f79ee6ad39a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk" Sep 11 17:01:57.241659 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.241641 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w9rh\" (UniqueName: \"kubernetes.io/projected/893ef306-3682-46f8-9824-7f79ee6ad39a-kube-api-access-2w9rh\") pod \"payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk\" (UID: \"893ef306-3682-46f8-9824-7f79ee6ad39a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk" Sep 11 17:01:57.242132 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.242115 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63232f3e-a2aa-4a72-a06c-4b4d9290d03c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j\" (UID: \"63232f3e-a2aa-4a72-a06c-4b4d9290d03c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j" Sep 11 17:01:57.242196 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.242178 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893ef306-3682-46f8-9824-7f79ee6ad39a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk\" (UID: \"893ef306-3682-46f8-9824-7f79ee6ad39a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk" Sep 11 17:01:57.254420 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.254396 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgggq\" (UniqueName: \"kubernetes.io/projected/63232f3e-a2aa-4a72-a06c-4b4d9290d03c-kube-api-access-rgggq\") pod \"payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j\" (UID: \"63232f3e-a2aa-4a72-a06c-4b4d9290d03c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j" Sep 11 17:01:57.254591 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.254453 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w9rh\" (UniqueName: \"kubernetes.io/projected/893ef306-3682-46f8-9824-7f79ee6ad39a-kube-api-access-2w9rh\") pod \"payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk\" (UID: \"893ef306-3682-46f8-9824-7f79ee6ad39a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk" Sep 11 17:01:57.280454 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.280427 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j" Sep 11 17:01:57.312586 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.312551 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk" Sep 11 17:01:57.559476 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.559449 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j"] Sep 11 17:01:57.560584 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:01:57.560550 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63232f3e_a2aa_4a72_a06c_4b4d9290d03c.slice/crio-9284c4ee3fcf62bfd260fbd9a9f54bbbe930b94117888b07ba074dbbcd4d09e2 WatchSource:0}: Error finding container 9284c4ee3fcf62bfd260fbd9a9f54bbbe930b94117888b07ba074dbbcd4d09e2: Status 404 returned error can't find the container with id 9284c4ee3fcf62bfd260fbd9a9f54bbbe930b94117888b07ba074dbbcd4d09e2 Sep 11 17:01:57.602213 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.602187 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk"] Sep 11 17:01:57.604155 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:01:57.604129 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893ef306_3682_46f8_9824_7f79ee6ad39a.slice/crio-457cbadfb472adc0b68934c30cb67f3816c0eb49613a017d259c28cf298e6540 WatchSource:0}: Error finding container 457cbadfb472adc0b68934c30cb67f3816c0eb49613a017d259c28cf298e6540: Status 404 returned error can't find the container with id 457cbadfb472adc0b68934c30cb67f3816c0eb49613a017d259c28cf298e6540 Sep 11 17:01:57.736304 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.736253 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk" event={"ID":"893ef306-3682-46f8-9824-7f79ee6ad39a","Type":"ContainerStarted","Data":"120a6a6720cbac263bd8ffbc277a4719e24ce415f1ae7d95b81e80a84d0d2590"} Sep 11 17:01:57.736512 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.736318 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk" event={"ID":"893ef306-3682-46f8-9824-7f79ee6ad39a","Type":"ContainerStarted","Data":"457cbadfb472adc0b68934c30cb67f3816c0eb49613a017d259c28cf298e6540"} Sep 11 17:01:57.736512 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.736450 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk" Sep 11 17:01:57.738068 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.738033 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j" event={"ID":"63232f3e-a2aa-4a72-a06c-4b4d9290d03c","Type":"ContainerStarted","Data":"55b43a9509db9a578fafa2a2409fc6c89b1dfd6d1de8a5f254b4d32e82baec6d"} Sep 11 17:01:57.738202 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.738094 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j" Sep 11 17:01:57.738202 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.738118 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j" event={"ID":"63232f3e-a2aa-4a72-a06c-4b4d9290d03c","Type":"ContainerStarted","Data":"9284c4ee3fcf62bfd260fbd9a9f54bbbe930b94117888b07ba074dbbcd4d09e2"} Sep 11 17:01:57.755935 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.755887 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk" podStartSLOduration=1.7558718070000001 podStartE2EDuration="1.755871807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:01:57.755104242 +0000 UTC m=+1133.163206536" watchObservedRunningTime="2026-09-11 17:01:57.755871807 +0000 UTC m=+1133.163974076" Sep 11 17:01:57.773781 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:01:57.773736 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j" podStartSLOduration=1.773718899 podStartE2EDuration="1.773718899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:01:57.772896045 +0000 UTC m=+1133.180998340" watchObservedRunningTime="2026-09-11 17:01:57.773718899 +0000 UTC m=+1133.181821169" Sep 11 17:02:08.745326 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:08.745294 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j" Sep 11 17:02:08.745724 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:08.745366 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk" Sep 11 17:02:38.628183 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:38.628147 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4"] Sep 11 17:02:38.630906 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:38.630886 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4" Sep 11 17:02:38.633739 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:38.633712 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6d422395\"" Sep 11 17:02:38.633859 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:38.633715 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6d422395-dockercfg-94vtl\"" Sep 11 17:02:38.642811 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:38.642781 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4"] Sep 11 17:02:38.743561 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:38.743505 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxpl4\" (UniqueName: \"kubernetes.io/projected/62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5-kube-api-access-nxpl4\") pod \"payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4\" (UID: \"62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4" Sep 11 17:02:38.743825 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:38.743576 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4\" (UID: \"62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4" Sep 11 17:02:38.844355 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:38.844306 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpl4\" (UniqueName: \"kubernetes.io/projected/62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5-kube-api-access-nxpl4\") pod \"payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4\" (UID: \"62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4" Sep 11 17:02:38.844355 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:38.844357 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4\" (UID: \"62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4" Sep 11 17:02:38.845099 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:38.845079 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4\" (UID: \"62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4" Sep 11 17:02:38.855759 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:38.855728 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxpl4\" (UniqueName: \"kubernetes.io/projected/62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5-kube-api-access-nxpl4\") pod \"payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4\" (UID: \"62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4" Sep 11 17:02:38.942234 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:38.942135 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4" Sep 11 17:02:39.219409 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:39.219381 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4"] Sep 11 17:02:39.223027 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:02:39.222991 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ca71b6_109c_4cd7_bf32_ac6f5bd7a5f5.slice/crio-004b61ffeb92f4fefee927d520228a1a968a7bce3385a118880b24629027b664 WatchSource:0}: Error finding container 004b61ffeb92f4fefee927d520228a1a968a7bce3385a118880b24629027b664: Status 404 returned error can't find the container with id 004b61ffeb92f4fefee927d520228a1a968a7bce3385a118880b24629027b664 Sep 11 17:02:39.980581 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:39.980545 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4" event={"ID":"62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5","Type":"ContainerStarted","Data":"4d7a6e994ed13611cc1e729707e352f298a646dfc3062c8d714804d61870e736"} Sep 11 17:02:39.980982 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:39.980599 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4" event={"ID":"62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5","Type":"ContainerStarted","Data":"004b61ffeb92f4fefee927d520228a1a968a7bce3385a118880b24629027b664"} Sep 11 17:02:39.980982 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:39.980631 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4" Sep 11 17:02:40.000280 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:40.000230 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4" podStartSLOduration=2.000213007 podStartE2EDuration="2.000213007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:02:39.999014001 +0000 UTC m=+1175.407116291" watchObservedRunningTime="2026-09-11 17:02:40.000213007 +0000 UTC m=+1175.408315275" Sep 11 17:02:44.994301 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:44.994260 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4268e3-k7w5h"] Sep 11 17:02:44.996913 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:44.996897 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4268e3-k7w5h" Sep 11 17:02:44.998222 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:44.998197 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c283b9b8-e15a-4be2-b805-1f83ec9c646c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4268e3-k7w5h\" (UID: \"c283b9b8-e15a-4be2-b805-1f83ec9c646c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4268e3-k7w5h" Sep 11 17:02:44.999748 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:44.999722 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 17:02:44.999748 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:44.999742 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pc4qb\"" Sep 11 17:02:45.006190 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:45.006167 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4268e3-k7w5h"] Sep 11 17:02:45.099069 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:45.099032 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c283b9b8-e15a-4be2-b805-1f83ec9c646c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4268e3-k7w5h\" (UID: \"c283b9b8-e15a-4be2-b805-1f83ec9c646c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4268e3-k7w5h" Sep 11 17:02:45.099662 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:45.099642 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c283b9b8-e15a-4be2-b805-1f83ec9c646c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4268e3-k7w5h\" (UID: \"c283b9b8-e15a-4be2-b805-1f83ec9c646c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4268e3-k7w5h" Sep 11 17:02:45.309286 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:45.309203 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4268e3-k7w5h" Sep 11 17:02:45.597588 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:45.597558 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4268e3-k7w5h"] Sep 11 17:02:45.599883 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:02:45.599850 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc283b9b8_e15a_4be2_b805_1f83ec9c646c.slice/crio-5607f6260dbba88b443a58eedbe2cba52f486fcf96e0406d8edda3cef48a7d0b WatchSource:0}: Error finding container 5607f6260dbba88b443a58eedbe2cba52f486fcf96e0406d8edda3cef48a7d0b: Status 404 returned error can't find the container with id 5607f6260dbba88b443a58eedbe2cba52f486fcf96e0406d8edda3cef48a7d0b Sep 11 17:02:46.016563 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:46.016529 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4268e3-k7w5h" event={"ID":"c283b9b8-e15a-4be2-b805-1f83ec9c646c","Type":"ContainerStarted","Data":"5607f6260dbba88b443a58eedbe2cba52f486fcf96e0406d8edda3cef48a7d0b"} Sep 11 17:02:47.026131 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:47.026096 2577 generic.go:356] "Generic (PLEG): container finished" podID="c283b9b8-e15a-4be2-b805-1f83ec9c646c" containerID="5ea560c877aa7d3f0b8a76d43544fb44e03ba839db83a0fbe95c57b32b74f069" exitCode=0 Sep 11 17:02:47.026522 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:47.026178 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4268e3-k7w5h" event={"ID":"c283b9b8-e15a-4be2-b805-1f83ec9c646c","Type":"ContainerDied","Data":"5ea560c877aa7d3f0b8a76d43544fb44e03ba839db83a0fbe95c57b32b74f069"} Sep 11 17:02:48.177854 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:48.177828 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4268e3-k7w5h" Sep 11 17:02:48.228937 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:48.228891 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c283b9b8-e15a-4be2-b805-1f83ec9c646c-maas-api-service-ca\") pod \"c283b9b8-e15a-4be2-b805-1f83ec9c646c\" (UID: \"c283b9b8-e15a-4be2-b805-1f83ec9c646c\") " Sep 11 17:02:48.229255 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:48.229229 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c283b9b8-e15a-4be2-b805-1f83ec9c646c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c283b9b8-e15a-4be2-b805-1f83ec9c646c" (UID: "c283b9b8-e15a-4be2-b805-1f83ec9c646c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:02:48.329741 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:48.329658 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c283b9b8-e15a-4be2-b805-1f83ec9c646c-maas-api-service-ca\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:02:49.038060 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:49.038026 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4268e3-k7w5h" Sep 11 17:02:49.038232 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:49.038072 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4268e3-k7w5h" event={"ID":"c283b9b8-e15a-4be2-b805-1f83ec9c646c","Type":"ContainerDied","Data":"5607f6260dbba88b443a58eedbe2cba52f486fcf96e0406d8edda3cef48a7d0b"} Sep 11 17:02:49.038232 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:49.038141 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5607f6260dbba88b443a58eedbe2cba52f486fcf96e0406d8edda3cef48a7d0b" Sep 11 17:02:50.988000 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:50.987966 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4" Sep 11 17:02:56.049006 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.048966 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4"] Sep 11 17:02:56.049410 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.049251 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4" podUID="1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa" containerName="payload-processing" containerID="cri-o://ac06863ec3e26cfbe4d874deb4a537a022618489388bbe876417834a8dfd8bf5" gracePeriod=30 Sep 11 17:02:56.058936 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.058300 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468"] Sep 11 17:02:56.058936 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.058567 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468" podUID="df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d" containerName="payload-pre-processing" containerID="cri-o://ef6ed3b984b86c498fc60c42dccf2de27f7d3a5502e1f82779746e0a514d72e5" gracePeriod=30 Sep 11 17:02:56.291770 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.291716 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4268e3-k7w5h"] Sep 11 17:02:56.307928 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.307850 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4268e3-k7w5h"] Sep 11 17:02:56.355185 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.355164 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4" Sep 11 17:02:56.358193 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.358174 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468" Sep 11 17:02:56.417402 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.416918 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znk2f\" (UniqueName: \"kubernetes.io/projected/df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d-kube-api-access-znk2f\") pod \"df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d\" (UID: \"df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d\") " Sep 11 17:02:56.417837 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.417709 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa-plugins-config-volume\") pod \"1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa\" (UID: \"1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa\") " Sep 11 17:02:56.417837 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.417790 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5lnd\" (UniqueName: \"kubernetes.io/projected/1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa-kube-api-access-b5lnd\") pod \"1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa\" (UID: \"1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa\") " Sep 11 17:02:56.418132 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.418107 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d-plugins-config-volume\") pod \"df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d\" (UID: \"df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d\") " Sep 11 17:02:56.418706 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.418674 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d" (UID: "df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:02:56.418812 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.418726 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa" (UID: "1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:02:56.422372 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.422342 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa-kube-api-access-b5lnd" (OuterVolumeSpecName: "kube-api-access-b5lnd") pod "1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa" (UID: "1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa"). InnerVolumeSpecName "kube-api-access-b5lnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:02:56.422672 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.422643 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d-kube-api-access-znk2f" (OuterVolumeSpecName: "kube-api-access-znk2f") pod "df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d" (UID: "df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d"). InnerVolumeSpecName "kube-api-access-znk2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:02:56.519950 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.519912 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5lnd\" (UniqueName: \"kubernetes.io/projected/1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa-kube-api-access-b5lnd\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:02:56.519950 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.519943 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:02:56.519950 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.519956 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znk2f\" (UniqueName: \"kubernetes.io/projected/df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d-kube-api-access-znk2f\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:02:56.520171 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:56.519969 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:02:57.087355 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:57.087315 2577 generic.go:356] "Generic (PLEG): container finished" podID="df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d" containerID="ef6ed3b984b86c498fc60c42dccf2de27f7d3a5502e1f82779746e0a514d72e5" exitCode=0 Sep 11 17:02:57.087846 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:57.087371 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468" Sep 11 17:02:57.087846 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:57.087400 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468" event={"ID":"df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d","Type":"ContainerDied","Data":"ef6ed3b984b86c498fc60c42dccf2de27f7d3a5502e1f82779746e0a514d72e5"} Sep 11 17:02:57.087846 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:57.087536 2577 scope.go:117] "RemoveContainer" containerID="ef6ed3b984b86c498fc60c42dccf2de27f7d3a5502e1f82779746e0a514d72e5" Sep 11 17:02:57.089005 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:57.088987 2577 generic.go:356] "Generic (PLEG): container finished" podID="1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa" containerID="ac06863ec3e26cfbe4d874deb4a537a022618489388bbe876417834a8dfd8bf5" exitCode=0 Sep 11 17:02:57.089074 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:57.089049 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4" Sep 11 17:02:57.095324 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:57.095297 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c283b9b8-e15a-4be2-b805-1f83ec9c646c" path="/var/lib/kubelet/pods/c283b9b8-e15a-4be2-b805-1f83ec9c646c/volumes" Sep 11 17:02:57.095975 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:57.095951 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468" event={"ID":"df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d","Type":"ContainerDied","Data":"8c375b8505650947abf79735fd4c13b0ca54efa099e4d43b212d8249558178a2"} Sep 11 17:02:57.096085 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:57.095983 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4" event={"ID":"1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa","Type":"ContainerDied","Data":"ac06863ec3e26cfbe4d874deb4a537a022618489388bbe876417834a8dfd8bf5"} Sep 11 17:02:57.096085 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:57.096019 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4" event={"ID":"1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa","Type":"ContainerDied","Data":"9167d8119e3560e6219c52818696557414fb36d1306f3c0ac0e6c7c12b030453"} Sep 11 17:02:57.099048 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:57.099030 2577 scope.go:117] "RemoveContainer" containerID="ef6ed3b984b86c498fc60c42dccf2de27f7d3a5502e1f82779746e0a514d72e5" Sep 11 17:02:57.099334 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:02:57.099314 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef6ed3b984b86c498fc60c42dccf2de27f7d3a5502e1f82779746e0a514d72e5\": container with ID starting with ef6ed3b984b86c498fc60c42dccf2de27f7d3a5502e1f82779746e0a514d72e5 not found: ID does not exist" containerID="ef6ed3b984b86c498fc60c42dccf2de27f7d3a5502e1f82779746e0a514d72e5" Sep 11 17:02:57.099416 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:57.099342 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef6ed3b984b86c498fc60c42dccf2de27f7d3a5502e1f82779746e0a514d72e5"} err="failed to get container status \"ef6ed3b984b86c498fc60c42dccf2de27f7d3a5502e1f82779746e0a514d72e5\": rpc error: code = NotFound desc = could not find container \"ef6ed3b984b86c498fc60c42dccf2de27f7d3a5502e1f82779746e0a514d72e5\": container with ID starting with ef6ed3b984b86c498fc60c42dccf2de27f7d3a5502e1f82779746e0a514d72e5 not found: ID does not exist" Sep 11 17:02:57.099416 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:57.099360 2577 scope.go:117] "RemoveContainer" containerID="ac06863ec3e26cfbe4d874deb4a537a022618489388bbe876417834a8dfd8bf5" Sep 11 17:02:57.108615 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:57.108597 2577 scope.go:117] "RemoveContainer" containerID="ac06863ec3e26cfbe4d874deb4a537a022618489388bbe876417834a8dfd8bf5" Sep 11 17:02:57.108867 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:02:57.108849 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac06863ec3e26cfbe4d874deb4a537a022618489388bbe876417834a8dfd8bf5\": container with ID starting with ac06863ec3e26cfbe4d874deb4a537a022618489388bbe876417834a8dfd8bf5 not found: ID does not exist" containerID="ac06863ec3e26cfbe4d874deb4a537a022618489388bbe876417834a8dfd8bf5" Sep 11 17:02:57.108930 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:57.108873 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac06863ec3e26cfbe4d874deb4a537a022618489388bbe876417834a8dfd8bf5"} err="failed to get container status \"ac06863ec3e26cfbe4d874deb4a537a022618489388bbe876417834a8dfd8bf5\": rpc error: code = NotFound desc = could not find container \"ac06863ec3e26cfbe4d874deb4a537a022618489388bbe876417834a8dfd8bf5\": container with ID starting with ac06863ec3e26cfbe4d874deb4a537a022618489388bbe876417834a8dfd8bf5 not found: ID does not exist" Sep 11 17:02:57.118658 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:57.118628 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4"] Sep 11 17:02:57.124162 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:57.124120 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4268e3-54594494d5-qdfn4"] Sep 11 17:02:57.137859 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:57.137833 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468"] Sep 11 17:02:57.141276 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:57.141254 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4268e3-699f8ff594-hg468"] Sep 11 17:02:59.095218 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:59.095185 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa" path="/var/lib/kubelet/pods/1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa/volumes" Sep 11 17:02:59.095634 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:02:59.095577 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d" path="/var/lib/kubelet/pods/df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d/volumes" Sep 11 17:03:05.073899 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:05.073861 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 17:03:05.075241 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:05.075218 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 17:03:09.687526 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:09.687462 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669"] Sep 11 17:03:24.708433 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:24.708392 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" podUID="2fa03b19-98f8-44e6-b696-d29613305934" containerName="main" containerID="cri-o://32bdb74eca1464bfa8c891459ce4cc494486b8d7ea75e63b377a61f906aa4df3" gracePeriod=45 Sep 11 17:03:24.962403 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:24.962346 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 17:03:24.985979 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:24.985948 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tg58\" (UniqueName: \"kubernetes.io/projected/2fa03b19-98f8-44e6-b696-d29613305934-kube-api-access-4tg58\") pod \"2fa03b19-98f8-44e6-b696-d29613305934\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " Sep 11 17:03:24.986132 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:24.986009 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-dshm\") pod \"2fa03b19-98f8-44e6-b696-d29613305934\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " Sep 11 17:03:24.986132 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:24.986087 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-tmp-dir\") pod \"2fa03b19-98f8-44e6-b696-d29613305934\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " Sep 11 17:03:24.986132 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:24.986110 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-model-cache\") pod \"2fa03b19-98f8-44e6-b696-d29613305934\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " Sep 11 17:03:24.986682 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:24.986340 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2fa03b19-98f8-44e6-b696-d29613305934" (UID: "2fa03b19-98f8-44e6-b696-d29613305934"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:03:24.986682 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:24.986411 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-model-cache" (OuterVolumeSpecName: "model-cache") pod "2fa03b19-98f8-44e6-b696-d29613305934" (UID: "2fa03b19-98f8-44e6-b696-d29613305934"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:03:24.986682 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:24.986160 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fa03b19-98f8-44e6-b696-d29613305934-tls-certs\") pod \"2fa03b19-98f8-44e6-b696-d29613305934\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " Sep 11 17:03:24.986682 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:24.986613 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-home\") pod \"2fa03b19-98f8-44e6-b696-d29613305934\" (UID: \"2fa03b19-98f8-44e6-b696-d29613305934\") " Sep 11 17:03:24.986972 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:24.986948 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-tmp-dir\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:03:24.987027 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:24.986969 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-model-cache\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:03:24.987241 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:24.987194 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-home" (OuterVolumeSpecName: "home") pod "2fa03b19-98f8-44e6-b696-d29613305934" (UID: "2fa03b19-98f8-44e6-b696-d29613305934"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:03:24.988812 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:24.988784 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa03b19-98f8-44e6-b696-d29613305934-kube-api-access-4tg58" (OuterVolumeSpecName: "kube-api-access-4tg58") pod "2fa03b19-98f8-44e6-b696-d29613305934" (UID: "2fa03b19-98f8-44e6-b696-d29613305934"). InnerVolumeSpecName "kube-api-access-4tg58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:03:24.988812 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:24.988800 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-dshm" (OuterVolumeSpecName: "dshm") pod "2fa03b19-98f8-44e6-b696-d29613305934" (UID: "2fa03b19-98f8-44e6-b696-d29613305934"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:03:24.989083 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:24.989057 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fa03b19-98f8-44e6-b696-d29613305934-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fa03b19-98f8-44e6-b696-d29613305934" (UID: "2fa03b19-98f8-44e6-b696-d29613305934"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 17:03:25.087440 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.087405 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-dshm\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:03:25.087440 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.087432 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fa03b19-98f8-44e6-b696-d29613305934-tls-certs\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:03:25.087440 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.087442 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fa03b19-98f8-44e6-b696-d29613305934-home\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:03:25.087440 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.087450 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tg58\" (UniqueName: \"kubernetes.io/projected/2fa03b19-98f8-44e6-b696-d29613305934-kube-api-access-4tg58\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:03:25.251239 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.251199 2577 generic.go:356] "Generic (PLEG): container finished" podID="2fa03b19-98f8-44e6-b696-d29613305934" containerID="32bdb74eca1464bfa8c891459ce4cc494486b8d7ea75e63b377a61f906aa4df3" exitCode=0 Sep 11 17:03:25.251440 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.251250 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" Sep 11 17:03:25.251440 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.251287 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" event={"ID":"2fa03b19-98f8-44e6-b696-d29613305934","Type":"ContainerDied","Data":"32bdb74eca1464bfa8c891459ce4cc494486b8d7ea75e63b377a61f906aa4df3"} Sep 11 17:03:25.251440 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.251342 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669" event={"ID":"2fa03b19-98f8-44e6-b696-d29613305934","Type":"ContainerDied","Data":"5c7146807a5d9c98238b01100760b7baad58cf5875cc57b77ad123e17e4bda11"} Sep 11 17:03:25.251440 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.251366 2577 scope.go:117] "RemoveContainer" containerID="32bdb74eca1464bfa8c891459ce4cc494486b8d7ea75e63b377a61f906aa4df3" Sep 11 17:03:25.260438 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.260420 2577 scope.go:117] "RemoveContainer" containerID="32bdb74eca1464bfa8c891459ce4cc494486b8d7ea75e63b377a61f906aa4df3" Sep 11 17:03:25.260738 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:03:25.260720 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32bdb74eca1464bfa8c891459ce4cc494486b8d7ea75e63b377a61f906aa4df3\": container with ID starting with 32bdb74eca1464bfa8c891459ce4cc494486b8d7ea75e63b377a61f906aa4df3 not found: ID does not exist" containerID="32bdb74eca1464bfa8c891459ce4cc494486b8d7ea75e63b377a61f906aa4df3" Sep 11 17:03:25.260817 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.260747 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32bdb74eca1464bfa8c891459ce4cc494486b8d7ea75e63b377a61f906aa4df3"} err="failed to get container status \"32bdb74eca1464bfa8c891459ce4cc494486b8d7ea75e63b377a61f906aa4df3\": rpc error: code = NotFound desc = could not find container \"32bdb74eca1464bfa8c891459ce4cc494486b8d7ea75e63b377a61f906aa4df3\": container with ID starting with 32bdb74eca1464bfa8c891459ce4cc494486b8d7ea75e63b377a61f906aa4df3 not found: ID does not exist" Sep 11 17:03:25.269770 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.269749 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669"] Sep 11 17:03:25.273673 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.273652 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-32d680/test-model-32d680-kserve-7b68cf6899-ck669"] Sep 11 17:03:25.691738 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.691659 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv"] Sep 11 17:03:25.692057 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.692046 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d" containerName="payload-pre-processing" Sep 11 17:03:25.692101 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.692059 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d" containerName="payload-pre-processing" Sep 11 17:03:25.692101 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.692076 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fa03b19-98f8-44e6-b696-d29613305934" containerName="main" Sep 11 17:03:25.692101 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.692082 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fa03b19-98f8-44e6-b696-d29613305934" containerName="main" Sep 11 17:03:25.692101 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.692095 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c283b9b8-e15a-4be2-b805-1f83ec9c646c" containerName="revoke-keys" Sep 11 17:03:25.692101 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.692102 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c283b9b8-e15a-4be2-b805-1f83ec9c646c" containerName="revoke-keys" Sep 11 17:03:25.692259 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.692109 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa" containerName="payload-processing" Sep 11 17:03:25.692259 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.692115 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa" containerName="payload-processing" Sep 11 17:03:25.692259 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.692182 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c5cfb02-37a1-4351-bf1c-d5b9a1e452aa" containerName="payload-processing" Sep 11 17:03:25.692259 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.692193 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fa03b19-98f8-44e6-b696-d29613305934" containerName="main" Sep 11 17:03:25.692259 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.692199 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c283b9b8-e15a-4be2-b805-1f83ec9c646c" containerName="revoke-keys" Sep 11 17:03:25.692259 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.692205 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="df3ca3c5-0f50-48b6-8e7b-a4c1ba9f001d" containerName="payload-pre-processing" Sep 11 17:03:25.695269 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.695249 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:25.698238 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.698220 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6d422395\"/\"default-dockercfg-ctrfr\"" Sep 11 17:03:25.699412 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.699392 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6d422395\"/\"e2e-lifecycle-model-6d422395-kserve-self-signed-certs\"" Sep 11 17:03:25.699511 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.699424 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6d422395\"/\"openshift-service-ca.crt\"" Sep 11 17:03:25.699511 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.699450 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6d422395\"/\"kube-root-ca.crt\"" Sep 11 17:03:25.706326 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.706305 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv"] Sep 11 17:03:25.794351 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.794316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnjbr\" (UniqueName: \"kubernetes.io/projected/ae09052e-2084-4af5-b678-b817133de228-kube-api-access-tnjbr\") pod \"e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:25.794758 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.794363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-model-cache\") pod \"e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:25.794758 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.794442 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae09052e-2084-4af5-b678-b817133de228-tls-certs\") pod \"e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:25.794758 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.794531 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-tmp-dir\") pod \"e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:25.794758 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.794575 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-home\") pod \"e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:25.794758 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.794611 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-dshm\") pod \"e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:25.895885 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.895847 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-dshm\") pod \"e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:25.895885 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.895888 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnjbr\" (UniqueName: \"kubernetes.io/projected/ae09052e-2084-4af5-b678-b817133de228-kube-api-access-tnjbr\") pod \"e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:25.896100 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.896005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-model-cache\") pod \"e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:25.896100 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.896082 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae09052e-2084-4af5-b678-b817133de228-tls-certs\") pod \"e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:25.896183 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.896112 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-tmp-dir\") pod \"e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:25.896183 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.896156 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-home\") pod \"e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:25.896434 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.896405 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-model-cache\") pod \"e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:25.896589 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.896459 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-tmp-dir\") pod \"e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:25.896589 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.896492 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-home\") pod \"e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:25.898184 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.898155 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-dshm\") pod \"e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:25.898479 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.898463 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae09052e-2084-4af5-b678-b817133de228-tls-certs\") pod \"e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:25.904785 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:25.904764 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnjbr\" (UniqueName: \"kubernetes.io/projected/ae09052e-2084-4af5-b678-b817133de228-kube-api-access-tnjbr\") pod \"e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:26.004986 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:26.004951 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:26.284704 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:26.284615 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv"] Sep 11 17:03:26.290582 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:03:26.290556 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae09052e_2084_4af5_b678_b817133de228.slice/crio-3043923f152b52b76d58dda41770cc60504b3c72c22b5a6215743d720fd0f230 WatchSource:0}: Error finding container 3043923f152b52b76d58dda41770cc60504b3c72c22b5a6215743d720fd0f230: Status 404 returned error can't find the container with id 3043923f152b52b76d58dda41770cc60504b3c72c22b5a6215743d720fd0f230 Sep 11 17:03:27.094852 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:27.094818 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fa03b19-98f8-44e6-b696-d29613305934" path="/var/lib/kubelet/pods/2fa03b19-98f8-44e6-b696-d29613305934/volumes" Sep 11 17:03:27.264317 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:27.264277 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" event={"ID":"ae09052e-2084-4af5-b678-b817133de228","Type":"ContainerStarted","Data":"1dcdc61d61a17cb2681034aa5f12be68c3fc51456c63743994d2e76d6b01ddc9"} Sep 11 17:03:27.264317 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:27.264323 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" event={"ID":"ae09052e-2084-4af5-b678-b817133de228","Type":"ContainerStarted","Data":"3043923f152b52b76d58dda41770cc60504b3c72c22b5a6215743d720fd0f230"} Sep 11 17:03:27.283363 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:27.283318 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" podStartSLOduration=2.283303138 podStartE2EDuration="2.283303138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:03:27.28182135 +0000 UTC m=+1222.689923644" watchObservedRunningTime="2026-09-11 17:03:27.283303138 +0000 UTC m=+1222.691405406" Sep 11 17:03:36.005463 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:36.005419 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:36.005980 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:36.005492 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:36.018290 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:36.018261 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:36.325916 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:36.325818 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:03:40.276161 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:40.276125 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw"] Sep 11 17:03:40.279599 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:40.279581 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw" Sep 11 17:03:40.282883 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:40.282854 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pc4qb\"" Sep 11 17:03:40.283014 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:40.282884 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 17:03:40.290353 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:40.290321 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw"] Sep 11 17:03:40.323467 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:40.323422 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8200ddb5-9fb3-4ed1-a0a5-0e21100395cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw\" (UID: \"8200ddb5-9fb3-4ed1-a0a5-0e21100395cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw" Sep 11 17:03:40.424220 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:40.424185 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8200ddb5-9fb3-4ed1-a0a5-0e21100395cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw\" (UID: \"8200ddb5-9fb3-4ed1-a0a5-0e21100395cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw" Sep 11 17:03:40.424841 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:40.424820 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8200ddb5-9fb3-4ed1-a0a5-0e21100395cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw\" (UID: \"8200ddb5-9fb3-4ed1-a0a5-0e21100395cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw" Sep 11 17:03:40.593605 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:40.593522 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw" Sep 11 17:03:40.871158 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:40.871132 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw"] Sep 11 17:03:40.872859 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:03:40.872824 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8200ddb5_9fb3_4ed1_a0a5_0e21100395cb.slice/crio-b530ba944ba0cd11e223f34bc78c6d7a8e075f8010538fbc8ea961ea9d63bcd4 WatchSource:0}: Error finding container b530ba944ba0cd11e223f34bc78c6d7a8e075f8010538fbc8ea961ea9d63bcd4: Status 404 returned error can't find the container with id b530ba944ba0cd11e223f34bc78c6d7a8e075f8010538fbc8ea961ea9d63bcd4 Sep 11 17:03:41.346058 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:41.346024 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw" event={"ID":"8200ddb5-9fb3-4ed1-a0a5-0e21100395cb","Type":"ContainerStarted","Data":"1e75cd41bda7722dc4c4678c7a057d8e800b8ca62989784140d9546bac386d29"} Sep 11 17:03:41.346431 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:41.346075 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw" event={"ID":"8200ddb5-9fb3-4ed1-a0a5-0e21100395cb","Type":"ContainerStarted","Data":"b530ba944ba0cd11e223f34bc78c6d7a8e075f8010538fbc8ea961ea9d63bcd4"} Sep 11 17:03:41.367818 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:41.367768 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw" podStartSLOduration=1.367752108 podStartE2EDuration="1.367752108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:03:41.365566036 +0000 UTC m=+1236.773668322" watchObservedRunningTime="2026-09-11 17:03:41.367752108 +0000 UTC m=+1236.775854376" Sep 11 17:03:42.352408 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:42.352377 2577 generic.go:356] "Generic (PLEG): container finished" podID="8200ddb5-9fb3-4ed1-a0a5-0e21100395cb" containerID="1e75cd41bda7722dc4c4678c7a057d8e800b8ca62989784140d9546bac386d29" exitCode=0 Sep 11 17:03:42.352849 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:42.352487 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw" event={"ID":"8200ddb5-9fb3-4ed1-a0a5-0e21100395cb","Type":"ContainerDied","Data":"1e75cd41bda7722dc4c4678c7a057d8e800b8ca62989784140d9546bac386d29"} Sep 11 17:03:43.499540 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:43.499513 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw" Sep 11 17:03:43.555894 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:43.555854 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8200ddb5-9fb3-4ed1-a0a5-0e21100395cb-maas-api-service-ca\") pod \"8200ddb5-9fb3-4ed1-a0a5-0e21100395cb\" (UID: \"8200ddb5-9fb3-4ed1-a0a5-0e21100395cb\") " Sep 11 17:03:43.556208 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:43.556184 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8200ddb5-9fb3-4ed1-a0a5-0e21100395cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8200ddb5-9fb3-4ed1-a0a5-0e21100395cb" (UID: "8200ddb5-9fb3-4ed1-a0a5-0e21100395cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:03:43.657076 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:43.656987 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8200ddb5-9fb3-4ed1-a0a5-0e21100395cb-maas-api-service-ca\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:03:44.364597 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:44.364564 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw" event={"ID":"8200ddb5-9fb3-4ed1-a0a5-0e21100395cb","Type":"ContainerDied","Data":"b530ba944ba0cd11e223f34bc78c6d7a8e075f8010538fbc8ea961ea9d63bcd4"} Sep 11 17:03:44.364597 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:44.364596 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw" Sep 11 17:03:44.364814 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:44.364613 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b530ba944ba0cd11e223f34bc78c6d7a8e075f8010538fbc8ea961ea9d63bcd4" Sep 11 17:03:50.975742 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:50.975691 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32d680-29819100-46zn2"] Sep 11 17:03:50.978185 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:50.978161 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-32d680-29819100-46zn2"] Sep 11 17:03:51.003260 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.003227 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf"] Sep 11 17:03:51.003547 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.003510 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf" podUID="5696d4b7-12a5-4508-aa80-7d6d4e1d9f68" containerName="payload-processing" containerID="cri-o://b0b9efd163fb10f1f038cc11d0603541c4041ba2d0d3ab216225b02349e57654" gracePeriod=30 Sep 11 17:03:51.009771 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.009742 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7"] Sep 11 17:03:51.010426 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.010364 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7" podUID="6ab5e888-ed40-44f9-b440-7f53f3df906b" containerName="payload-pre-processing" containerID="cri-o://470cafe5b4e3d96b0badec3d653ebe83d7d46ff6877e39e9c5e2499e21b50fa3" gracePeriod=30 Sep 11 17:03:51.094367 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.094332 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeba4068-9144-4fca-b299-92f0d5b63ac2" path="/var/lib/kubelet/pods/eeba4068-9144-4fca-b299-92f0d5b63ac2/volumes" Sep 11 17:03:51.152694 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.152626 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw"] Sep 11 17:03:51.154837 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.154783 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-32d680-ms4mw"] Sep 11 17:03:51.275968 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.275940 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf" Sep 11 17:03:51.306004 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.305980 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7" Sep 11 17:03:51.335209 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.335173 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzvwg\" (UniqueName: \"kubernetes.io/projected/5696d4b7-12a5-4508-aa80-7d6d4e1d9f68-kube-api-access-nzvwg\") pod \"5696d4b7-12a5-4508-aa80-7d6d4e1d9f68\" (UID: \"5696d4b7-12a5-4508-aa80-7d6d4e1d9f68\") " Sep 11 17:03:51.335450 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.335427 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5696d4b7-12a5-4508-aa80-7d6d4e1d9f68-plugins-config-volume\") pod \"5696d4b7-12a5-4508-aa80-7d6d4e1d9f68\" (UID: \"5696d4b7-12a5-4508-aa80-7d6d4e1d9f68\") " Sep 11 17:03:51.335831 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.335807 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5696d4b7-12a5-4508-aa80-7d6d4e1d9f68-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5696d4b7-12a5-4508-aa80-7d6d4e1d9f68" (UID: "5696d4b7-12a5-4508-aa80-7d6d4e1d9f68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:03:51.337468 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.337445 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5696d4b7-12a5-4508-aa80-7d6d4e1d9f68-kube-api-access-nzvwg" (OuterVolumeSpecName: "kube-api-access-nzvwg") pod "5696d4b7-12a5-4508-aa80-7d6d4e1d9f68" (UID: "5696d4b7-12a5-4508-aa80-7d6d4e1d9f68"). InnerVolumeSpecName "kube-api-access-nzvwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:03:51.403772 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.403738 2577 generic.go:356] "Generic (PLEG): container finished" podID="6ab5e888-ed40-44f9-b440-7f53f3df906b" containerID="470cafe5b4e3d96b0badec3d653ebe83d7d46ff6877e39e9c5e2499e21b50fa3" exitCode=0 Sep 11 17:03:51.403897 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.403784 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7" Sep 11 17:03:51.403897 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.403821 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7" event={"ID":"6ab5e888-ed40-44f9-b440-7f53f3df906b","Type":"ContainerDied","Data":"470cafe5b4e3d96b0badec3d653ebe83d7d46ff6877e39e9c5e2499e21b50fa3"} Sep 11 17:03:51.403897 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.403867 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7" event={"ID":"6ab5e888-ed40-44f9-b440-7f53f3df906b","Type":"ContainerDied","Data":"74c9ddd67123a5c2c3d01064717efcbd20b21fc62f07e93b8b980a73eb9500e1"} Sep 11 17:03:51.403897 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.403894 2577 scope.go:117] "RemoveContainer" containerID="470cafe5b4e3d96b0badec3d653ebe83d7d46ff6877e39e9c5e2499e21b50fa3" Sep 11 17:03:51.405077 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.405060 2577 generic.go:356] "Generic (PLEG): container finished" podID="5696d4b7-12a5-4508-aa80-7d6d4e1d9f68" containerID="b0b9efd163fb10f1f038cc11d0603541c4041ba2d0d3ab216225b02349e57654" exitCode=0 Sep 11 17:03:51.405160 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.405102 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf" Sep 11 17:03:51.405160 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.405122 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf" event={"ID":"5696d4b7-12a5-4508-aa80-7d6d4e1d9f68","Type":"ContainerDied","Data":"b0b9efd163fb10f1f038cc11d0603541c4041ba2d0d3ab216225b02349e57654"} Sep 11 17:03:51.405266 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.405162 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf" event={"ID":"5696d4b7-12a5-4508-aa80-7d6d4e1d9f68","Type":"ContainerDied","Data":"a17dc4b18f7a94f37dde6fa0b6f37cd8d13aabc5cc634942ce1eb42fa7783f0c"} Sep 11 17:03:51.413068 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.413051 2577 scope.go:117] "RemoveContainer" containerID="470cafe5b4e3d96b0badec3d653ebe83d7d46ff6877e39e9c5e2499e21b50fa3" Sep 11 17:03:51.413302 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:03:51.413284 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"470cafe5b4e3d96b0badec3d653ebe83d7d46ff6877e39e9c5e2499e21b50fa3\": container with ID starting with 470cafe5b4e3d96b0badec3d653ebe83d7d46ff6877e39e9c5e2499e21b50fa3 not found: ID does not exist" containerID="470cafe5b4e3d96b0badec3d653ebe83d7d46ff6877e39e9c5e2499e21b50fa3" Sep 11 17:03:51.413351 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.413309 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470cafe5b4e3d96b0badec3d653ebe83d7d46ff6877e39e9c5e2499e21b50fa3"} err="failed to get container status \"470cafe5b4e3d96b0badec3d653ebe83d7d46ff6877e39e9c5e2499e21b50fa3\": rpc error: code = NotFound desc = could not find container \"470cafe5b4e3d96b0badec3d653ebe83d7d46ff6877e39e9c5e2499e21b50fa3\": container with ID starting with 470cafe5b4e3d96b0badec3d653ebe83d7d46ff6877e39e9c5e2499e21b50fa3 not found: ID does not exist" Sep 11 17:03:51.413351 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.413322 2577 scope.go:117] "RemoveContainer" containerID="b0b9efd163fb10f1f038cc11d0603541c4041ba2d0d3ab216225b02349e57654" Sep 11 17:03:51.422026 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.421585 2577 scope.go:117] "RemoveContainer" containerID="b0b9efd163fb10f1f038cc11d0603541c4041ba2d0d3ab216225b02349e57654" Sep 11 17:03:51.422026 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:03:51.421937 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0b9efd163fb10f1f038cc11d0603541c4041ba2d0d3ab216225b02349e57654\": container with ID starting with b0b9efd163fb10f1f038cc11d0603541c4041ba2d0d3ab216225b02349e57654 not found: ID does not exist" containerID="b0b9efd163fb10f1f038cc11d0603541c4041ba2d0d3ab216225b02349e57654" Sep 11 17:03:51.422026 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.421963 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0b9efd163fb10f1f038cc11d0603541c4041ba2d0d3ab216225b02349e57654"} err="failed to get container status \"b0b9efd163fb10f1f038cc11d0603541c4041ba2d0d3ab216225b02349e57654\": rpc error: code = NotFound desc = could not find container \"b0b9efd163fb10f1f038cc11d0603541c4041ba2d0d3ab216225b02349e57654\": container with ID starting with b0b9efd163fb10f1f038cc11d0603541c4041ba2d0d3ab216225b02349e57654 not found: ID does not exist" Sep 11 17:03:51.426342 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.426315 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf"] Sep 11 17:03:51.428300 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.428283 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-32d680-5bf5cf8c97-g6pvf"] Sep 11 17:03:51.436883 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.436863 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25kqr\" (UniqueName: \"kubernetes.io/projected/6ab5e888-ed40-44f9-b440-7f53f3df906b-kube-api-access-25kqr\") pod \"6ab5e888-ed40-44f9-b440-7f53f3df906b\" (UID: \"6ab5e888-ed40-44f9-b440-7f53f3df906b\") " Sep 11 17:03:51.436976 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.436964 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5e888-ed40-44f9-b440-7f53f3df906b-plugins-config-volume\") pod \"6ab5e888-ed40-44f9-b440-7f53f3df906b\" (UID: \"6ab5e888-ed40-44f9-b440-7f53f3df906b\") " Sep 11 17:03:51.437208 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.437195 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5696d4b7-12a5-4508-aa80-7d6d4e1d9f68-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:03:51.437274 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.437209 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzvwg\" (UniqueName: \"kubernetes.io/projected/5696d4b7-12a5-4508-aa80-7d6d4e1d9f68-kube-api-access-nzvwg\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:03:51.437325 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.437267 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab5e888-ed40-44f9-b440-7f53f3df906b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ab5e888-ed40-44f9-b440-7f53f3df906b" (UID: "6ab5e888-ed40-44f9-b440-7f53f3df906b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:03:51.438854 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.438836 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab5e888-ed40-44f9-b440-7f53f3df906b-kube-api-access-25kqr" (OuterVolumeSpecName: "kube-api-access-25kqr") pod "6ab5e888-ed40-44f9-b440-7f53f3df906b" (UID: "6ab5e888-ed40-44f9-b440-7f53f3df906b"). InnerVolumeSpecName "kube-api-access-25kqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:03:51.538614 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.538534 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab5e888-ed40-44f9-b440-7f53f3df906b-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:03:51.538614 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.538562 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25kqr\" (UniqueName: \"kubernetes.io/projected/6ab5e888-ed40-44f9-b440-7f53f3df906b-kube-api-access-25kqr\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:03:51.728321 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.728289 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7"] Sep 11 17:03:51.731953 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:51.731921 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-32d680-85b754fb74-s4vf7"] Sep 11 17:03:53.099865 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:53.099827 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5696d4b7-12a5-4508-aa80-7d6d4e1d9f68" path="/var/lib/kubelet/pods/5696d4b7-12a5-4508-aa80-7d6d4e1d9f68/volumes" Sep 11 17:03:53.100243 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:53.100181 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ab5e888-ed40-44f9-b440-7f53f3df906b" path="/var/lib/kubelet/pods/6ab5e888-ed40-44f9-b440-7f53f3df906b/volumes" Sep 11 17:03:53.100519 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:03:53.100489 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8200ddb5-9fb3-4ed1-a0a5-0e21100395cb" path="/var/lib/kubelet/pods/8200ddb5-9fb3-4ed1-a0a5-0e21100395cb/volumes" Sep 11 17:04:02.496512 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.496419 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd"] Sep 11 17:04:02.497052 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.496935 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ab5e888-ed40-44f9-b440-7f53f3df906b" containerName="payload-pre-processing" Sep 11 17:04:02.497052 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.496958 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ab5e888-ed40-44f9-b440-7f53f3df906b" containerName="payload-pre-processing" Sep 11 17:04:02.497052 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.496974 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8200ddb5-9fb3-4ed1-a0a5-0e21100395cb" containerName="revoke-keys" Sep 11 17:04:02.497052 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.496993 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8200ddb5-9fb3-4ed1-a0a5-0e21100395cb" containerName="revoke-keys" Sep 11 17:04:02.497052 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.497003 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5696d4b7-12a5-4508-aa80-7d6d4e1d9f68" containerName="payload-processing" Sep 11 17:04:02.497052 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.497012 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5696d4b7-12a5-4508-aa80-7d6d4e1d9f68" containerName="payload-processing" Sep 11 17:04:02.497323 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.497124 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ab5e888-ed40-44f9-b440-7f53f3df906b" containerName="payload-pre-processing" Sep 11 17:04:02.497323 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.497133 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8200ddb5-9fb3-4ed1-a0a5-0e21100395cb" containerName="revoke-keys" Sep 11 17:04:02.497323 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.497141 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5696d4b7-12a5-4508-aa80-7d6d4e1d9f68" containerName="payload-processing" Sep 11 17:04:02.501791 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.501773 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:02.505959 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.505932 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2dc95c\"/\"openshift-service-ca.crt\"" Sep 11 17:04:02.506092 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.505993 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2dc95c\"/\"e2e-conflict-b-44a0e379-kserve-self-signed-certs\"" Sep 11 17:04:02.506092 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.506024 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2dc95c\"/\"default-dockercfg-rnfvr\"" Sep 11 17:04:02.506229 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.506093 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2dc95c\"/\"kube-root-ca.crt\"" Sep 11 17:04:02.515763 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.515739 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd"] Sep 11 17:04:02.641790 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.641752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn8hg\" (UniqueName: \"kubernetes.io/projected/48d1143f-53ca-4c56-a53b-62e38beb64eb-kube-api-access-vn8hg\") pod \"e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:02.641979 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.641824 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-model-cache\") pod \"e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:02.641979 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.641911 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-dshm\") pod \"e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:02.641979 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.641952 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-tmp-dir\") pod \"e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:02.642119 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.641981 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48d1143f-53ca-4c56-a53b-62e38beb64eb-tls-certs\") pod \"e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:02.642119 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.642014 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-home\") pod \"e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:02.743610 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.743557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-model-cache\") pod \"e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:02.743809 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.743659 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-dshm\") pod \"e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:02.743809 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.743699 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-tmp-dir\") pod \"e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:02.743809 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.743726 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48d1143f-53ca-4c56-a53b-62e38beb64eb-tls-certs\") pod \"e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:02.743809 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.743758 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-home\") pod \"e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:02.744062 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.743894 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn8hg\" (UniqueName: \"kubernetes.io/projected/48d1143f-53ca-4c56-a53b-62e38beb64eb-kube-api-access-vn8hg\") pod \"e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:02.744123 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.744107 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-tmp-dir\") pod \"e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:02.744181 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.744127 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-model-cache\") pod \"e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:02.744363 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.744342 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-home\") pod \"e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:02.746444 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.746407 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-dshm\") pod \"e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:02.746624 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.746509 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48d1143f-53ca-4c56-a53b-62e38beb64eb-tls-certs\") pod \"e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:02.765733 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.765706 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn8hg\" (UniqueName: \"kubernetes.io/projected/48d1143f-53ca-4c56-a53b-62e38beb64eb-kube-api-access-vn8hg\") pod \"e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:02.811549 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:02.811513 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:03.091893 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:04:03.091858 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48d1143f_53ca_4c56_a53b_62e38beb64eb.slice/crio-3702ac3a981acb907d5b72c2c8d9f221e84c398efd7d15b28485bf7e07d6f4ca WatchSource:0}: Error finding container 3702ac3a981acb907d5b72c2c8d9f221e84c398efd7d15b28485bf7e07d6f4ca: Status 404 returned error can't find the container with id 3702ac3a981acb907d5b72c2c8d9f221e84c398efd7d15b28485bf7e07d6f4ca Sep 11 17:04:03.095378 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:03.095353 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd"] Sep 11 17:04:03.472958 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:03.472846 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" event={"ID":"48d1143f-53ca-4c56-a53b-62e38beb64eb","Type":"ContainerStarted","Data":"3adc0acba5d866c7cbb86df28a604f545072cbd96d1e8a6240f305d7fa5bb3fb"} Sep 11 17:04:03.472958 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:03.472890 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" event={"ID":"48d1143f-53ca-4c56-a53b-62e38beb64eb","Type":"ContainerStarted","Data":"3702ac3a981acb907d5b72c2c8d9f221e84c398efd7d15b28485bf7e07d6f4ca"} Sep 11 17:04:03.501507 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:03.501453 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" podStartSLOduration=1.5014343220000002 podStartE2EDuration="1.501434322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:04:03.499665874 +0000 UTC m=+1258.907768174" watchObservedRunningTime="2026-09-11 17:04:03.501434322 +0000 UTC m=+1258.909536637" Sep 11 17:04:07.780994 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:07.780959 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5"] Sep 11 17:04:07.785642 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:07.785620 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5" Sep 11 17:04:07.788540 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:07.788520 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2fae12\"" Sep 11 17:04:07.788748 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:07.788734 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2fae12-dockercfg-zhwpc\"" Sep 11 17:04:07.791967 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:07.791835 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb27089f-c888-4453-8bd4-591406ef170e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5\" (UID: \"eb27089f-c888-4453-8bd4-591406ef170e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5" Sep 11 17:04:07.791967 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:07.791951 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jh2x\" (UniqueName: \"kubernetes.io/projected/eb27089f-c888-4453-8bd4-591406ef170e-kube-api-access-5jh2x\") pod \"payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5\" (UID: \"eb27089f-c888-4453-8bd4-591406ef170e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5" Sep 11 17:04:07.795139 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:07.795115 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5"] Sep 11 17:04:07.835907 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:07.835867 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m"] Sep 11 17:04:07.839410 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:07.839392 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m" Sep 11 17:04:07.860081 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:07.860050 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m"] Sep 11 17:04:07.892695 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:07.892658 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb27089f-c888-4453-8bd4-591406ef170e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5\" (UID: \"eb27089f-c888-4453-8bd4-591406ef170e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5" Sep 11 17:04:07.892872 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:07.892719 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmmgl\" (UniqueName: \"kubernetes.io/projected/e7356d5c-3dd3-4a49-96f6-675eb840fbfe-kube-api-access-hmmgl\") pod \"payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m\" (UID: \"e7356d5c-3dd3-4a49-96f6-675eb840fbfe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m" Sep 11 17:04:07.892872 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:07.892782 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7356d5c-3dd3-4a49-96f6-675eb840fbfe-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m\" (UID: \"e7356d5c-3dd3-4a49-96f6-675eb840fbfe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m" Sep 11 17:04:07.892992 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:07.892874 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jh2x\" (UniqueName: \"kubernetes.io/projected/eb27089f-c888-4453-8bd4-591406ef170e-kube-api-access-5jh2x\") pod \"payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5\" (UID: \"eb27089f-c888-4453-8bd4-591406ef170e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5" Sep 11 17:04:07.893325 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:07.893304 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb27089f-c888-4453-8bd4-591406ef170e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5\" (UID: \"eb27089f-c888-4453-8bd4-591406ef170e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5" Sep 11 17:04:07.902520 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:07.902476 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jh2x\" (UniqueName: \"kubernetes.io/projected/eb27089f-c888-4453-8bd4-591406ef170e-kube-api-access-5jh2x\") pod \"payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5\" (UID: \"eb27089f-c888-4453-8bd4-591406ef170e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5" Sep 11 17:04:07.993597 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:07.993560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7356d5c-3dd3-4a49-96f6-675eb840fbfe-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m\" (UID: \"e7356d5c-3dd3-4a49-96f6-675eb840fbfe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m" Sep 11 17:04:07.993782 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:07.993649 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmmgl\" (UniqueName: \"kubernetes.io/projected/e7356d5c-3dd3-4a49-96f6-675eb840fbfe-kube-api-access-hmmgl\") pod \"payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m\" (UID: \"e7356d5c-3dd3-4a49-96f6-675eb840fbfe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m" Sep 11 17:04:07.994252 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:07.994231 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7356d5c-3dd3-4a49-96f6-675eb840fbfe-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m\" (UID: \"e7356d5c-3dd3-4a49-96f6-675eb840fbfe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m" Sep 11 17:04:08.015377 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:08.015348 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmmgl\" (UniqueName: \"kubernetes.io/projected/e7356d5c-3dd3-4a49-96f6-675eb840fbfe-kube-api-access-hmmgl\") pod \"payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m\" (UID: \"e7356d5c-3dd3-4a49-96f6-675eb840fbfe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m" Sep 11 17:04:08.096613 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:08.096488 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5" Sep 11 17:04:08.149840 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:08.149391 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m" Sep 11 17:04:08.374864 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:08.374836 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5"] Sep 11 17:04:08.376311 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:04:08.376287 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb27089f_c888_4453_8bd4_591406ef170e.slice/crio-285bc629482d8e8eb9102a2e75e8cba87581f55528b5202c986afe7296794770 WatchSource:0}: Error finding container 285bc629482d8e8eb9102a2e75e8cba87581f55528b5202c986afe7296794770: Status 404 returned error can't find the container with id 285bc629482d8e8eb9102a2e75e8cba87581f55528b5202c986afe7296794770 Sep 11 17:04:08.437713 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:08.437687 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m"] Sep 11 17:04:08.438988 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:04:08.438964 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7356d5c_3dd3_4a49_96f6_675eb840fbfe.slice/crio-cd8b8233743cb5e4e7b269880e033d9508580e9e00db011609e489334ed9170e WatchSource:0}: Error finding container cd8b8233743cb5e4e7b269880e033d9508580e9e00db011609e489334ed9170e: Status 404 returned error can't find the container with id cd8b8233743cb5e4e7b269880e033d9508580e9e00db011609e489334ed9170e Sep 11 17:04:08.502430 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:08.502397 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5" event={"ID":"eb27089f-c888-4453-8bd4-591406ef170e","Type":"ContainerStarted","Data":"661ea18016f0b27c46b500cb909485f7be3ef67f96ada47b07c7dd61d2b9c3e3"} Sep 11 17:04:08.502577 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:08.502469 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5" Sep 11 17:04:08.502577 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:08.502521 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5" event={"ID":"eb27089f-c888-4453-8bd4-591406ef170e","Type":"ContainerStarted","Data":"285bc629482d8e8eb9102a2e75e8cba87581f55528b5202c986afe7296794770"} Sep 11 17:04:08.504041 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:08.504020 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m" event={"ID":"e7356d5c-3dd3-4a49-96f6-675eb840fbfe","Type":"ContainerStarted","Data":"f157c6d41193ed09a956270ae850c2993c945ebd9ce79da6453426af5d8abe5b"} Sep 11 17:04:08.504147 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:08.504064 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m" event={"ID":"e7356d5c-3dd3-4a49-96f6-675eb840fbfe","Type":"ContainerStarted","Data":"cd8b8233743cb5e4e7b269880e033d9508580e9e00db011609e489334ed9170e"} Sep 11 17:04:08.504147 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:08.504087 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m" Sep 11 17:04:08.526721 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:08.526668 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5" podStartSLOduration=1.526645878 podStartE2EDuration="1.526645878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:04:08.525064662 +0000 UTC m=+1263.933166957" watchObservedRunningTime="2026-09-11 17:04:08.526645878 +0000 UTC m=+1263.934748149" Sep 11 17:04:08.555308 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:08.548554 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m" podStartSLOduration=1.5485318970000002 podStartE2EDuration="1.548531897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:04:08.547544389 +0000 UTC m=+1263.955646669" watchObservedRunningTime="2026-09-11 17:04:08.548531897 +0000 UTC m=+1263.956634167" Sep 11 17:04:12.812159 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:12.812114 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:12.812620 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:12.812256 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:12.827987 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:12.827959 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:13.543337 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:13.543305 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:04:19.512392 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:19.512356 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5" Sep 11 17:04:19.512792 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:19.512531 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m" Sep 11 17:04:42.833192 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:42.833158 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d"] Sep 11 17:04:42.838402 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:42.838376 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d" Sep 11 17:04:42.841398 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:42.841373 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-327a3a-dockercfg-xc57x\"" Sep 11 17:04:42.841842 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:42.841822 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-327a3a\"" Sep 11 17:04:42.847364 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:42.846902 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d"] Sep 11 17:04:42.860671 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:42.860643 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj"] Sep 11 17:04:42.864313 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:42.864291 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj" Sep 11 17:04:42.873628 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:42.873602 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj"] Sep 11 17:04:42.938485 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:42.938448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199e0b37-0bca-40f7-ac6a-31370fe0ec47-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d\" (UID: \"199e0b37-0bca-40f7-ac6a-31370fe0ec47\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d" Sep 11 17:04:42.938663 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:42.938556 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bkc6\" (UniqueName: \"kubernetes.io/projected/199e0b37-0bca-40f7-ac6a-31370fe0ec47-kube-api-access-7bkc6\") pod \"payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d\" (UID: \"199e0b37-0bca-40f7-ac6a-31370fe0ec47\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d" Sep 11 17:04:43.039994 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.039958 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c19eeac-7303-41ce-a036-f426073c85a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj\" (UID: \"2c19eeac-7303-41ce-a036-f426073c85a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj" Sep 11 17:04:43.040161 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.040016 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfzkr\" (UniqueName: \"kubernetes.io/projected/2c19eeac-7303-41ce-a036-f426073c85a7-kube-api-access-nfzkr\") pod \"payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj\" (UID: \"2c19eeac-7303-41ce-a036-f426073c85a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj" Sep 11 17:04:43.040161 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.040126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199e0b37-0bca-40f7-ac6a-31370fe0ec47-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d\" (UID: \"199e0b37-0bca-40f7-ac6a-31370fe0ec47\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d" Sep 11 17:04:43.040246 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.040200 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bkc6\" (UniqueName: \"kubernetes.io/projected/199e0b37-0bca-40f7-ac6a-31370fe0ec47-kube-api-access-7bkc6\") pod \"payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d\" (UID: \"199e0b37-0bca-40f7-ac6a-31370fe0ec47\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d" Sep 11 17:04:43.040832 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.040815 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199e0b37-0bca-40f7-ac6a-31370fe0ec47-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d\" (UID: \"199e0b37-0bca-40f7-ac6a-31370fe0ec47\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d" Sep 11 17:04:43.050365 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.050340 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bkc6\" (UniqueName: \"kubernetes.io/projected/199e0b37-0bca-40f7-ac6a-31370fe0ec47-kube-api-access-7bkc6\") pod \"payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d\" (UID: \"199e0b37-0bca-40f7-ac6a-31370fe0ec47\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d" Sep 11 17:04:43.141725 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.141640 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c19eeac-7303-41ce-a036-f426073c85a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj\" (UID: \"2c19eeac-7303-41ce-a036-f426073c85a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj" Sep 11 17:04:43.141725 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.141693 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfzkr\" (UniqueName: \"kubernetes.io/projected/2c19eeac-7303-41ce-a036-f426073c85a7-kube-api-access-nfzkr\") pod \"payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj\" (UID: \"2c19eeac-7303-41ce-a036-f426073c85a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj" Sep 11 17:04:43.142397 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.142376 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c19eeac-7303-41ce-a036-f426073c85a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj\" (UID: \"2c19eeac-7303-41ce-a036-f426073c85a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj" Sep 11 17:04:43.150874 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.150851 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfzkr\" (UniqueName: \"kubernetes.io/projected/2c19eeac-7303-41ce-a036-f426073c85a7-kube-api-access-nfzkr\") pod \"payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj\" (UID: \"2c19eeac-7303-41ce-a036-f426073c85a7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj" Sep 11 17:04:43.150978 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.150965 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d" Sep 11 17:04:43.176946 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.176919 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj" Sep 11 17:04:43.427662 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.427633 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d"] Sep 11 17:04:43.429175 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:04:43.429145 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199e0b37_0bca_40f7_ac6a_31370fe0ec47.slice/crio-24935a95c4d93460cf807af5888ac9f097a12920ff0273977de8b5b5d53bf9d6 WatchSource:0}: Error finding container 24935a95c4d93460cf807af5888ac9f097a12920ff0273977de8b5b5d53bf9d6: Status 404 returned error can't find the container with id 24935a95c4d93460cf807af5888ac9f097a12920ff0273977de8b5b5d53bf9d6 Sep 11 17:04:43.465070 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.465043 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj"] Sep 11 17:04:43.467063 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:04:43.467034 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c19eeac_7303_41ce_a036_f426073c85a7.slice/crio-a2f37e948d34e13e0cfdc3059fd14d00a4b365db39005b07fd3f20fa452e8467 WatchSource:0}: Error finding container a2f37e948d34e13e0cfdc3059fd14d00a4b365db39005b07fd3f20fa452e8467: Status 404 returned error can't find the container with id a2f37e948d34e13e0cfdc3059fd14d00a4b365db39005b07fd3f20fa452e8467 Sep 11 17:04:43.713457 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.713354 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj" event={"ID":"2c19eeac-7303-41ce-a036-f426073c85a7","Type":"ContainerStarted","Data":"f3f50f2c16b4070a67d22d9bda8d62058fbdfc40b04ea7feb35b3715f8d66361"} Sep 11 17:04:43.713457 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.713422 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj" Sep 11 17:04:43.713687 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.713486 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj" event={"ID":"2c19eeac-7303-41ce-a036-f426073c85a7","Type":"ContainerStarted","Data":"a2f37e948d34e13e0cfdc3059fd14d00a4b365db39005b07fd3f20fa452e8467"} Sep 11 17:04:43.715035 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.715008 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d" event={"ID":"199e0b37-0bca-40f7-ac6a-31370fe0ec47","Type":"ContainerStarted","Data":"3b42e4844e3f4012a338f2bbfc0a4fca4bdfbed9dfd8c1c85464937ca89701f3"} Sep 11 17:04:43.715152 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.715056 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d" event={"ID":"199e0b37-0bca-40f7-ac6a-31370fe0ec47","Type":"ContainerStarted","Data":"24935a95c4d93460cf807af5888ac9f097a12920ff0273977de8b5b5d53bf9d6"} Sep 11 17:04:43.715152 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.715132 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d" Sep 11 17:04:43.733356 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.733299 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj" podStartSLOduration=1.733275387 podStartE2EDuration="1.733275387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:04:43.730572679 +0000 UTC m=+1299.138674969" watchObservedRunningTime="2026-09-11 17:04:43.733275387 +0000 UTC m=+1299.141377664" Sep 11 17:04:43.750968 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:43.750916 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d" podStartSLOduration=1.750900599 podStartE2EDuration="1.750900599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:04:43.750168364 +0000 UTC m=+1299.158270657" watchObservedRunningTime="2026-09-11 17:04:43.750900599 +0000 UTC m=+1299.159002868" Sep 11 17:04:54.725104 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:54.725069 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj" Sep 11 17:04:54.725524 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:04:54.725149 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d" Sep 11 17:05:05.198907 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:05.198869 2577 scope.go:117] "RemoveContainer" containerID="a3bb7fe2c80d50b8ab23566b47c0675025a653bc41429c6871432fd4209cd031" Sep 11 17:05:11.114744 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:11.114707 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4"] Sep 11 17:05:11.115213 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:11.114966 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4" podUID="62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5" containerName="payload-pre-processing" containerID="cri-o://4d7a6e994ed13611cc1e729707e352f298a646dfc3062c8d714804d61870e736" gracePeriod=30 Sep 11 17:05:11.398381 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:11.398352 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4" Sep 11 17:05:11.524804 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:11.524761 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxpl4\" (UniqueName: \"kubernetes.io/projected/62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5-kube-api-access-nxpl4\") pod \"62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5\" (UID: \"62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5\") " Sep 11 17:05:11.525003 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:11.524896 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5-plugins-config-volume\") pod \"62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5\" (UID: \"62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5\") " Sep 11 17:05:11.525256 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:11.525226 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5" (UID: "62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:05:11.526910 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:11.526878 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5-kube-api-access-nxpl4" (OuterVolumeSpecName: "kube-api-access-nxpl4") pod "62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5" (UID: "62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5"). InnerVolumeSpecName "kube-api-access-nxpl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:05:11.626795 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:11.626693 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:05:11.626795 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:11.626733 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxpl4\" (UniqueName: \"kubernetes.io/projected/62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5-kube-api-access-nxpl4\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:05:11.875206 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:11.875172 2577 generic.go:356] "Generic (PLEG): container finished" podID="62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5" containerID="4d7a6e994ed13611cc1e729707e352f298a646dfc3062c8d714804d61870e736" exitCode=0 Sep 11 17:05:11.875402 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:11.875215 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4" Sep 11 17:05:11.875402 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:11.875222 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4" event={"ID":"62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5","Type":"ContainerDied","Data":"4d7a6e994ed13611cc1e729707e352f298a646dfc3062c8d714804d61870e736"} Sep 11 17:05:11.875402 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:11.875275 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4" event={"ID":"62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5","Type":"ContainerDied","Data":"004b61ffeb92f4fefee927d520228a1a968a7bce3385a118880b24629027b664"} Sep 11 17:05:11.875402 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:11.875303 2577 scope.go:117] "RemoveContainer" containerID="4d7a6e994ed13611cc1e729707e352f298a646dfc3062c8d714804d61870e736" Sep 11 17:05:11.884718 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:11.884700 2577 scope.go:117] "RemoveContainer" containerID="4d7a6e994ed13611cc1e729707e352f298a646dfc3062c8d714804d61870e736" Sep 11 17:05:11.885017 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:05:11.884995 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d7a6e994ed13611cc1e729707e352f298a646dfc3062c8d714804d61870e736\": container with ID starting with 4d7a6e994ed13611cc1e729707e352f298a646dfc3062c8d714804d61870e736 not found: ID does not exist" containerID="4d7a6e994ed13611cc1e729707e352f298a646dfc3062c8d714804d61870e736" Sep 11 17:05:11.885096 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:11.885025 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7a6e994ed13611cc1e729707e352f298a646dfc3062c8d714804d61870e736"} err="failed to get container status \"4d7a6e994ed13611cc1e729707e352f298a646dfc3062c8d714804d61870e736\": rpc error: code = NotFound desc = could not find container \"4d7a6e994ed13611cc1e729707e352f298a646dfc3062c8d714804d61870e736\": container with ID starting with 4d7a6e994ed13611cc1e729707e352f298a646dfc3062c8d714804d61870e736 not found: ID does not exist" Sep 11 17:05:11.906546 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:11.906511 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4"] Sep 11 17:05:11.912488 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:11.912454 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d422395-7bddff4f6b-sjwl4"] Sep 11 17:05:13.094071 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:13.094037 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5" path="/var/lib/kubelet/pods/62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5/volumes" Sep 11 17:05:19.577370 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:19.577330 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv"] Sep 11 17:05:34.596297 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:34.596237 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" podUID="ae09052e-2084-4af5-b678-b817133de228" containerName="main" containerID="cri-o://1dcdc61d61a17cb2681034aa5f12be68c3fc51456c63743994d2e76d6b01ddc9" gracePeriod=45 Sep 11 17:05:34.878713 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:34.878680 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:05:34.964612 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:34.964575 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-dshm\") pod \"ae09052e-2084-4af5-b678-b817133de228\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " Sep 11 17:05:34.964799 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:34.964627 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-tmp-dir\") pod \"ae09052e-2084-4af5-b678-b817133de228\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " Sep 11 17:05:34.964799 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:34.964668 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnjbr\" (UniqueName: \"kubernetes.io/projected/ae09052e-2084-4af5-b678-b817133de228-kube-api-access-tnjbr\") pod \"ae09052e-2084-4af5-b678-b817133de228\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " Sep 11 17:05:34.964799 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:34.964705 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-model-cache\") pod \"ae09052e-2084-4af5-b678-b817133de228\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " Sep 11 17:05:34.964799 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:34.964751 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae09052e-2084-4af5-b678-b817133de228-tls-certs\") pod \"ae09052e-2084-4af5-b678-b817133de228\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " Sep 11 17:05:34.964799 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:34.964793 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-home\") pod \"ae09052e-2084-4af5-b678-b817133de228\" (UID: \"ae09052e-2084-4af5-b678-b817133de228\") " Sep 11 17:05:34.965084 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:34.964884 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae09052e-2084-4af5-b678-b817133de228" (UID: "ae09052e-2084-4af5-b678-b817133de228"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:05:34.965149 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:34.965132 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-home" (OuterVolumeSpecName: "home") pod "ae09052e-2084-4af5-b678-b817133de228" (UID: "ae09052e-2084-4af5-b678-b817133de228"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:05:34.965266 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:34.965240 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae09052e-2084-4af5-b678-b817133de228" (UID: "ae09052e-2084-4af5-b678-b817133de228"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:05:34.965716 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:34.965650 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-tmp-dir\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:05:34.965716 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:34.965677 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-model-cache\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:05:34.965716 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:34.965691 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-home\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:05:34.967238 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:34.967203 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae09052e-2084-4af5-b678-b817133de228-kube-api-access-tnjbr" (OuterVolumeSpecName: "kube-api-access-tnjbr") pod "ae09052e-2084-4af5-b678-b817133de228" (UID: "ae09052e-2084-4af5-b678-b817133de228"). InnerVolumeSpecName "kube-api-access-tnjbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:05:34.967238 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:34.967215 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-dshm" (OuterVolumeSpecName: "dshm") pod "ae09052e-2084-4af5-b678-b817133de228" (UID: "ae09052e-2084-4af5-b678-b817133de228"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:05:34.967464 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:34.967446 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae09052e-2084-4af5-b678-b817133de228-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae09052e-2084-4af5-b678-b817133de228" (UID: "ae09052e-2084-4af5-b678-b817133de228"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 17:05:35.006675 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:35.006642 2577 generic.go:356] "Generic (PLEG): container finished" podID="ae09052e-2084-4af5-b678-b817133de228" containerID="1dcdc61d61a17cb2681034aa5f12be68c3fc51456c63743994d2e76d6b01ddc9" exitCode=0 Sep 11 17:05:35.006836 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:35.006687 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" Sep 11 17:05:35.006836 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:35.006723 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" event={"ID":"ae09052e-2084-4af5-b678-b817133de228","Type":"ContainerDied","Data":"1dcdc61d61a17cb2681034aa5f12be68c3fc51456c63743994d2e76d6b01ddc9"} Sep 11 17:05:35.006836 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:35.006766 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv" event={"ID":"ae09052e-2084-4af5-b678-b817133de228","Type":"ContainerDied","Data":"3043923f152b52b76d58dda41770cc60504b3c72c22b5a6215743d720fd0f230"} Sep 11 17:05:35.006836 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:35.006784 2577 scope.go:117] "RemoveContainer" containerID="1dcdc61d61a17cb2681034aa5f12be68c3fc51456c63743994d2e76d6b01ddc9" Sep 11 17:05:35.018532 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:35.016806 2577 scope.go:117] "RemoveContainer" containerID="1dcdc61d61a17cb2681034aa5f12be68c3fc51456c63743994d2e76d6b01ddc9" Sep 11 17:05:35.020945 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:05:35.020908 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dcdc61d61a17cb2681034aa5f12be68c3fc51456c63743994d2e76d6b01ddc9\": container with ID starting with 1dcdc61d61a17cb2681034aa5f12be68c3fc51456c63743994d2e76d6b01ddc9 not found: ID does not exist" containerID="1dcdc61d61a17cb2681034aa5f12be68c3fc51456c63743994d2e76d6b01ddc9" Sep 11 17:05:35.021077 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:35.020957 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dcdc61d61a17cb2681034aa5f12be68c3fc51456c63743994d2e76d6b01ddc9"} err="failed to get container status \"1dcdc61d61a17cb2681034aa5f12be68c3fc51456c63743994d2e76d6b01ddc9\": rpc error: code = NotFound desc = could not find container \"1dcdc61d61a17cb2681034aa5f12be68c3fc51456c63743994d2e76d6b01ddc9\": container with ID starting with 1dcdc61d61a17cb2681034aa5f12be68c3fc51456c63743994d2e76d6b01ddc9 not found: ID does not exist" Sep 11 17:05:35.036683 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:35.036653 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv"] Sep 11 17:05:35.041476 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:35.041448 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-6d422395/e2e-lifecycle-model-6d422395-kserve-676b5d7fdc-8s7lv"] Sep 11 17:05:35.066809 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:35.066782 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae09052e-2084-4af5-b678-b817133de228-dshm\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:05:35.066809 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:35.066809 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnjbr\" (UniqueName: \"kubernetes.io/projected/ae09052e-2084-4af5-b678-b817133de228-kube-api-access-tnjbr\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:05:35.066983 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:35.066819 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae09052e-2084-4af5-b678-b817133de228-tls-certs\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:05:35.094229 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:35.094200 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae09052e-2084-4af5-b678-b817133de228" path="/var/lib/kubelet/pods/ae09052e-2084-4af5-b678-b817133de228/volumes" Sep 11 17:05:37.765806 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:37.765771 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd"] Sep 11 17:05:52.785159 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:52.785113 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" podUID="48d1143f-53ca-4c56-a53b-62e38beb64eb" containerName="main" containerID="cri-o://3adc0acba5d866c7cbb86df28a604f545072cbd96d1e8a6240f305d7fa5bb3fb" gracePeriod=45 Sep 11 17:05:53.037473 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.037409 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:05:53.111794 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.111763 2577 generic.go:356] "Generic (PLEG): container finished" podID="48d1143f-53ca-4c56-a53b-62e38beb64eb" containerID="3adc0acba5d866c7cbb86df28a604f545072cbd96d1e8a6240f305d7fa5bb3fb" exitCode=0 Sep 11 17:05:53.111988 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.111804 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" Sep 11 17:05:53.111988 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.111843 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" event={"ID":"48d1143f-53ca-4c56-a53b-62e38beb64eb","Type":"ContainerDied","Data":"3adc0acba5d866c7cbb86df28a604f545072cbd96d1e8a6240f305d7fa5bb3fb"} Sep 11 17:05:53.111988 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.111892 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd" event={"ID":"48d1143f-53ca-4c56-a53b-62e38beb64eb","Type":"ContainerDied","Data":"3702ac3a981acb907d5b72c2c8d9f221e84c398efd7d15b28485bf7e07d6f4ca"} Sep 11 17:05:53.111988 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.111912 2577 scope.go:117] "RemoveContainer" containerID="3adc0acba5d866c7cbb86df28a604f545072cbd96d1e8a6240f305d7fa5bb3fb" Sep 11 17:05:53.121462 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.121442 2577 scope.go:117] "RemoveContainer" containerID="3adc0acba5d866c7cbb86df28a604f545072cbd96d1e8a6240f305d7fa5bb3fb" Sep 11 17:05:53.121805 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:05:53.121786 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3adc0acba5d866c7cbb86df28a604f545072cbd96d1e8a6240f305d7fa5bb3fb\": container with ID starting with 3adc0acba5d866c7cbb86df28a604f545072cbd96d1e8a6240f305d7fa5bb3fb not found: ID does not exist" containerID="3adc0acba5d866c7cbb86df28a604f545072cbd96d1e8a6240f305d7fa5bb3fb" Sep 11 17:05:53.121883 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.121815 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3adc0acba5d866c7cbb86df28a604f545072cbd96d1e8a6240f305d7fa5bb3fb"} err="failed to get container status \"3adc0acba5d866c7cbb86df28a604f545072cbd96d1e8a6240f305d7fa5bb3fb\": rpc error: code = NotFound desc = could not find container \"3adc0acba5d866c7cbb86df28a604f545072cbd96d1e8a6240f305d7fa5bb3fb\": container with ID starting with 3adc0acba5d866c7cbb86df28a604f545072cbd96d1e8a6240f305d7fa5bb3fb not found: ID does not exist" Sep 11 17:05:53.134146 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.134123 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-tmp-dir\") pod \"48d1143f-53ca-4c56-a53b-62e38beb64eb\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " Sep 11 17:05:53.134277 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.134185 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-model-cache\") pod \"48d1143f-53ca-4c56-a53b-62e38beb64eb\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " Sep 11 17:05:53.134277 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.134203 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-home\") pod \"48d1143f-53ca-4c56-a53b-62e38beb64eb\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " Sep 11 17:05:53.134277 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.134228 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48d1143f-53ca-4c56-a53b-62e38beb64eb-tls-certs\") pod \"48d1143f-53ca-4c56-a53b-62e38beb64eb\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " Sep 11 17:05:53.134452 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.134308 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vn8hg\" (UniqueName: \"kubernetes.io/projected/48d1143f-53ca-4c56-a53b-62e38beb64eb-kube-api-access-vn8hg\") pod \"48d1143f-53ca-4c56-a53b-62e38beb64eb\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " Sep 11 17:05:53.134452 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.134324 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-dshm\") pod \"48d1143f-53ca-4c56-a53b-62e38beb64eb\" (UID: \"48d1143f-53ca-4c56-a53b-62e38beb64eb\") " Sep 11 17:05:53.134452 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.134374 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48d1143f-53ca-4c56-a53b-62e38beb64eb" (UID: "48d1143f-53ca-4c56-a53b-62e38beb64eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:05:53.134452 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.134440 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "48d1143f-53ca-4c56-a53b-62e38beb64eb" (UID: "48d1143f-53ca-4c56-a53b-62e38beb64eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:05:53.134659 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.134462 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-home" (OuterVolumeSpecName: "home") pod "48d1143f-53ca-4c56-a53b-62e38beb64eb" (UID: "48d1143f-53ca-4c56-a53b-62e38beb64eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:05:53.137044 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.135686 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-tmp-dir\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:05:53.137044 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.135718 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-model-cache\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:05:53.137044 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.135730 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-home\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:05:53.138542 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.137364 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-dshm" (OuterVolumeSpecName: "dshm") pod "48d1143f-53ca-4c56-a53b-62e38beb64eb" (UID: "48d1143f-53ca-4c56-a53b-62e38beb64eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:05:53.138542 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.137414 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48d1143f-53ca-4c56-a53b-62e38beb64eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48d1143f-53ca-4c56-a53b-62e38beb64eb" (UID: "48d1143f-53ca-4c56-a53b-62e38beb64eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 17:05:53.138542 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.137662 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d1143f-53ca-4c56-a53b-62e38beb64eb-kube-api-access-vn8hg" (OuterVolumeSpecName: "kube-api-access-vn8hg") pod "48d1143f-53ca-4c56-a53b-62e38beb64eb" (UID: "48d1143f-53ca-4c56-a53b-62e38beb64eb"). InnerVolumeSpecName "kube-api-access-vn8hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:05:53.236693 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.236651 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48d1143f-53ca-4c56-a53b-62e38beb64eb-tls-certs\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:05:53.236693 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.236690 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn8hg\" (UniqueName: \"kubernetes.io/projected/48d1143f-53ca-4c56-a53b-62e38beb64eb-kube-api-access-vn8hg\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:05:53.236693 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.236703 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48d1143f-53ca-4c56-a53b-62e38beb64eb-dshm\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:05:53.437339 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.437306 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd"] Sep 11 17:05:53.444644 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:53.444617 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2dc95c/e2e-conflict-b-44a0e379-kserve-7cd68f496d-llwhd"] Sep 11 17:05:55.107516 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:05:55.107471 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48d1143f-53ca-4c56-a53b-62e38beb64eb" path="/var/lib/kubelet/pods/48d1143f-53ca-4c56-a53b-62e38beb64eb/volumes" Sep 11 17:06:05.222198 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:05.222160 2577 scope.go:117] "RemoveContainer" containerID="d64001afd40dc4d36ee1585cadf3a681eb842e94777703c921662bc1eeb28ab7" Sep 11 17:06:05.231052 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:05.231027 2577 scope.go:117] "RemoveContainer" containerID="9c0df5e499f74c8c7d7c4daeaeaac3ae18da72b980d4a70b70ac54d3d37837ee" Sep 11 17:06:20.457365 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.457329 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv"] Sep 11 17:06:20.457809 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.457794 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48d1143f-53ca-4c56-a53b-62e38beb64eb" containerName="main" Sep 11 17:06:20.457858 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.457811 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="48d1143f-53ca-4c56-a53b-62e38beb64eb" containerName="main" Sep 11 17:06:20.457858 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.457837 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5" containerName="payload-pre-processing" Sep 11 17:06:20.457858 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.457843 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5" containerName="payload-pre-processing" Sep 11 17:06:20.457858 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.457854 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae09052e-2084-4af5-b678-b817133de228" containerName="main" Sep 11 17:06:20.457858 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.457859 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae09052e-2084-4af5-b678-b817133de228" containerName="main" Sep 11 17:06:20.458035 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.457942 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae09052e-2084-4af5-b678-b817133de228" containerName="main" Sep 11 17:06:20.458035 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.457954 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="48d1143f-53ca-4c56-a53b-62e38beb64eb" containerName="main" Sep 11 17:06:20.458035 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.457963 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="62ca71b6-109c-4cd7-bf32-ac6f5bd7a5f5" containerName="payload-pre-processing" Sep 11 17:06:20.462662 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.462643 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:20.467747 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.467705 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2dc95c\"/\"distinct-w4-2dc95c-kserve-self-signed-certs\"" Sep 11 17:06:20.467926 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.467771 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2dc95c\"/\"openshift-service-ca.crt\"" Sep 11 17:06:20.467926 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.467796 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2dc95c\"/\"default-dockercfg-rnfvr\"" Sep 11 17:06:20.467926 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.467801 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2dc95c\"/\"kube-root-ca.crt\"" Sep 11 17:06:20.475717 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.475691 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv"] Sep 11 17:06:20.595255 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.595221 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-dshm\") pod \"distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:20.595424 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.595265 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-home\") pod \"distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:20.595424 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.595301 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkfrs\" (UniqueName: \"kubernetes.io/projected/ffc657f2-91a2-4b62-a387-ca98962e4bd5-kube-api-access-kkfrs\") pod \"distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:20.595424 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.595344 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc657f2-91a2-4b62-a387-ca98962e4bd5-tls-certs\") pod \"distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:20.595424 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.595383 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-tmp-dir\") pod \"distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:20.595424 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.595414 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-model-cache\") pod \"distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:20.696759 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.696720 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-home\") pod \"distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:20.696939 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.696771 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkfrs\" (UniqueName: \"kubernetes.io/projected/ffc657f2-91a2-4b62-a387-ca98962e4bd5-kube-api-access-kkfrs\") pod \"distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:20.696939 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.696819 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc657f2-91a2-4b62-a387-ca98962e4bd5-tls-certs\") pod \"distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:20.696939 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.696854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-tmp-dir\") pod \"distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:20.696939 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.696894 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-model-cache\") pod \"distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:20.697151 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.696993 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-dshm\") pod \"distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:20.697210 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.697176 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-home\") pod \"distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:20.697272 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.697250 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-tmp-dir\") pod \"distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:20.697368 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.697345 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-model-cache\") pod \"distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:20.699294 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.699268 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-dshm\") pod \"distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:20.699460 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.699447 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc657f2-91a2-4b62-a387-ca98962e4bd5-tls-certs\") pod \"distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:20.708865 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.708805 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkfrs\" (UniqueName: \"kubernetes.io/projected/ffc657f2-91a2-4b62-a387-ca98962e4bd5-kube-api-access-kkfrs\") pod \"distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:20.772642 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:20.772604 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:21.054704 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:21.054677 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv"] Sep 11 17:06:21.056693 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:06:21.056662 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffc657f2_91a2_4b62_a387_ca98962e4bd5.slice/crio-83ed7900d8c48c020f0b148a74df28fdf8b80796b713ff3b42ce048a177c8c57 WatchSource:0}: Error finding container 83ed7900d8c48c020f0b148a74df28fdf8b80796b713ff3b42ce048a177c8c57: Status 404 returned error can't find the container with id 83ed7900d8c48c020f0b148a74df28fdf8b80796b713ff3b42ce048a177c8c57 Sep 11 17:06:21.266668 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:21.266630 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" event={"ID":"ffc657f2-91a2-4b62-a387-ca98962e4bd5","Type":"ContainerStarted","Data":"0f9fc01d08e7c5d56d1f89104aae6fd700d702281a89bea5c6a9291a08af0cc4"} Sep 11 17:06:21.266668 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:21.266669 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" event={"ID":"ffc657f2-91a2-4b62-a387-ca98962e4bd5","Type":"ContainerStarted","Data":"83ed7900d8c48c020f0b148a74df28fdf8b80796b713ff3b42ce048a177c8c57"} Sep 11 17:06:21.287126 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:21.287074 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" podStartSLOduration=1.28705639 podStartE2EDuration="1.28705639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:06:21.286294502 +0000 UTC m=+1396.694396800" watchObservedRunningTime="2026-09-11 17:06:21.28705639 +0000 UTC m=+1396.695158659" Sep 11 17:06:30.773053 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:30.773017 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:30.773527 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:30.773075 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:30.785609 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:30.785583 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:31.330119 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:31.330089 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:06:49.096951 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.096919 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6"] Sep 11 17:06:49.100956 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.100931 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.105380 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.105354 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-327a3a\"/\"kube-root-ca.crt\"" Sep 11 17:06:49.105534 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.105388 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-327a3a\"/\"openshift-service-ca.crt\"" Sep 11 17:06:49.105534 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.105409 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-327a3a\"/\"default-dockercfg-5rzw7\"" Sep 11 17:06:49.105534 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.105461 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-327a3a\"/\"auth-test-model-327a3a-kserve-self-signed-certs\"" Sep 11 17:06:49.111263 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.111242 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6"] Sep 11 17:06:49.179101 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.179070 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-home\") pod \"auth-test-model-327a3a-kserve-6769cb45d7-z5nk6\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.179291 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.179115 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkl27\" (UniqueName: \"kubernetes.io/projected/00d5f85c-1121-494f-879e-c0989cf91231-kube-api-access-wkl27\") pod \"auth-test-model-327a3a-kserve-6769cb45d7-z5nk6\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.179291 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.179193 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-model-cache\") pod \"auth-test-model-327a3a-kserve-6769cb45d7-z5nk6\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.179291 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.179217 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-dshm\") pod \"auth-test-model-327a3a-kserve-6769cb45d7-z5nk6\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.179291 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.179258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-tmp-dir\") pod \"auth-test-model-327a3a-kserve-6769cb45d7-z5nk6\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.179542 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.179433 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d5f85c-1121-494f-879e-c0989cf91231-tls-certs\") pod \"auth-test-model-327a3a-kserve-6769cb45d7-z5nk6\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.280822 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.280779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d5f85c-1121-494f-879e-c0989cf91231-tls-certs\") pod \"auth-test-model-327a3a-kserve-6769cb45d7-z5nk6\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.281012 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.280835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-home\") pod \"auth-test-model-327a3a-kserve-6769cb45d7-z5nk6\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.281012 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.280872 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkl27\" (UniqueName: \"kubernetes.io/projected/00d5f85c-1121-494f-879e-c0989cf91231-kube-api-access-wkl27\") pod \"auth-test-model-327a3a-kserve-6769cb45d7-z5nk6\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.281012 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.280921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-model-cache\") pod \"auth-test-model-327a3a-kserve-6769cb45d7-z5nk6\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.281190 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.281030 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-dshm\") pod \"auth-test-model-327a3a-kserve-6769cb45d7-z5nk6\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.281190 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.281085 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-tmp-dir\") pod \"auth-test-model-327a3a-kserve-6769cb45d7-z5nk6\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.281391 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.281369 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-home\") pod \"auth-test-model-327a3a-kserve-6769cb45d7-z5nk6\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.281490 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.281469 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-model-cache\") pod \"auth-test-model-327a3a-kserve-6769cb45d7-z5nk6\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.281567 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.281537 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-tmp-dir\") pod \"auth-test-model-327a3a-kserve-6769cb45d7-z5nk6\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.283255 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.283232 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-dshm\") pod \"auth-test-model-327a3a-kserve-6769cb45d7-z5nk6\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.283432 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.283412 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d5f85c-1121-494f-879e-c0989cf91231-tls-certs\") pod \"auth-test-model-327a3a-kserve-6769cb45d7-z5nk6\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.290977 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.290953 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkl27\" (UniqueName: \"kubernetes.io/projected/00d5f85c-1121-494f-879e-c0989cf91231-kube-api-access-wkl27\") pod \"auth-test-model-327a3a-kserve-6769cb45d7-z5nk6\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.412592 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.412512 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:49.685685 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:49.685660 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6"] Sep 11 17:06:49.687838 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:06:49.687801 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00d5f85c_1121_494f_879e_c0989cf91231.slice/crio-1604d028ec6c2cdea0d2964abd2bf897f561acd399e6a264fe24815836e83971 WatchSource:0}: Error finding container 1604d028ec6c2cdea0d2964abd2bf897f561acd399e6a264fe24815836e83971: Status 404 returned error can't find the container with id 1604d028ec6c2cdea0d2964abd2bf897f561acd399e6a264fe24815836e83971 Sep 11 17:06:50.421229 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:50.421188 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" event={"ID":"00d5f85c-1121-494f-879e-c0989cf91231","Type":"ContainerStarted","Data":"1edb8c8400c41dc375349405d74a0550d9e4886a132f430387b07c662c70fdc0"} Sep 11 17:06:50.421725 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:50.421242 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" event={"ID":"00d5f85c-1121-494f-879e-c0989cf91231","Type":"ContainerStarted","Data":"1604d028ec6c2cdea0d2964abd2bf897f561acd399e6a264fe24815836e83971"} Sep 11 17:06:50.439844 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:50.439792 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" podStartSLOduration=1.439775575 podStartE2EDuration="1.439775575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:06:50.438846021 +0000 UTC m=+1425.846948309" watchObservedRunningTime="2026-09-11 17:06:50.439775575 +0000 UTC m=+1425.847877843" Sep 11 17:06:59.413007 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:59.412921 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:59.413386 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:59.413003 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:59.430114 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:59.430082 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:06:59.480073 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:06:59.480045 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:07:05.266390 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:05.266357 2577 scope.go:117] "RemoveContainer" containerID="5ea3e3a15c0ab4206361ed32f3509a19ea23a21d91f7c8d816db3b6b1a5dd772" Sep 11 17:07:22.730709 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.730670 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n"] Sep 11 17:07:22.735966 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.735943 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n" Sep 11 17:07:22.738916 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.738887 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-935ef2-dockercfg-bx72z\"" Sep 11 17:07:22.739057 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.739001 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-935ef2\"" Sep 11 17:07:22.746011 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.745794 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n"] Sep 11 17:07:22.757315 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.757290 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24"] Sep 11 17:07:22.760817 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.760795 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24" Sep 11 17:07:22.782867 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.782842 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24"] Sep 11 17:07:22.810956 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.810927 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ad7566-a0fd-45e7-8b5c-bce5d664b417-plugins-config-volume\") pod \"payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24\" (UID: \"05ad7566-a0fd-45e7-8b5c-bce5d664b417\") " pod="openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24" Sep 11 17:07:22.811116 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.810986 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdg89\" (UniqueName: \"kubernetes.io/projected/ad288b40-fe10-4d49-8b2b-0660e6fe20f3-kube-api-access-bdg89\") pod \"payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n\" (UID: \"ad288b40-fe10-4d49-8b2b-0660e6fe20f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n" Sep 11 17:07:22.811116 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.811052 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad288b40-fe10-4d49-8b2b-0660e6fe20f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n\" (UID: \"ad288b40-fe10-4d49-8b2b-0660e6fe20f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n" Sep 11 17:07:22.811195 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.811152 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp2lz\" (UniqueName: \"kubernetes.io/projected/05ad7566-a0fd-45e7-8b5c-bce5d664b417-kube-api-access-cp2lz\") pod \"payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24\" (UID: \"05ad7566-a0fd-45e7-8b5c-bce5d664b417\") " pod="openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24" Sep 11 17:07:22.912361 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.912328 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp2lz\" (UniqueName: \"kubernetes.io/projected/05ad7566-a0fd-45e7-8b5c-bce5d664b417-kube-api-access-cp2lz\") pod \"payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24\" (UID: \"05ad7566-a0fd-45e7-8b5c-bce5d664b417\") " pod="openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24" Sep 11 17:07:22.912562 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.912402 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ad7566-a0fd-45e7-8b5c-bce5d664b417-plugins-config-volume\") pod \"payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24\" (UID: \"05ad7566-a0fd-45e7-8b5c-bce5d664b417\") " pod="openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24" Sep 11 17:07:22.912562 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.912472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdg89\" (UniqueName: \"kubernetes.io/projected/ad288b40-fe10-4d49-8b2b-0660e6fe20f3-kube-api-access-bdg89\") pod \"payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n\" (UID: \"ad288b40-fe10-4d49-8b2b-0660e6fe20f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n" Sep 11 17:07:22.912562 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.912525 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad288b40-fe10-4d49-8b2b-0660e6fe20f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n\" (UID: \"ad288b40-fe10-4d49-8b2b-0660e6fe20f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n" Sep 11 17:07:22.913141 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.913117 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ad7566-a0fd-45e7-8b5c-bce5d664b417-plugins-config-volume\") pod \"payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24\" (UID: \"05ad7566-a0fd-45e7-8b5c-bce5d664b417\") " pod="openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24" Sep 11 17:07:22.913269 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.913250 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad288b40-fe10-4d49-8b2b-0660e6fe20f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n\" (UID: \"ad288b40-fe10-4d49-8b2b-0660e6fe20f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n" Sep 11 17:07:22.928296 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.928263 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp2lz\" (UniqueName: \"kubernetes.io/projected/05ad7566-a0fd-45e7-8b5c-bce5d664b417-kube-api-access-cp2lz\") pod \"payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24\" (UID: \"05ad7566-a0fd-45e7-8b5c-bce5d664b417\") " pod="openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24" Sep 11 17:07:22.929161 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:22.929133 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdg89\" (UniqueName: \"kubernetes.io/projected/ad288b40-fe10-4d49-8b2b-0660e6fe20f3-kube-api-access-bdg89\") pod \"payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n\" (UID: \"ad288b40-fe10-4d49-8b2b-0660e6fe20f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n" Sep 11 17:07:23.050716 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:23.050674 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n" Sep 11 17:07:23.071990 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:23.071957 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24" Sep 11 17:07:23.328703 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:23.328672 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n"] Sep 11 17:07:23.330832 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:07:23.330806 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad288b40_fe10_4d49_8b2b_0660e6fe20f3.slice/crio-7c1a18a902e6377ef971a5b59218e88bf7b8f93682e897c365d6716ff3b86c5c WatchSource:0}: Error finding container 7c1a18a902e6377ef971a5b59218e88bf7b8f93682e897c365d6716ff3b86c5c: Status 404 returned error can't find the container with id 7c1a18a902e6377ef971a5b59218e88bf7b8f93682e897c365d6716ff3b86c5c Sep 11 17:07:23.372901 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:23.372878 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24"] Sep 11 17:07:23.374469 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:07:23.374443 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ad7566_a0fd_45e7_8b5c_bce5d664b417.slice/crio-cc658826fb9d77998008dc040849465e8995451d15fc854e5c78940722af11b9 WatchSource:0}: Error finding container cc658826fb9d77998008dc040849465e8995451d15fc854e5c78940722af11b9: Status 404 returned error can't find the container with id cc658826fb9d77998008dc040849465e8995451d15fc854e5c78940722af11b9 Sep 11 17:07:23.604864 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:23.604776 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24" event={"ID":"05ad7566-a0fd-45e7-8b5c-bce5d664b417","Type":"ContainerStarted","Data":"43cada8a1f87e9ee2eb66680f59c68716509471aa4e57c3ef61d332f82c7b034"} Sep 11 17:07:23.604864 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:23.604831 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24" event={"ID":"05ad7566-a0fd-45e7-8b5c-bce5d664b417","Type":"ContainerStarted","Data":"cc658826fb9d77998008dc040849465e8995451d15fc854e5c78940722af11b9"} Sep 11 17:07:23.605096 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:23.605016 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24" Sep 11 17:07:23.606643 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:23.606616 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n" event={"ID":"ad288b40-fe10-4d49-8b2b-0660e6fe20f3","Type":"ContainerStarted","Data":"65ff03acd710b164a5a0a8c7886f41c02e2fcf5571ab5400d2e2187d450bccab"} Sep 11 17:07:23.606759 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:23.606672 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n" Sep 11 17:07:23.606759 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:23.606696 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n" event={"ID":"ad288b40-fe10-4d49-8b2b-0660e6fe20f3","Type":"ContainerStarted","Data":"7c1a18a902e6377ef971a5b59218e88bf7b8f93682e897c365d6716ff3b86c5c"} Sep 11 17:07:23.622192 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:23.622128 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24" podStartSLOduration=1.622109412 podStartE2EDuration="1.622109412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:07:23.621776638 +0000 UTC m=+1459.029878929" watchObservedRunningTime="2026-09-11 17:07:23.622109412 +0000 UTC m=+1459.030211682" Sep 11 17:07:23.639353 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:23.639304 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n" podStartSLOduration=1.6392893499999999 podStartE2EDuration="1.63928935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:07:23.638570431 +0000 UTC m=+1459.046672701" watchObservedRunningTime="2026-09-11 17:07:23.63928935 +0000 UTC m=+1459.047391636" Sep 11 17:07:34.614111 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:34.614079 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24" Sep 11 17:07:34.614587 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:34.614474 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n" Sep 11 17:07:52.803736 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:52.803706 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk"] Sep 11 17:07:52.807459 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:52.807442 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk" Sep 11 17:07:52.810380 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:52.810355 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b34361-dockercfg-tvbt2\"" Sep 11 17:07:52.810380 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:52.810379 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b34361\"" Sep 11 17:07:52.817960 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:52.817929 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk"] Sep 11 17:07:52.915933 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:52.915897 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a4644e-19da-41bd-9f28-54fadebba249-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk\" (UID: \"88a4644e-19da-41bd-9f28-54fadebba249\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk" Sep 11 17:07:52.916130 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:52.915950 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb75b\" (UniqueName: \"kubernetes.io/projected/88a4644e-19da-41bd-9f28-54fadebba249-kube-api-access-sb75b\") pod \"payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk\" (UID: \"88a4644e-19da-41bd-9f28-54fadebba249\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk" Sep 11 17:07:53.016654 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:53.016619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a4644e-19da-41bd-9f28-54fadebba249-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk\" (UID: \"88a4644e-19da-41bd-9f28-54fadebba249\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk" Sep 11 17:07:53.016839 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:53.016670 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb75b\" (UniqueName: \"kubernetes.io/projected/88a4644e-19da-41bd-9f28-54fadebba249-kube-api-access-sb75b\") pod \"payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk\" (UID: \"88a4644e-19da-41bd-9f28-54fadebba249\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk" Sep 11 17:07:53.017340 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:53.017321 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a4644e-19da-41bd-9f28-54fadebba249-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk\" (UID: \"88a4644e-19da-41bd-9f28-54fadebba249\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk" Sep 11 17:07:53.025908 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:53.025880 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb75b\" (UniqueName: \"kubernetes.io/projected/88a4644e-19da-41bd-9f28-54fadebba249-kube-api-access-sb75b\") pod \"payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk\" (UID: \"88a4644e-19da-41bd-9f28-54fadebba249\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk" Sep 11 17:07:53.128083 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:53.127945 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk" Sep 11 17:07:53.408324 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:53.408292 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk"] Sep 11 17:07:53.410078 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:07:53.410053 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a4644e_19da_41bd_9f28_54fadebba249.slice/crio-a5b9351dd5c5859a981f108ec82b205701ae8104d1272c69298ef22d8e3a3309 WatchSource:0}: Error finding container a5b9351dd5c5859a981f108ec82b205701ae8104d1272c69298ef22d8e3a3309: Status 404 returned error can't find the container with id a5b9351dd5c5859a981f108ec82b205701ae8104d1272c69298ef22d8e3a3309 Sep 11 17:07:53.777973 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:53.777938 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk" event={"ID":"88a4644e-19da-41bd-9f28-54fadebba249","Type":"ContainerStarted","Data":"027a0b920e900578c3ba57ee5ec7fb1e864a8442b4fe0148af7faf6ddaad3131"} Sep 11 17:07:53.778151 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:53.777999 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk" event={"ID":"88a4644e-19da-41bd-9f28-54fadebba249","Type":"ContainerStarted","Data":"a5b9351dd5c5859a981f108ec82b205701ae8104d1272c69298ef22d8e3a3309"} Sep 11 17:07:53.778151 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:53.778024 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk" Sep 11 17:07:53.795846 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:53.795794 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk" podStartSLOduration=1.7957776650000001 podStartE2EDuration="1.795777665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:07:53.795142764 +0000 UTC m=+1489.203245059" watchObservedRunningTime="2026-09-11 17:07:53.795777665 +0000 UTC m=+1489.203879937" Sep 11 17:07:54.514425 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:07:54.514387 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh"] Sep 11 17:08:04.787138 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:04.787103 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk" Sep 11 17:08:05.108700 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:05.108614 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 17:08:05.111031 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:05.111010 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 17:08:05.715303 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:05.715260 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc"] Sep 11 17:08:05.715547 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:05.715513 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc" podUID="9e122d49-2235-456c-a734-757161c1d060" containerName="payload-processing" containerID="cri-o://66ad23eb1ebf4d9531786979ee2d2d92c6024f645e12924275b4ab5250aff43e" gracePeriod=30 Sep 11 17:08:05.848936 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:05.848906 2577 generic.go:356] "Generic (PLEG): container finished" podID="9e122d49-2235-456c-a734-757161c1d060" containerID="66ad23eb1ebf4d9531786979ee2d2d92c6024f645e12924275b4ab5250aff43e" exitCode=0 Sep 11 17:08:05.849291 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:05.848980 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc" event={"ID":"9e122d49-2235-456c-a734-757161c1d060","Type":"ContainerDied","Data":"66ad23eb1ebf4d9531786979ee2d2d92c6024f645e12924275b4ab5250aff43e"} Sep 11 17:08:05.996256 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:05.996016 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc" Sep 11 17:08:06.165807 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:06.165763 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch9sj\" (UniqueName: \"kubernetes.io/projected/9e122d49-2235-456c-a734-757161c1d060-kube-api-access-ch9sj\") pod \"9e122d49-2235-456c-a734-757161c1d060\" (UID: \"9e122d49-2235-456c-a734-757161c1d060\") " Sep 11 17:08:06.165997 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:06.165880 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e122d49-2235-456c-a734-757161c1d060-plugins-config-volume\") pod \"9e122d49-2235-456c-a734-757161c1d060\" (UID: \"9e122d49-2235-456c-a734-757161c1d060\") " Sep 11 17:08:06.166227 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:06.166202 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e122d49-2235-456c-a734-757161c1d060-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9e122d49-2235-456c-a734-757161c1d060" (UID: "9e122d49-2235-456c-a734-757161c1d060"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:08:06.168084 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:06.168052 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e122d49-2235-456c-a734-757161c1d060-kube-api-access-ch9sj" (OuterVolumeSpecName: "kube-api-access-ch9sj") pod "9e122d49-2235-456c-a734-757161c1d060" (UID: "9e122d49-2235-456c-a734-757161c1d060"). InnerVolumeSpecName "kube-api-access-ch9sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:08:06.267123 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:06.267085 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e122d49-2235-456c-a734-757161c1d060-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:08:06.267123 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:06.267117 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch9sj\" (UniqueName: \"kubernetes.io/projected/9e122d49-2235-456c-a734-757161c1d060-kube-api-access-ch9sj\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:08:06.855693 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:06.855656 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc" event={"ID":"9e122d49-2235-456c-a734-757161c1d060","Type":"ContainerDied","Data":"581b61fa1df2f880ea530be6f71e6020d319bb0e5d9464e417c71ce0b80504a7"} Sep 11 17:08:06.855693 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:06.855683 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc" Sep 11 17:08:06.856215 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:06.855719 2577 scope.go:117] "RemoveContainer" containerID="66ad23eb1ebf4d9531786979ee2d2d92c6024f645e12924275b4ab5250aff43e" Sep 11 17:08:06.879820 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:06.879795 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc"] Sep 11 17:08:06.883438 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:06.883412 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-52b0af-6b764dfd47-hhjtc"] Sep 11 17:08:07.099681 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:07.099645 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e122d49-2235-456c-a734-757161c1d060" path="/var/lib/kubelet/pods/9e122d49-2235-456c-a734-757161c1d060/volumes" Sep 11 17:08:09.535854 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.535786 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" podUID="55a78ab3-31cd-4480-89fb-3507de99bb39" containerName="main" containerID="cri-o://acd57c73f5f3b94b0ccb653a3faa3b3fce0fe51cb9a7fcfb9198296fd5b68966" gracePeriod=45 Sep 11 17:08:09.798833 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.798766 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:08:09.874083 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.874049 2577 generic.go:356] "Generic (PLEG): container finished" podID="55a78ab3-31cd-4480-89fb-3507de99bb39" containerID="acd57c73f5f3b94b0ccb653a3faa3b3fce0fe51cb9a7fcfb9198296fd5b68966" exitCode=0 Sep 11 17:08:09.874269 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.874127 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" event={"ID":"55a78ab3-31cd-4480-89fb-3507de99bb39","Type":"ContainerDied","Data":"acd57c73f5f3b94b0ccb653a3faa3b3fce0fe51cb9a7fcfb9198296fd5b68966"} Sep 11 17:08:09.874269 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.874150 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" Sep 11 17:08:09.874269 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.874164 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh" event={"ID":"55a78ab3-31cd-4480-89fb-3507de99bb39","Type":"ContainerDied","Data":"2a6dd2ab31707baf75876ea310e59c32a9f272749398b3f4a8504de490c08807"} Sep 11 17:08:09.874269 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.874194 2577 scope.go:117] "RemoveContainer" containerID="acd57c73f5f3b94b0ccb653a3faa3b3fce0fe51cb9a7fcfb9198296fd5b68966" Sep 11 17:08:09.883371 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.883345 2577 scope.go:117] "RemoveContainer" containerID="acd57c73f5f3b94b0ccb653a3faa3b3fce0fe51cb9a7fcfb9198296fd5b68966" Sep 11 17:08:09.883724 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:08:09.883692 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd57c73f5f3b94b0ccb653a3faa3b3fce0fe51cb9a7fcfb9198296fd5b68966\": container with ID starting with acd57c73f5f3b94b0ccb653a3faa3b3fce0fe51cb9a7fcfb9198296fd5b68966 not found: ID does not exist" containerID="acd57c73f5f3b94b0ccb653a3faa3b3fce0fe51cb9a7fcfb9198296fd5b68966" Sep 11 17:08:09.883838 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.883739 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd57c73f5f3b94b0ccb653a3faa3b3fce0fe51cb9a7fcfb9198296fd5b68966"} err="failed to get container status \"acd57c73f5f3b94b0ccb653a3faa3b3fce0fe51cb9a7fcfb9198296fd5b68966\": rpc error: code = NotFound desc = could not find container \"acd57c73f5f3b94b0ccb653a3faa3b3fce0fe51cb9a7fcfb9198296fd5b68966\": container with ID starting with acd57c73f5f3b94b0ccb653a3faa3b3fce0fe51cb9a7fcfb9198296fd5b68966 not found: ID does not exist" Sep 11 17:08:09.902239 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.902213 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-model-cache\") pod \"55a78ab3-31cd-4480-89fb-3507de99bb39\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " Sep 11 17:08:09.902413 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.902259 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-home\") pod \"55a78ab3-31cd-4480-89fb-3507de99bb39\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " Sep 11 17:08:09.902413 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.902310 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skqpt\" (UniqueName: \"kubernetes.io/projected/55a78ab3-31cd-4480-89fb-3507de99bb39-kube-api-access-skqpt\") pod \"55a78ab3-31cd-4480-89fb-3507de99bb39\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " Sep 11 17:08:09.902413 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.902395 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-dshm\") pod \"55a78ab3-31cd-4480-89fb-3507de99bb39\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " Sep 11 17:08:09.902593 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.902452 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a78ab3-31cd-4480-89fb-3507de99bb39-tls-certs\") pod \"55a78ab3-31cd-4480-89fb-3507de99bb39\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " Sep 11 17:08:09.902593 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.902516 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-model-cache" (OuterVolumeSpecName: "model-cache") pod "55a78ab3-31cd-4480-89fb-3507de99bb39" (UID: "55a78ab3-31cd-4480-89fb-3507de99bb39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:08:09.902593 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.902567 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-tmp-dir\") pod \"55a78ab3-31cd-4480-89fb-3507de99bb39\" (UID: \"55a78ab3-31cd-4480-89fb-3507de99bb39\") " Sep 11 17:08:09.903232 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.903148 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55a78ab3-31cd-4480-89fb-3507de99bb39" (UID: "55a78ab3-31cd-4480-89fb-3507de99bb39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:08:09.903232 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.903206 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-home" (OuterVolumeSpecName: "home") pod "55a78ab3-31cd-4480-89fb-3507de99bb39" (UID: "55a78ab3-31cd-4480-89fb-3507de99bb39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:08:09.903628 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.903602 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-tmp-dir\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:08:09.903747 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.903632 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-model-cache\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:08:09.903747 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.903648 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-home\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:08:09.905166 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.905140 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-dshm" (OuterVolumeSpecName: "dshm") pod "55a78ab3-31cd-4480-89fb-3507de99bb39" (UID: "55a78ab3-31cd-4480-89fb-3507de99bb39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:08:09.905263 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.905189 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a78ab3-31cd-4480-89fb-3507de99bb39-kube-api-access-skqpt" (OuterVolumeSpecName: "kube-api-access-skqpt") pod "55a78ab3-31cd-4480-89fb-3507de99bb39" (UID: "55a78ab3-31cd-4480-89fb-3507de99bb39"). InnerVolumeSpecName "kube-api-access-skqpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:08:09.905263 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:09.905188 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55a78ab3-31cd-4480-89fb-3507de99bb39-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55a78ab3-31cd-4480-89fb-3507de99bb39" (UID: "55a78ab3-31cd-4480-89fb-3507de99bb39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 17:08:10.004792 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:10.004749 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skqpt\" (UniqueName: \"kubernetes.io/projected/55a78ab3-31cd-4480-89fb-3507de99bb39-kube-api-access-skqpt\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:08:10.004792 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:10.004784 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a78ab3-31cd-4480-89fb-3507de99bb39-dshm\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:08:10.004792 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:10.004798 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a78ab3-31cd-4480-89fb-3507de99bb39-tls-certs\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:08:10.196395 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:10.196363 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh"] Sep 11 17:08:10.200307 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:10.200279 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-52b0af/distinct-w0-52b0af-kserve-5c7c795b49-6lqfh"] Sep 11 17:08:11.094866 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:11.094836 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55a78ab3-31cd-4480-89fb-3507de99bb39" path="/var/lib/kubelet/pods/55a78ab3-31cd-4480-89fb-3507de99bb39/volumes" Sep 11 17:08:34.678345 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:34.678305 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24"] Sep 11 17:08:34.681327 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:34.678912 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24" podUID="05ad7566-a0fd-45e7-8b5c-bce5d664b417" containerName="payload-processing" containerID="cri-o://43cada8a1f87e9ee2eb66680f59c68716509471aa4e57c3ef61d332f82c7b034" gracePeriod=30 Sep 11 17:08:34.685634 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:34.683232 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n"] Sep 11 17:08:34.685634 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:34.683455 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n" podUID="ad288b40-fe10-4d49-8b2b-0660e6fe20f3" containerName="payload-pre-processing" containerID="cri-o://65ff03acd710b164a5a0a8c7886f41c02e2fcf5571ab5400d2e2187d450bccab" gracePeriod=30 Sep 11 17:08:34.978513 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:34.978474 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24" Sep 11 17:08:34.981163 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:34.981143 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n" Sep 11 17:08:35.022517 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.017707 2577 generic.go:356] "Generic (PLEG): container finished" podID="ad288b40-fe10-4d49-8b2b-0660e6fe20f3" containerID="65ff03acd710b164a5a0a8c7886f41c02e2fcf5571ab5400d2e2187d450bccab" exitCode=0 Sep 11 17:08:35.022517 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.017753 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n" Sep 11 17:08:35.022517 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.017949 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n" event={"ID":"ad288b40-fe10-4d49-8b2b-0660e6fe20f3","Type":"ContainerDied","Data":"65ff03acd710b164a5a0a8c7886f41c02e2fcf5571ab5400d2e2187d450bccab"} Sep 11 17:08:35.022517 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.018001 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n" event={"ID":"ad288b40-fe10-4d49-8b2b-0660e6fe20f3","Type":"ContainerDied","Data":"7c1a18a902e6377ef971a5b59218e88bf7b8f93682e897c365d6716ff3b86c5c"} Sep 11 17:08:35.022517 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.018040 2577 scope.go:117] "RemoveContainer" containerID="65ff03acd710b164a5a0a8c7886f41c02e2fcf5571ab5400d2e2187d450bccab" Sep 11 17:08:35.022517 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.020760 2577 generic.go:356] "Generic (PLEG): container finished" podID="05ad7566-a0fd-45e7-8b5c-bce5d664b417" containerID="43cada8a1f87e9ee2eb66680f59c68716509471aa4e57c3ef61d332f82c7b034" exitCode=0 Sep 11 17:08:35.022517 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.020804 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24" event={"ID":"05ad7566-a0fd-45e7-8b5c-bce5d664b417","Type":"ContainerDied","Data":"43cada8a1f87e9ee2eb66680f59c68716509471aa4e57c3ef61d332f82c7b034"} Sep 11 17:08:35.022517 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.020849 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24" event={"ID":"05ad7566-a0fd-45e7-8b5c-bce5d664b417","Type":"ContainerDied","Data":"cc658826fb9d77998008dc040849465e8995451d15fc854e5c78940722af11b9"} Sep 11 17:08:35.022517 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.020923 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24" Sep 11 17:08:35.040239 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.040117 2577 scope.go:117] "RemoveContainer" containerID="65ff03acd710b164a5a0a8c7886f41c02e2fcf5571ab5400d2e2187d450bccab" Sep 11 17:08:35.040847 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:08:35.040797 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ff03acd710b164a5a0a8c7886f41c02e2fcf5571ab5400d2e2187d450bccab\": container with ID starting with 65ff03acd710b164a5a0a8c7886f41c02e2fcf5571ab5400d2e2187d450bccab not found: ID does not exist" containerID="65ff03acd710b164a5a0a8c7886f41c02e2fcf5571ab5400d2e2187d450bccab" Sep 11 17:08:35.040971 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.040907 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ff03acd710b164a5a0a8c7886f41c02e2fcf5571ab5400d2e2187d450bccab"} err="failed to get container status \"65ff03acd710b164a5a0a8c7886f41c02e2fcf5571ab5400d2e2187d450bccab\": rpc error: code = NotFound desc = could not find container \"65ff03acd710b164a5a0a8c7886f41c02e2fcf5571ab5400d2e2187d450bccab\": container with ID starting with 65ff03acd710b164a5a0a8c7886f41c02e2fcf5571ab5400d2e2187d450bccab not found: ID does not exist" Sep 11 17:08:35.040971 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.040938 2577 scope.go:117] "RemoveContainer" containerID="43cada8a1f87e9ee2eb66680f59c68716509471aa4e57c3ef61d332f82c7b034" Sep 11 17:08:35.042510 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.042477 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdg89\" (UniqueName: \"kubernetes.io/projected/ad288b40-fe10-4d49-8b2b-0660e6fe20f3-kube-api-access-bdg89\") pod \"ad288b40-fe10-4d49-8b2b-0660e6fe20f3\" (UID: \"ad288b40-fe10-4d49-8b2b-0660e6fe20f3\") " Sep 11 17:08:35.042593 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.042550 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cp2lz\" (UniqueName: \"kubernetes.io/projected/05ad7566-a0fd-45e7-8b5c-bce5d664b417-kube-api-access-cp2lz\") pod \"05ad7566-a0fd-45e7-8b5c-bce5d664b417\" (UID: \"05ad7566-a0fd-45e7-8b5c-bce5d664b417\") " Sep 11 17:08:35.042654 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.042604 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ad7566-a0fd-45e7-8b5c-bce5d664b417-plugins-config-volume\") pod \"05ad7566-a0fd-45e7-8b5c-bce5d664b417\" (UID: \"05ad7566-a0fd-45e7-8b5c-bce5d664b417\") " Sep 11 17:08:35.042712 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.042677 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad288b40-fe10-4d49-8b2b-0660e6fe20f3-plugins-config-volume\") pod \"ad288b40-fe10-4d49-8b2b-0660e6fe20f3\" (UID: \"ad288b40-fe10-4d49-8b2b-0660e6fe20f3\") " Sep 11 17:08:35.043248 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.043221 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05ad7566-a0fd-45e7-8b5c-bce5d664b417-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "05ad7566-a0fd-45e7-8b5c-bce5d664b417" (UID: "05ad7566-a0fd-45e7-8b5c-bce5d664b417"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:08:35.043626 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.043569 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad288b40-fe10-4d49-8b2b-0660e6fe20f3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad288b40-fe10-4d49-8b2b-0660e6fe20f3" (UID: "ad288b40-fe10-4d49-8b2b-0660e6fe20f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:08:35.047266 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.047235 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ad7566-a0fd-45e7-8b5c-bce5d664b417-kube-api-access-cp2lz" (OuterVolumeSpecName: "kube-api-access-cp2lz") pod "05ad7566-a0fd-45e7-8b5c-bce5d664b417" (UID: "05ad7566-a0fd-45e7-8b5c-bce5d664b417"). InnerVolumeSpecName "kube-api-access-cp2lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:08:35.049084 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.049054 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad288b40-fe10-4d49-8b2b-0660e6fe20f3-kube-api-access-bdg89" (OuterVolumeSpecName: "kube-api-access-bdg89") pod "ad288b40-fe10-4d49-8b2b-0660e6fe20f3" (UID: "ad288b40-fe10-4d49-8b2b-0660e6fe20f3"). InnerVolumeSpecName "kube-api-access-bdg89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:08:35.067469 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.067328 2577 scope.go:117] "RemoveContainer" containerID="43cada8a1f87e9ee2eb66680f59c68716509471aa4e57c3ef61d332f82c7b034" Sep 11 17:08:35.068114 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:08:35.068054 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43cada8a1f87e9ee2eb66680f59c68716509471aa4e57c3ef61d332f82c7b034\": container with ID starting with 43cada8a1f87e9ee2eb66680f59c68716509471aa4e57c3ef61d332f82c7b034 not found: ID does not exist" containerID="43cada8a1f87e9ee2eb66680f59c68716509471aa4e57c3ef61d332f82c7b034" Sep 11 17:08:35.068241 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.068127 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43cada8a1f87e9ee2eb66680f59c68716509471aa4e57c3ef61d332f82c7b034"} err="failed to get container status \"43cada8a1f87e9ee2eb66680f59c68716509471aa4e57c3ef61d332f82c7b034\": rpc error: code = NotFound desc = could not find container \"43cada8a1f87e9ee2eb66680f59c68716509471aa4e57c3ef61d332f82c7b034\": container with ID starting with 43cada8a1f87e9ee2eb66680f59c68716509471aa4e57c3ef61d332f82c7b034 not found: ID does not exist" Sep 11 17:08:35.143997 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.143966 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad288b40-fe10-4d49-8b2b-0660e6fe20f3-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:08:35.143997 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.143995 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdg89\" (UniqueName: \"kubernetes.io/projected/ad288b40-fe10-4d49-8b2b-0660e6fe20f3-kube-api-access-bdg89\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:08:35.144188 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.144008 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp2lz\" (UniqueName: \"kubernetes.io/projected/05ad7566-a0fd-45e7-8b5c-bce5d664b417-kube-api-access-cp2lz\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:08:35.144188 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.144017 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ad7566-a0fd-45e7-8b5c-bce5d664b417-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:08:35.342428 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.342398 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n"] Sep 11 17:08:35.352892 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.352860 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-935ef2-67fc767bcd-qlg4n"] Sep 11 17:08:35.374330 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.374294 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24"] Sep 11 17:08:35.383440 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:35.383410 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-935ef2-55f98b5cb7-j7h24"] Sep 11 17:08:37.095233 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:37.095191 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05ad7566-a0fd-45e7-8b5c-bce5d664b417" path="/var/lib/kubelet/pods/05ad7566-a0fd-45e7-8b5c-bce5d664b417/volumes" Sep 11 17:08:37.096674 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:37.096616 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad288b40-fe10-4d49-8b2b-0660e6fe20f3" path="/var/lib/kubelet/pods/ad288b40-fe10-4d49-8b2b-0660e6fe20f3/volumes" Sep 11 17:08:42.201415 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.201368 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b34361-zncm5"] Sep 11 17:08:42.201828 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.201802 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05ad7566-a0fd-45e7-8b5c-bce5d664b417" containerName="payload-processing" Sep 11 17:08:42.201828 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.201814 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="05ad7566-a0fd-45e7-8b5c-bce5d664b417" containerName="payload-processing" Sep 11 17:08:42.201828 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.201828 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad288b40-fe10-4d49-8b2b-0660e6fe20f3" containerName="payload-pre-processing" Sep 11 17:08:42.201937 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.201833 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad288b40-fe10-4d49-8b2b-0660e6fe20f3" containerName="payload-pre-processing" Sep 11 17:08:42.201937 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.201843 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e122d49-2235-456c-a734-757161c1d060" containerName="payload-processing" Sep 11 17:08:42.201937 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.201849 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e122d49-2235-456c-a734-757161c1d060" containerName="payload-processing" Sep 11 17:08:42.201937 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.201862 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55a78ab3-31cd-4480-89fb-3507de99bb39" containerName="main" Sep 11 17:08:42.201937 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.201867 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="55a78ab3-31cd-4480-89fb-3507de99bb39" containerName="main" Sep 11 17:08:42.201937 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.201926 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="55a78ab3-31cd-4480-89fb-3507de99bb39" containerName="main" Sep 11 17:08:42.201937 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.201934 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad288b40-fe10-4d49-8b2b-0660e6fe20f3" containerName="payload-pre-processing" Sep 11 17:08:42.202148 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.201942 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e122d49-2235-456c-a734-757161c1d060" containerName="payload-processing" Sep 11 17:08:42.202148 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.201949 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="05ad7566-a0fd-45e7-8b5c-bce5d664b417" containerName="payload-processing" Sep 11 17:08:42.206184 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.206161 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b34361-zncm5" Sep 11 17:08:42.208981 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.208960 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 17:08:42.208981 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.208971 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pc4qb\"" Sep 11 17:08:42.213944 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.213918 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b34361-zncm5"] Sep 11 17:08:42.312875 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.312833 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761f41c3-6a08-450c-9b23-87199c294e93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b34361-zncm5\" (UID: \"761f41c3-6a08-450c-9b23-87199c294e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b34361-zncm5" Sep 11 17:08:42.413931 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.413888 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761f41c3-6a08-450c-9b23-87199c294e93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b34361-zncm5\" (UID: \"761f41c3-6a08-450c-9b23-87199c294e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b34361-zncm5" Sep 11 17:08:42.414599 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.414578 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761f41c3-6a08-450c-9b23-87199c294e93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b34361-zncm5\" (UID: \"761f41c3-6a08-450c-9b23-87199c294e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b34361-zncm5" Sep 11 17:08:42.518865 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.518827 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b34361-zncm5" Sep 11 17:08:42.829858 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:42.829829 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b34361-zncm5"] Sep 11 17:08:42.832102 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:08:42.832073 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761f41c3_6a08_450c_9b23_87199c294e93.slice/crio-ad1eab3669f4df4e4203299eedc1400d1dadd065ac9ddc5b443b3b2d707ca4fd WatchSource:0}: Error finding container ad1eab3669f4df4e4203299eedc1400d1dadd065ac9ddc5b443b3b2d707ca4fd: Status 404 returned error can't find the container with id ad1eab3669f4df4e4203299eedc1400d1dadd065ac9ddc5b443b3b2d707ca4fd Sep 11 17:08:43.071401 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:43.071364 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b34361-zncm5" event={"ID":"761f41c3-6a08-450c-9b23-87199c294e93","Type":"ContainerStarted","Data":"ad1eab3669f4df4e4203299eedc1400d1dadd065ac9ddc5b443b3b2d707ca4fd"} Sep 11 17:08:44.079203 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:44.079164 2577 generic.go:356] "Generic (PLEG): container finished" podID="761f41c3-6a08-450c-9b23-87199c294e93" containerID="61bf70e580aea25ebee883d56915b216a8fa2fe866f12ff7f088a00e62747dc1" exitCode=0 Sep 11 17:08:44.079614 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:44.079249 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b34361-zncm5" event={"ID":"761f41c3-6a08-450c-9b23-87199c294e93","Type":"ContainerDied","Data":"61bf70e580aea25ebee883d56915b216a8fa2fe866f12ff7f088a00e62747dc1"} Sep 11 17:08:45.228177 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:45.228145 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b34361-zncm5" Sep 11 17:08:45.341574 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:45.341463 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761f41c3-6a08-450c-9b23-87199c294e93-maas-api-service-ca\") pod \"761f41c3-6a08-450c-9b23-87199c294e93\" (UID: \"761f41c3-6a08-450c-9b23-87199c294e93\") " Sep 11 17:08:45.341805 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:45.341780 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761f41c3-6a08-450c-9b23-87199c294e93-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "761f41c3-6a08-450c-9b23-87199c294e93" (UID: "761f41c3-6a08-450c-9b23-87199c294e93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:08:45.442414 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:45.442382 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761f41c3-6a08-450c-9b23-87199c294e93-maas-api-service-ca\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:08:46.090691 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:46.090660 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b34361-zncm5" Sep 11 17:08:46.090691 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:46.090681 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b34361-zncm5" event={"ID":"761f41c3-6a08-450c-9b23-87199c294e93","Type":"ContainerDied","Data":"ad1eab3669f4df4e4203299eedc1400d1dadd065ac9ddc5b443b3b2d707ca4fd"} Sep 11 17:08:46.090916 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:46.090720 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad1eab3669f4df4e4203299eedc1400d1dadd065ac9ddc5b443b3b2d707ca4fd" Sep 11 17:08:57.808420 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:57.808385 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk"] Sep 11 17:08:57.808910 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:57.808680 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk" podUID="88a4644e-19da-41bd-9f28-54fadebba249" containerName="payload-pre-processing" containerID="cri-o://027a0b920e900578c3ba57ee5ec7fb1e864a8442b4fe0148af7faf6ddaad3131" gracePeriod=30 Sep 11 17:08:58.072785 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:58.072721 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk" Sep 11 17:08:58.088350 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:58.088321 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b34361-zncm5"] Sep 11 17:08:58.096266 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:58.096223 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b34361-zncm5"] Sep 11 17:08:58.159275 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:58.159237 2577 generic.go:356] "Generic (PLEG): container finished" podID="88a4644e-19da-41bd-9f28-54fadebba249" containerID="027a0b920e900578c3ba57ee5ec7fb1e864a8442b4fe0148af7faf6ddaad3131" exitCode=0 Sep 11 17:08:58.159462 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:58.159288 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk" Sep 11 17:08:58.159462 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:58.159324 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk" event={"ID":"88a4644e-19da-41bd-9f28-54fadebba249","Type":"ContainerDied","Data":"027a0b920e900578c3ba57ee5ec7fb1e864a8442b4fe0148af7faf6ddaad3131"} Sep 11 17:08:58.159462 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:58.159388 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk" event={"ID":"88a4644e-19da-41bd-9f28-54fadebba249","Type":"ContainerDied","Data":"a5b9351dd5c5859a981f108ec82b205701ae8104d1272c69298ef22d8e3a3309"} Sep 11 17:08:58.159462 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:58.159420 2577 scope.go:117] "RemoveContainer" containerID="027a0b920e900578c3ba57ee5ec7fb1e864a8442b4fe0148af7faf6ddaad3131" Sep 11 17:08:58.168390 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:58.168370 2577 scope.go:117] "RemoveContainer" containerID="027a0b920e900578c3ba57ee5ec7fb1e864a8442b4fe0148af7faf6ddaad3131" Sep 11 17:08:58.168665 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:08:58.168648 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027a0b920e900578c3ba57ee5ec7fb1e864a8442b4fe0148af7faf6ddaad3131\": container with ID starting with 027a0b920e900578c3ba57ee5ec7fb1e864a8442b4fe0148af7faf6ddaad3131 not found: ID does not exist" containerID="027a0b920e900578c3ba57ee5ec7fb1e864a8442b4fe0148af7faf6ddaad3131" Sep 11 17:08:58.168715 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:58.168677 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027a0b920e900578c3ba57ee5ec7fb1e864a8442b4fe0148af7faf6ddaad3131"} err="failed to get container status \"027a0b920e900578c3ba57ee5ec7fb1e864a8442b4fe0148af7faf6ddaad3131\": rpc error: code = NotFound desc = could not find container \"027a0b920e900578c3ba57ee5ec7fb1e864a8442b4fe0148af7faf6ddaad3131\": container with ID starting with 027a0b920e900578c3ba57ee5ec7fb1e864a8442b4fe0148af7faf6ddaad3131 not found: ID does not exist" Sep 11 17:08:58.173129 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:58.173105 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb75b\" (UniqueName: \"kubernetes.io/projected/88a4644e-19da-41bd-9f28-54fadebba249-kube-api-access-sb75b\") pod \"88a4644e-19da-41bd-9f28-54fadebba249\" (UID: \"88a4644e-19da-41bd-9f28-54fadebba249\") " Sep 11 17:08:58.173244 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:58.173180 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a4644e-19da-41bd-9f28-54fadebba249-plugins-config-volume\") pod \"88a4644e-19da-41bd-9f28-54fadebba249\" (UID: \"88a4644e-19da-41bd-9f28-54fadebba249\") " Sep 11 17:08:58.173565 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:58.173545 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a4644e-19da-41bd-9f28-54fadebba249-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "88a4644e-19da-41bd-9f28-54fadebba249" (UID: "88a4644e-19da-41bd-9f28-54fadebba249"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:08:58.175185 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:58.175164 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a4644e-19da-41bd-9f28-54fadebba249-kube-api-access-sb75b" (OuterVolumeSpecName: "kube-api-access-sb75b") pod "88a4644e-19da-41bd-9f28-54fadebba249" (UID: "88a4644e-19da-41bd-9f28-54fadebba249"). InnerVolumeSpecName "kube-api-access-sb75b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:08:58.274522 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:58.274450 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb75b\" (UniqueName: \"kubernetes.io/projected/88a4644e-19da-41bd-9f28-54fadebba249-kube-api-access-sb75b\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:08:58.274522 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:58.274485 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a4644e-19da-41bd-9f28-54fadebba249-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:08:58.503059 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:58.503026 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk"] Sep 11 17:08:58.515389 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:58.515359 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b34361-56d4b5f69d-8p4nk"] Sep 11 17:08:59.094656 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:59.094621 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="761f41c3-6a08-450c-9b23-87199c294e93" path="/var/lib/kubelet/pods/761f41c3-6a08-450c-9b23-87199c294e93/volumes" Sep 11 17:08:59.095176 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:08:59.095154 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88a4644e-19da-41bd-9f28-54fadebba249" path="/var/lib/kubelet/pods/88a4644e-19da-41bd-9f28-54fadebba249/volumes" Sep 11 17:09:05.294758 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:05.294725 2577 scope.go:117] "RemoveContainer" containerID="5ea560c877aa7d3f0b8a76d43544fb44e03ba839db83a0fbe95c57b32b74f069" Sep 11 17:09:05.711828 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:05.711745 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc"] Sep 11 17:09:05.712151 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:05.712140 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a4644e-19da-41bd-9f28-54fadebba249" containerName="payload-pre-processing" Sep 11 17:09:05.712194 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:05.712153 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a4644e-19da-41bd-9f28-54fadebba249" containerName="payload-pre-processing" Sep 11 17:09:05.712194 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:05.712167 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="761f41c3-6a08-450c-9b23-87199c294e93" containerName="revoke-keys" Sep 11 17:09:05.712194 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:05.712172 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="761f41c3-6a08-450c-9b23-87199c294e93" containerName="revoke-keys" Sep 11 17:09:05.712290 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:05.712244 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="761f41c3-6a08-450c-9b23-87199c294e93" containerName="revoke-keys" Sep 11 17:09:05.712290 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:05.712253 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a4644e-19da-41bd-9f28-54fadebba249" containerName="payload-pre-processing" Sep 11 17:09:05.715102 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:05.715086 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc" Sep 11 17:09:05.717868 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:05.717827 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 17:09:05.718020 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:05.717939 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pc4qb\"" Sep 11 17:09:05.725558 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:05.725533 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc"] Sep 11 17:09:05.742776 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:05.742744 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f04664-ad0c-49a4-a3e6-bc7d0d862ae4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc\" (UID: \"58f04664-ad0c-49a4-a3e6-bc7d0d862ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc" Sep 11 17:09:05.843287 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:05.843244 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f04664-ad0c-49a4-a3e6-bc7d0d862ae4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc\" (UID: \"58f04664-ad0c-49a4-a3e6-bc7d0d862ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc" Sep 11 17:09:05.843934 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:05.843913 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f04664-ad0c-49a4-a3e6-bc7d0d862ae4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc\" (UID: \"58f04664-ad0c-49a4-a3e6-bc7d0d862ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc" Sep 11 17:09:06.027324 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:06.027283 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc" Sep 11 17:09:06.298117 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:06.298087 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc"] Sep 11 17:09:06.299916 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:09:06.299890 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f04664_ad0c_49a4_a3e6_bc7d0d862ae4.slice/crio-32f06194513752d9bf31f64e3e2b2bd59e46e462d6d8729bc9af70e50de93eed WatchSource:0}: Error finding container 32f06194513752d9bf31f64e3e2b2bd59e46e462d6d8729bc9af70e50de93eed: Status 404 returned error can't find the container with id 32f06194513752d9bf31f64e3e2b2bd59e46e462d6d8729bc9af70e50de93eed Sep 11 17:09:07.209380 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:07.209343 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc" event={"ID":"58f04664-ad0c-49a4-a3e6-bc7d0d862ae4","Type":"ContainerStarted","Data":"344314ee4311f7a1b6e2ea05f9849051fd95a9dbd9b1109e074f9f741b233c7e"} Sep 11 17:09:07.209589 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:07.209396 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc" event={"ID":"58f04664-ad0c-49a4-a3e6-bc7d0d862ae4","Type":"ContainerStarted","Data":"32f06194513752d9bf31f64e3e2b2bd59e46e462d6d8729bc9af70e50de93eed"} Sep 11 17:09:07.226447 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:07.226353 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc" podStartSLOduration=2.22633178 podStartE2EDuration="2.22633178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:09:07.225492654 +0000 UTC m=+1562.633594929" watchObservedRunningTime="2026-09-11 17:09:07.22633178 +0000 UTC m=+1562.634434052" Sep 11 17:09:08.216209 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:08.216170 2577 generic.go:356] "Generic (PLEG): container finished" podID="58f04664-ad0c-49a4-a3e6-bc7d0d862ae4" containerID="344314ee4311f7a1b6e2ea05f9849051fd95a9dbd9b1109e074f9f741b233c7e" exitCode=0 Sep 11 17:09:08.216691 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:08.216234 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc" event={"ID":"58f04664-ad0c-49a4-a3e6-bc7d0d862ae4","Type":"ContainerDied","Data":"344314ee4311f7a1b6e2ea05f9849051fd95a9dbd9b1109e074f9f741b233c7e"} Sep 11 17:09:09.368874 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:09.368846 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc" Sep 11 17:09:09.479168 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:09.479085 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f04664-ad0c-49a4-a3e6-bc7d0d862ae4-maas-api-service-ca\") pod \"58f04664-ad0c-49a4-a3e6-bc7d0d862ae4\" (UID: \"58f04664-ad0c-49a4-a3e6-bc7d0d862ae4\") " Sep 11 17:09:09.479445 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:09.479421 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f04664-ad0c-49a4-a3e6-bc7d0d862ae4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "58f04664-ad0c-49a4-a3e6-bc7d0d862ae4" (UID: "58f04664-ad0c-49a4-a3e6-bc7d0d862ae4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:09:09.580462 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:09.580432 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58f04664-ad0c-49a4-a3e6-bc7d0d862ae4-maas-api-service-ca\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:09:10.233097 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:10.233058 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc" event={"ID":"58f04664-ad0c-49a4-a3e6-bc7d0d862ae4","Type":"ContainerDied","Data":"32f06194513752d9bf31f64e3e2b2bd59e46e462d6d8729bc9af70e50de93eed"} Sep 11 17:09:10.233097 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:10.233097 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc" Sep 11 17:09:10.233322 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:10.233109 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32f06194513752d9bf31f64e3e2b2bd59e46e462d6d8729bc9af70e50de93eed" Sep 11 17:09:16.357966 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:16.357932 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk"] Sep 11 17:09:16.358459 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:16.358159 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk" podUID="ebb32df2-192e-44f9-9eb0-dab17559e266" containerName="payload-pre-processing" containerID="cri-o://7f483a149fea48cd7b2693254d424bcd2c31135ff7b7074a3d968d9103dd690e" gracePeriod=30 Sep 11 17:09:16.528036 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:16.528002 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc"] Sep 11 17:09:16.530451 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:16.530424 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b6653-zh2rc"] Sep 11 17:09:16.622051 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:16.621960 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk" Sep 11 17:09:16.746363 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:16.746335 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb32df2-192e-44f9-9eb0-dab17559e266-plugins-config-volume\") pod \"ebb32df2-192e-44f9-9eb0-dab17559e266\" (UID: \"ebb32df2-192e-44f9-9eb0-dab17559e266\") " Sep 11 17:09:16.746544 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:16.746372 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5qcv\" (UniqueName: \"kubernetes.io/projected/ebb32df2-192e-44f9-9eb0-dab17559e266-kube-api-access-t5qcv\") pod \"ebb32df2-192e-44f9-9eb0-dab17559e266\" (UID: \"ebb32df2-192e-44f9-9eb0-dab17559e266\") " Sep 11 17:09:16.746815 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:16.746786 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb32df2-192e-44f9-9eb0-dab17559e266-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ebb32df2-192e-44f9-9eb0-dab17559e266" (UID: "ebb32df2-192e-44f9-9eb0-dab17559e266"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:09:16.748486 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:16.748461 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb32df2-192e-44f9-9eb0-dab17559e266-kube-api-access-t5qcv" (OuterVolumeSpecName: "kube-api-access-t5qcv") pod "ebb32df2-192e-44f9-9eb0-dab17559e266" (UID: "ebb32df2-192e-44f9-9eb0-dab17559e266"). InnerVolumeSpecName "kube-api-access-t5qcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:09:16.848312 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:16.848271 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb32df2-192e-44f9-9eb0-dab17559e266-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:09:16.848312 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:16.848314 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5qcv\" (UniqueName: \"kubernetes.io/projected/ebb32df2-192e-44f9-9eb0-dab17559e266-kube-api-access-t5qcv\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:09:17.094552 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:17.094492 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58f04664-ad0c-49a4-a3e6-bc7d0d862ae4" path="/var/lib/kubelet/pods/58f04664-ad0c-49a4-a3e6-bc7d0d862ae4/volumes" Sep 11 17:09:17.270944 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:17.270907 2577 generic.go:356] "Generic (PLEG): container finished" podID="ebb32df2-192e-44f9-9eb0-dab17559e266" containerID="7f483a149fea48cd7b2693254d424bcd2c31135ff7b7074a3d968d9103dd690e" exitCode=0 Sep 11 17:09:17.271128 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:17.270953 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk" Sep 11 17:09:17.271128 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:17.271007 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk" event={"ID":"ebb32df2-192e-44f9-9eb0-dab17559e266","Type":"ContainerDied","Data":"7f483a149fea48cd7b2693254d424bcd2c31135ff7b7074a3d968d9103dd690e"} Sep 11 17:09:17.271128 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:17.271072 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk" event={"ID":"ebb32df2-192e-44f9-9eb0-dab17559e266","Type":"ContainerDied","Data":"988cc7033b26df57e31e790f4432b10a88f9d2937155b4ceb55a0cadcad5d60f"} Sep 11 17:09:17.271128 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:17.271099 2577 scope.go:117] "RemoveContainer" containerID="7f483a149fea48cd7b2693254d424bcd2c31135ff7b7074a3d968d9103dd690e" Sep 11 17:09:17.280019 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:17.279996 2577 scope.go:117] "RemoveContainer" containerID="7f483a149fea48cd7b2693254d424bcd2c31135ff7b7074a3d968d9103dd690e" Sep 11 17:09:17.280301 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:09:17.280284 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f483a149fea48cd7b2693254d424bcd2c31135ff7b7074a3d968d9103dd690e\": container with ID starting with 7f483a149fea48cd7b2693254d424bcd2c31135ff7b7074a3d968d9103dd690e not found: ID does not exist" containerID="7f483a149fea48cd7b2693254d424bcd2c31135ff7b7074a3d968d9103dd690e" Sep 11 17:09:17.280350 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:17.280308 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f483a149fea48cd7b2693254d424bcd2c31135ff7b7074a3d968d9103dd690e"} err="failed to get container status \"7f483a149fea48cd7b2693254d424bcd2c31135ff7b7074a3d968d9103dd690e\": rpc error: code = NotFound desc = could not find container \"7f483a149fea48cd7b2693254d424bcd2c31135ff7b7074a3d968d9103dd690e\": container with ID starting with 7f483a149fea48cd7b2693254d424bcd2c31135ff7b7074a3d968d9103dd690e not found: ID does not exist" Sep 11 17:09:17.289128 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:17.289101 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk"] Sep 11 17:09:17.295306 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:17.295281 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6b6653-79965c98fd-t84xk"] Sep 11 17:09:19.094567 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:19.094536 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebb32df2-192e-44f9-9eb0-dab17559e266" path="/var/lib/kubelet/pods/ebb32df2-192e-44f9-9eb0-dab17559e266/volumes" Sep 11 17:09:35.364068 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:35.364030 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk"] Sep 11 17:09:35.364614 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:35.364305 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk" podUID="893ef306-3682-46f8-9824-7f79ee6ad39a" containerName="payload-processing" containerID="cri-o://120a6a6720cbac263bd8ffbc277a4719e24ce415f1ae7d95b81e80a84d0d2590" gracePeriod=30 Sep 11 17:09:35.370126 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:35.370097 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j"] Sep 11 17:09:35.370342 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:35.370315 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j" podUID="63232f3e-a2aa-4a72-a06c-4b4d9290d03c" containerName="payload-pre-processing" containerID="cri-o://55b43a9509db9a578fafa2a2409fc6c89b1dfd6d1de8a5f254b4d32e82baec6d" gracePeriod=30 Sep 11 17:09:35.651204 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:35.651175 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk" Sep 11 17:09:35.654001 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:35.653980 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j" Sep 11 17:09:35.833048 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:35.833013 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgggq\" (UniqueName: \"kubernetes.io/projected/63232f3e-a2aa-4a72-a06c-4b4d9290d03c-kube-api-access-rgggq\") pod \"63232f3e-a2aa-4a72-a06c-4b4d9290d03c\" (UID: \"63232f3e-a2aa-4a72-a06c-4b4d9290d03c\") " Sep 11 17:09:35.833048 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:35.833048 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2w9rh\" (UniqueName: \"kubernetes.io/projected/893ef306-3682-46f8-9824-7f79ee6ad39a-kube-api-access-2w9rh\") pod \"893ef306-3682-46f8-9824-7f79ee6ad39a\" (UID: \"893ef306-3682-46f8-9824-7f79ee6ad39a\") " Sep 11 17:09:35.833260 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:35.833085 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893ef306-3682-46f8-9824-7f79ee6ad39a-plugins-config-volume\") pod \"893ef306-3682-46f8-9824-7f79ee6ad39a\" (UID: \"893ef306-3682-46f8-9824-7f79ee6ad39a\") " Sep 11 17:09:35.833260 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:35.833182 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63232f3e-a2aa-4a72-a06c-4b4d9290d03c-plugins-config-volume\") pod \"63232f3e-a2aa-4a72-a06c-4b4d9290d03c\" (UID: \"63232f3e-a2aa-4a72-a06c-4b4d9290d03c\") " Sep 11 17:09:35.833485 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:35.833460 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63232f3e-a2aa-4a72-a06c-4b4d9290d03c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "63232f3e-a2aa-4a72-a06c-4b4d9290d03c" (UID: "63232f3e-a2aa-4a72-a06c-4b4d9290d03c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:09:35.833591 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:35.833489 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/893ef306-3682-46f8-9824-7f79ee6ad39a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "893ef306-3682-46f8-9824-7f79ee6ad39a" (UID: "893ef306-3682-46f8-9824-7f79ee6ad39a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:09:35.835266 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:35.835242 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63232f3e-a2aa-4a72-a06c-4b4d9290d03c-kube-api-access-rgggq" (OuterVolumeSpecName: "kube-api-access-rgggq") pod "63232f3e-a2aa-4a72-a06c-4b4d9290d03c" (UID: "63232f3e-a2aa-4a72-a06c-4b4d9290d03c"). InnerVolumeSpecName "kube-api-access-rgggq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:09:35.835373 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:35.835279 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893ef306-3682-46f8-9824-7f79ee6ad39a-kube-api-access-2w9rh" (OuterVolumeSpecName: "kube-api-access-2w9rh") pod "893ef306-3682-46f8-9824-7f79ee6ad39a" (UID: "893ef306-3682-46f8-9824-7f79ee6ad39a"). InnerVolumeSpecName "kube-api-access-2w9rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:09:35.934172 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:35.934091 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgggq\" (UniqueName: \"kubernetes.io/projected/63232f3e-a2aa-4a72-a06c-4b4d9290d03c-kube-api-access-rgggq\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:09:35.934172 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:35.934121 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w9rh\" (UniqueName: \"kubernetes.io/projected/893ef306-3682-46f8-9824-7f79ee6ad39a-kube-api-access-2w9rh\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:09:35.934172 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:35.934131 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893ef306-3682-46f8-9824-7f79ee6ad39a-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:09:35.934172 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:35.934143 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63232f3e-a2aa-4a72-a06c-4b4d9290d03c-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:09:36.377006 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:36.376971 2577 generic.go:356] "Generic (PLEG): container finished" podID="893ef306-3682-46f8-9824-7f79ee6ad39a" containerID="120a6a6720cbac263bd8ffbc277a4719e24ce415f1ae7d95b81e80a84d0d2590" exitCode=0 Sep 11 17:09:36.377458 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:36.377034 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk" Sep 11 17:09:36.377458 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:36.377056 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk" event={"ID":"893ef306-3682-46f8-9824-7f79ee6ad39a","Type":"ContainerDied","Data":"120a6a6720cbac263bd8ffbc277a4719e24ce415f1ae7d95b81e80a84d0d2590"} Sep 11 17:09:36.377458 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:36.377120 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk" event={"ID":"893ef306-3682-46f8-9824-7f79ee6ad39a","Type":"ContainerDied","Data":"457cbadfb472adc0b68934c30cb67f3816c0eb49613a017d259c28cf298e6540"} Sep 11 17:09:36.377458 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:36.377153 2577 scope.go:117] "RemoveContainer" containerID="120a6a6720cbac263bd8ffbc277a4719e24ce415f1ae7d95b81e80a84d0d2590" Sep 11 17:09:36.378553 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:36.378444 2577 generic.go:356] "Generic (PLEG): container finished" podID="63232f3e-a2aa-4a72-a06c-4b4d9290d03c" containerID="55b43a9509db9a578fafa2a2409fc6c89b1dfd6d1de8a5f254b4d32e82baec6d" exitCode=0 Sep 11 17:09:36.378553 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:36.378512 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j" Sep 11 17:09:36.378680 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:36.378519 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j" event={"ID":"63232f3e-a2aa-4a72-a06c-4b4d9290d03c","Type":"ContainerDied","Data":"55b43a9509db9a578fafa2a2409fc6c89b1dfd6d1de8a5f254b4d32e82baec6d"} Sep 11 17:09:36.378680 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:36.378621 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j" event={"ID":"63232f3e-a2aa-4a72-a06c-4b4d9290d03c","Type":"ContainerDied","Data":"9284c4ee3fcf62bfd260fbd9a9f54bbbe930b94117888b07ba074dbbcd4d09e2"} Sep 11 17:09:36.394710 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:36.394693 2577 scope.go:117] "RemoveContainer" containerID="120a6a6720cbac263bd8ffbc277a4719e24ce415f1ae7d95b81e80a84d0d2590" Sep 11 17:09:36.394986 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:09:36.394964 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"120a6a6720cbac263bd8ffbc277a4719e24ce415f1ae7d95b81e80a84d0d2590\": container with ID starting with 120a6a6720cbac263bd8ffbc277a4719e24ce415f1ae7d95b81e80a84d0d2590 not found: ID does not exist" containerID="120a6a6720cbac263bd8ffbc277a4719e24ce415f1ae7d95b81e80a84d0d2590" Sep 11 17:09:36.395110 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:36.394995 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"120a6a6720cbac263bd8ffbc277a4719e24ce415f1ae7d95b81e80a84d0d2590"} err="failed to get container status \"120a6a6720cbac263bd8ffbc277a4719e24ce415f1ae7d95b81e80a84d0d2590\": rpc error: code = NotFound desc = could not find container \"120a6a6720cbac263bd8ffbc277a4719e24ce415f1ae7d95b81e80a84d0d2590\": container with ID starting with 120a6a6720cbac263bd8ffbc277a4719e24ce415f1ae7d95b81e80a84d0d2590 not found: ID does not exist" Sep 11 17:09:36.395110 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:36.395012 2577 scope.go:117] "RemoveContainer" containerID="55b43a9509db9a578fafa2a2409fc6c89b1dfd6d1de8a5f254b4d32e82baec6d" Sep 11 17:09:36.403271 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:36.403251 2577 scope.go:117] "RemoveContainer" containerID="55b43a9509db9a578fafa2a2409fc6c89b1dfd6d1de8a5f254b4d32e82baec6d" Sep 11 17:09:36.403635 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:09:36.403613 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b43a9509db9a578fafa2a2409fc6c89b1dfd6d1de8a5f254b4d32e82baec6d\": container with ID starting with 55b43a9509db9a578fafa2a2409fc6c89b1dfd6d1de8a5f254b4d32e82baec6d not found: ID does not exist" containerID="55b43a9509db9a578fafa2a2409fc6c89b1dfd6d1de8a5f254b4d32e82baec6d" Sep 11 17:09:36.403716 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:36.403642 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b43a9509db9a578fafa2a2409fc6c89b1dfd6d1de8a5f254b4d32e82baec6d"} err="failed to get container status \"55b43a9509db9a578fafa2a2409fc6c89b1dfd6d1de8a5f254b4d32e82baec6d\": rpc error: code = NotFound desc = could not find container \"55b43a9509db9a578fafa2a2409fc6c89b1dfd6d1de8a5f254b4d32e82baec6d\": container with ID starting with 55b43a9509db9a578fafa2a2409fc6c89b1dfd6d1de8a5f254b4d32e82baec6d not found: ID does not exist" Sep 11 17:09:36.405109 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:36.405091 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk"] Sep 11 17:09:36.408872 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:36.408853 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-bff65f-7dd7d745b5-r9pvk"] Sep 11 17:09:36.420965 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:36.420934 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j"] Sep 11 17:09:36.424142 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:36.424117 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-bff65f-6b9bf45474-xdg4j"] Sep 11 17:09:37.100184 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:37.100150 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63232f3e-a2aa-4a72-a06c-4b4d9290d03c" path="/var/lib/kubelet/pods/63232f3e-a2aa-4a72-a06c-4b4d9290d03c/volumes" Sep 11 17:09:37.100540 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:09:37.100527 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="893ef306-3682-46f8-9824-7f79ee6ad39a" path="/var/lib/kubelet/pods/893ef306-3682-46f8-9824-7f79ee6ad39a/volumes" Sep 11 17:10:05.339647 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:05.339551 2577 scope.go:117] "RemoveContainer" containerID="1e75cd41bda7722dc4c4678c7a057d8e800b8ca62989784140d9546bac386d29" Sep 11 17:10:21.770187 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:21.770148 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6"] Sep 11 17:10:33.738073 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:33.738035 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv"] Sep 11 17:10:36.789328 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:36.789284 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" podUID="00d5f85c-1121-494f-879e-c0989cf91231" containerName="main" containerID="cri-o://1edb8c8400c41dc375349405d74a0550d9e4886a132f430387b07c662c70fdc0" gracePeriod=45 Sep 11 17:10:37.042776 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.042716 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:10:37.079851 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.079822 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-home\") pod \"00d5f85c-1121-494f-879e-c0989cf91231\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " Sep 11 17:10:37.079851 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.079854 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-dshm\") pod \"00d5f85c-1121-494f-879e-c0989cf91231\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " Sep 11 17:10:37.080034 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.079875 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-tmp-dir\") pod \"00d5f85c-1121-494f-879e-c0989cf91231\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " Sep 11 17:10:37.080034 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.079978 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-model-cache\") pod \"00d5f85c-1121-494f-879e-c0989cf91231\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " Sep 11 17:10:37.080034 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.080007 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkl27\" (UniqueName: \"kubernetes.io/projected/00d5f85c-1121-494f-879e-c0989cf91231-kube-api-access-wkl27\") pod \"00d5f85c-1121-494f-879e-c0989cf91231\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " Sep 11 17:10:37.080237 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.080039 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d5f85c-1121-494f-879e-c0989cf91231-tls-certs\") pod \"00d5f85c-1121-494f-879e-c0989cf91231\" (UID: \"00d5f85c-1121-494f-879e-c0989cf91231\") " Sep 11 17:10:37.080237 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.080079 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-home" (OuterVolumeSpecName: "home") pod "00d5f85c-1121-494f-879e-c0989cf91231" (UID: "00d5f85c-1121-494f-879e-c0989cf91231"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:10:37.080237 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.080197 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "00d5f85c-1121-494f-879e-c0989cf91231" (UID: "00d5f85c-1121-494f-879e-c0989cf91231"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:10:37.080237 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.080212 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-model-cache" (OuterVolumeSpecName: "model-cache") pod "00d5f85c-1121-494f-879e-c0989cf91231" (UID: "00d5f85c-1121-494f-879e-c0989cf91231"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:10:37.080424 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.080354 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-home\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:10:37.080424 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.080372 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-tmp-dir\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:10:37.080424 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.080387 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-model-cache\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:10:37.082136 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.082114 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-dshm" (OuterVolumeSpecName: "dshm") pod "00d5f85c-1121-494f-879e-c0989cf91231" (UID: "00d5f85c-1121-494f-879e-c0989cf91231"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:10:37.082207 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.082181 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00d5f85c-1121-494f-879e-c0989cf91231-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00d5f85c-1121-494f-879e-c0989cf91231" (UID: "00d5f85c-1121-494f-879e-c0989cf91231"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 17:10:37.082207 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.082182 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d5f85c-1121-494f-879e-c0989cf91231-kube-api-access-wkl27" (OuterVolumeSpecName: "kube-api-access-wkl27") pod "00d5f85c-1121-494f-879e-c0989cf91231" (UID: "00d5f85c-1121-494f-879e-c0989cf91231"). InnerVolumeSpecName "kube-api-access-wkl27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:10:37.180878 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.180840 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkl27\" (UniqueName: \"kubernetes.io/projected/00d5f85c-1121-494f-879e-c0989cf91231-kube-api-access-wkl27\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:10:37.180878 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.180872 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d5f85c-1121-494f-879e-c0989cf91231-tls-certs\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:10:37.180878 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.180885 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d5f85c-1121-494f-879e-c0989cf91231-dshm\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:10:37.705213 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.705176 2577 generic.go:356] "Generic (PLEG): container finished" podID="00d5f85c-1121-494f-879e-c0989cf91231" containerID="1edb8c8400c41dc375349405d74a0550d9e4886a132f430387b07c662c70fdc0" exitCode=0 Sep 11 17:10:37.705213 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.705218 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" Sep 11 17:10:37.705422 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.705257 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" event={"ID":"00d5f85c-1121-494f-879e-c0989cf91231","Type":"ContainerDied","Data":"1edb8c8400c41dc375349405d74a0550d9e4886a132f430387b07c662c70fdc0"} Sep 11 17:10:37.705422 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.705304 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6" event={"ID":"00d5f85c-1121-494f-879e-c0989cf91231","Type":"ContainerDied","Data":"1604d028ec6c2cdea0d2964abd2bf897f561acd399e6a264fe24815836e83971"} Sep 11 17:10:37.705422 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.705322 2577 scope.go:117] "RemoveContainer" containerID="1edb8c8400c41dc375349405d74a0550d9e4886a132f430387b07c662c70fdc0" Sep 11 17:10:37.714456 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.714438 2577 scope.go:117] "RemoveContainer" containerID="1edb8c8400c41dc375349405d74a0550d9e4886a132f430387b07c662c70fdc0" Sep 11 17:10:37.714768 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:10:37.714746 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1edb8c8400c41dc375349405d74a0550d9e4886a132f430387b07c662c70fdc0\": container with ID starting with 1edb8c8400c41dc375349405d74a0550d9e4886a132f430387b07c662c70fdc0 not found: ID does not exist" containerID="1edb8c8400c41dc375349405d74a0550d9e4886a132f430387b07c662c70fdc0" Sep 11 17:10:37.714850 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.714778 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1edb8c8400c41dc375349405d74a0550d9e4886a132f430387b07c662c70fdc0"} err="failed to get container status \"1edb8c8400c41dc375349405d74a0550d9e4886a132f430387b07c662c70fdc0\": rpc error: code = NotFound desc = could not find container \"1edb8c8400c41dc375349405d74a0550d9e4886a132f430387b07c662c70fdc0\": container with ID starting with 1edb8c8400c41dc375349405d74a0550d9e4886a132f430387b07c662c70fdc0 not found: ID does not exist" Sep 11 17:10:37.723550 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.723526 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6"] Sep 11 17:10:37.727423 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:37.727399 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-327a3a/auth-test-model-327a3a-kserve-6769cb45d7-z5nk6"] Sep 11 17:10:39.094873 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:39.094839 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00d5f85c-1121-494f-879e-c0989cf91231" path="/var/lib/kubelet/pods/00d5f85c-1121-494f-879e-c0989cf91231/volumes" Sep 11 17:10:44.702721 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:44.702684 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz"] Sep 11 17:10:44.703350 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:44.702951 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz" podUID="9ae08bd5-de0f-44e9-ac94-8f6761648502" containerName="payload-processing" containerID="cri-o://cb4974c7885f618e7232d21c3d8f958f406bf49b40485368e210a26f293b03c5" gracePeriod=30 Sep 11 17:10:44.710736 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:44.710706 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc"] Sep 11 17:10:44.711015 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:44.710968 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc" podUID="2a899ee3-cc24-4b0c-865f-972feea82a70" containerName="payload-pre-processing" containerID="cri-o://4ac14724347c11e448aafc7a4afd6820c8921a6dfb31f82296491418208d4002" gracePeriod=30 Sep 11 17:10:44.998095 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:44.998071 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz" Sep 11 17:10:45.000717 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.000697 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc" Sep 11 17:10:45.055828 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.055798 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a899ee3-cc24-4b0c-865f-972feea82a70-plugins-config-volume\") pod \"2a899ee3-cc24-4b0c-865f-972feea82a70\" (UID: \"2a899ee3-cc24-4b0c-865f-972feea82a70\") " Sep 11 17:10:45.055997 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.055853 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ae08bd5-de0f-44e9-ac94-8f6761648502-plugins-config-volume\") pod \"9ae08bd5-de0f-44e9-ac94-8f6761648502\" (UID: \"9ae08bd5-de0f-44e9-ac94-8f6761648502\") " Sep 11 17:10:45.055997 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.055925 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kqpn\" (UniqueName: \"kubernetes.io/projected/9ae08bd5-de0f-44e9-ac94-8f6761648502-kube-api-access-8kqpn\") pod \"9ae08bd5-de0f-44e9-ac94-8f6761648502\" (UID: \"9ae08bd5-de0f-44e9-ac94-8f6761648502\") " Sep 11 17:10:45.055997 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.055973 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svtrv\" (UniqueName: \"kubernetes.io/projected/2a899ee3-cc24-4b0c-865f-972feea82a70-kube-api-access-svtrv\") pod \"2a899ee3-cc24-4b0c-865f-972feea82a70\" (UID: \"2a899ee3-cc24-4b0c-865f-972feea82a70\") " Sep 11 17:10:45.056213 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.056188 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a899ee3-cc24-4b0c-865f-972feea82a70-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a899ee3-cc24-4b0c-865f-972feea82a70" (UID: "2a899ee3-cc24-4b0c-865f-972feea82a70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:10:45.056257 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.056229 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae08bd5-de0f-44e9-ac94-8f6761648502-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ae08bd5-de0f-44e9-ac94-8f6761648502" (UID: "9ae08bd5-de0f-44e9-ac94-8f6761648502"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:10:45.058136 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.058101 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae08bd5-de0f-44e9-ac94-8f6761648502-kube-api-access-8kqpn" (OuterVolumeSpecName: "kube-api-access-8kqpn") pod "9ae08bd5-de0f-44e9-ac94-8f6761648502" (UID: "9ae08bd5-de0f-44e9-ac94-8f6761648502"). InnerVolumeSpecName "kube-api-access-8kqpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:10:45.058136 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.058122 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a899ee3-cc24-4b0c-865f-972feea82a70-kube-api-access-svtrv" (OuterVolumeSpecName: "kube-api-access-svtrv") pod "2a899ee3-cc24-4b0c-865f-972feea82a70" (UID: "2a899ee3-cc24-4b0c-865f-972feea82a70"). InnerVolumeSpecName "kube-api-access-svtrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:10:45.157832 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.157791 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a899ee3-cc24-4b0c-865f-972feea82a70-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:10:45.157832 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.157827 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ae08bd5-de0f-44e9-ac94-8f6761648502-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:10:45.158023 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.157845 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kqpn\" (UniqueName: \"kubernetes.io/projected/9ae08bd5-de0f-44e9-ac94-8f6761648502-kube-api-access-8kqpn\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:10:45.158023 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.157867 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svtrv\" (UniqueName: \"kubernetes.io/projected/2a899ee3-cc24-4b0c-865f-972feea82a70-kube-api-access-svtrv\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:10:45.749674 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.749633 2577 generic.go:356] "Generic (PLEG): container finished" podID="9ae08bd5-de0f-44e9-ac94-8f6761648502" containerID="cb4974c7885f618e7232d21c3d8f958f406bf49b40485368e210a26f293b03c5" exitCode=0 Sep 11 17:10:45.750154 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.749682 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz" Sep 11 17:10:45.750154 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.749688 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz" event={"ID":"9ae08bd5-de0f-44e9-ac94-8f6761648502","Type":"ContainerDied","Data":"cb4974c7885f618e7232d21c3d8f958f406bf49b40485368e210a26f293b03c5"} Sep 11 17:10:45.750154 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.749759 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz" event={"ID":"9ae08bd5-de0f-44e9-ac94-8f6761648502","Type":"ContainerDied","Data":"d5f3a78a63b4366db0d815309d51470e01925860265495277b4e19481ddb162a"} Sep 11 17:10:45.750154 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.749793 2577 scope.go:117] "RemoveContainer" containerID="cb4974c7885f618e7232d21c3d8f958f406bf49b40485368e210a26f293b03c5" Sep 11 17:10:45.751247 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.751114 2577 generic.go:356] "Generic (PLEG): container finished" podID="2a899ee3-cc24-4b0c-865f-972feea82a70" containerID="4ac14724347c11e448aafc7a4afd6820c8921a6dfb31f82296491418208d4002" exitCode=0 Sep 11 17:10:45.751247 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.751160 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc" Sep 11 17:10:45.751247 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.751167 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc" event={"ID":"2a899ee3-cc24-4b0c-865f-972feea82a70","Type":"ContainerDied","Data":"4ac14724347c11e448aafc7a4afd6820c8921a6dfb31f82296491418208d4002"} Sep 11 17:10:45.751247 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.751215 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc" event={"ID":"2a899ee3-cc24-4b0c-865f-972feea82a70","Type":"ContainerDied","Data":"95a8c81aca8e0e395fd03e7a70aa2628bb346d9680ad433d8f75e97cc2efca13"} Sep 11 17:10:45.759280 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.759265 2577 scope.go:117] "RemoveContainer" containerID="cb4974c7885f618e7232d21c3d8f958f406bf49b40485368e210a26f293b03c5" Sep 11 17:10:45.759636 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:10:45.759610 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb4974c7885f618e7232d21c3d8f958f406bf49b40485368e210a26f293b03c5\": container with ID starting with cb4974c7885f618e7232d21c3d8f958f406bf49b40485368e210a26f293b03c5 not found: ID does not exist" containerID="cb4974c7885f618e7232d21c3d8f958f406bf49b40485368e210a26f293b03c5" Sep 11 17:10:45.759731 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.759640 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4974c7885f618e7232d21c3d8f958f406bf49b40485368e210a26f293b03c5"} err="failed to get container status \"cb4974c7885f618e7232d21c3d8f958f406bf49b40485368e210a26f293b03c5\": rpc error: code = NotFound desc = could not find container \"cb4974c7885f618e7232d21c3d8f958f406bf49b40485368e210a26f293b03c5\": container with ID starting with cb4974c7885f618e7232d21c3d8f958f406bf49b40485368e210a26f293b03c5 not found: ID does not exist" Sep 11 17:10:45.759731 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.759656 2577 scope.go:117] "RemoveContainer" containerID="4ac14724347c11e448aafc7a4afd6820c8921a6dfb31f82296491418208d4002" Sep 11 17:10:45.768519 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.768439 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz"] Sep 11 17:10:45.770096 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.770067 2577 scope.go:117] "RemoveContainer" containerID="4ac14724347c11e448aafc7a4afd6820c8921a6dfb31f82296491418208d4002" Sep 11 17:10:45.770629 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:10:45.770603 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac14724347c11e448aafc7a4afd6820c8921a6dfb31f82296491418208d4002\": container with ID starting with 4ac14724347c11e448aafc7a4afd6820c8921a6dfb31f82296491418208d4002 not found: ID does not exist" containerID="4ac14724347c11e448aafc7a4afd6820c8921a6dfb31f82296491418208d4002" Sep 11 17:10:45.770720 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.770637 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac14724347c11e448aafc7a4afd6820c8921a6dfb31f82296491418208d4002"} err="failed to get container status \"4ac14724347c11e448aafc7a4afd6820c8921a6dfb31f82296491418208d4002\": rpc error: code = NotFound desc = could not find container \"4ac14724347c11e448aafc7a4afd6820c8921a6dfb31f82296491418208d4002\": container with ID starting with 4ac14724347c11e448aafc7a4afd6820c8921a6dfb31f82296491418208d4002 not found: ID does not exist" Sep 11 17:10:45.774091 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.774070 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2dc95c-7f8985dd8c-s8psz"] Sep 11 17:10:45.784587 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.784566 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc"] Sep 11 17:10:45.790320 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:45.790299 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2dc95c-5f974f57d6-tf8fc"] Sep 11 17:10:47.099763 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:47.099726 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a899ee3-cc24-4b0c-865f-972feea82a70" path="/var/lib/kubelet/pods/2a899ee3-cc24-4b0c-865f-972feea82a70/volumes" Sep 11 17:10:47.100151 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:47.100094 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ae08bd5-de0f-44e9-ac94-8f6761648502" path="/var/lib/kubelet/pods/9ae08bd5-de0f-44e9-ac94-8f6761648502/volumes" Sep 11 17:10:48.758623 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:48.758579 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" podUID="ffc657f2-91a2-4b62-a387-ca98962e4bd5" containerName="main" containerID="cri-o://0f9fc01d08e7c5d56d1f89104aae6fd700d702281a89bea5c6a9291a08af0cc4" gracePeriod=45 Sep 11 17:10:49.009848 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.009789 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:10:49.093183 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.093151 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-home\") pod \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " Sep 11 17:10:49.093351 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.093206 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-tmp-dir\") pod \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " Sep 11 17:10:49.093351 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.093249 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-dshm\") pod \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " Sep 11 17:10:49.093351 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.093287 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc657f2-91a2-4b62-a387-ca98962e4bd5-tls-certs\") pod \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " Sep 11 17:10:49.093351 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.093322 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkfrs\" (UniqueName: \"kubernetes.io/projected/ffc657f2-91a2-4b62-a387-ca98962e4bd5-kube-api-access-kkfrs\") pod \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " Sep 11 17:10:49.093622 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.093375 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-model-cache\") pod \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\" (UID: \"ffc657f2-91a2-4b62-a387-ca98962e4bd5\") " Sep 11 17:10:49.093622 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.093479 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-home" (OuterVolumeSpecName: "home") pod "ffc657f2-91a2-4b62-a387-ca98962e4bd5" (UID: "ffc657f2-91a2-4b62-a387-ca98962e4bd5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:10:49.093736 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.093715 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-model-cache" (OuterVolumeSpecName: "model-cache") pod "ffc657f2-91a2-4b62-a387-ca98962e4bd5" (UID: "ffc657f2-91a2-4b62-a387-ca98962e4bd5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:10:49.093796 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.093740 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-home\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:10:49.093796 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.093467 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ffc657f2-91a2-4b62-a387-ca98962e4bd5" (UID: "ffc657f2-91a2-4b62-a387-ca98962e4bd5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:10:49.096275 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.096236 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-dshm" (OuterVolumeSpecName: "dshm") pod "ffc657f2-91a2-4b62-a387-ca98962e4bd5" (UID: "ffc657f2-91a2-4b62-a387-ca98962e4bd5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:10:49.096275 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.096266 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc657f2-91a2-4b62-a387-ca98962e4bd5-kube-api-access-kkfrs" (OuterVolumeSpecName: "kube-api-access-kkfrs") pod "ffc657f2-91a2-4b62-a387-ca98962e4bd5" (UID: "ffc657f2-91a2-4b62-a387-ca98962e4bd5"). InnerVolumeSpecName "kube-api-access-kkfrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:10:49.096421 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.096354 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc657f2-91a2-4b62-a387-ca98962e4bd5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ffc657f2-91a2-4b62-a387-ca98962e4bd5" (UID: "ffc657f2-91a2-4b62-a387-ca98962e4bd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 17:10:49.194999 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.194959 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-model-cache\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:10:49.194999 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.194989 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-tmp-dir\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:10:49.194999 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.195001 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc657f2-91a2-4b62-a387-ca98962e4bd5-dshm\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:10:49.195234 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.195012 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc657f2-91a2-4b62-a387-ca98962e4bd5-tls-certs\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:10:49.195234 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.195024 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkfrs\" (UniqueName: \"kubernetes.io/projected/ffc657f2-91a2-4b62-a387-ca98962e4bd5-kube-api-access-kkfrs\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:10:49.774189 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.774149 2577 generic.go:356] "Generic (PLEG): container finished" podID="ffc657f2-91a2-4b62-a387-ca98962e4bd5" containerID="0f9fc01d08e7c5d56d1f89104aae6fd700d702281a89bea5c6a9291a08af0cc4" exitCode=0 Sep 11 17:10:49.774669 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.774196 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" Sep 11 17:10:49.774669 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.774229 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" event={"ID":"ffc657f2-91a2-4b62-a387-ca98962e4bd5","Type":"ContainerDied","Data":"0f9fc01d08e7c5d56d1f89104aae6fd700d702281a89bea5c6a9291a08af0cc4"} Sep 11 17:10:49.774669 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.774278 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv" event={"ID":"ffc657f2-91a2-4b62-a387-ca98962e4bd5","Type":"ContainerDied","Data":"83ed7900d8c48c020f0b148a74df28fdf8b80796b713ff3b42ce048a177c8c57"} Sep 11 17:10:49.774669 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.774294 2577 scope.go:117] "RemoveContainer" containerID="0f9fc01d08e7c5d56d1f89104aae6fd700d702281a89bea5c6a9291a08af0cc4" Sep 11 17:10:49.783102 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.783085 2577 scope.go:117] "RemoveContainer" containerID="0f9fc01d08e7c5d56d1f89104aae6fd700d702281a89bea5c6a9291a08af0cc4" Sep 11 17:10:49.783386 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:10:49.783366 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f9fc01d08e7c5d56d1f89104aae6fd700d702281a89bea5c6a9291a08af0cc4\": container with ID starting with 0f9fc01d08e7c5d56d1f89104aae6fd700d702281a89bea5c6a9291a08af0cc4 not found: ID does not exist" containerID="0f9fc01d08e7c5d56d1f89104aae6fd700d702281a89bea5c6a9291a08af0cc4" Sep 11 17:10:49.783437 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.783396 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9fc01d08e7c5d56d1f89104aae6fd700d702281a89bea5c6a9291a08af0cc4"} err="failed to get container status \"0f9fc01d08e7c5d56d1f89104aae6fd700d702281a89bea5c6a9291a08af0cc4\": rpc error: code = NotFound desc = could not find container \"0f9fc01d08e7c5d56d1f89104aae6fd700d702281a89bea5c6a9291a08af0cc4\": container with ID starting with 0f9fc01d08e7c5d56d1f89104aae6fd700d702281a89bea5c6a9291a08af0cc4 not found: ID does not exist" Sep 11 17:10:49.795718 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.795691 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv"] Sep 11 17:10:49.799692 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:49.799670 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2dc95c/distinct-w4-2dc95c-kserve-5bf455c9cc-8k2cv"] Sep 11 17:10:51.093721 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:10:51.093689 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffc657f2-91a2-4b62-a387-ca98962e4bd5" path="/var/lib/kubelet/pods/ffc657f2-91a2-4b62-a387-ca98962e4bd5/volumes" Sep 11 17:13:05.138863 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:05.138835 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 17:13:05.142855 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:05.142832 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 17:13:35.331546 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.331485 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m"] Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332012 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a899ee3-cc24-4b0c-865f-972feea82a70" containerName="payload-pre-processing" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332033 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a899ee3-cc24-4b0c-865f-972feea82a70" containerName="payload-pre-processing" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332045 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb32df2-192e-44f9-9eb0-dab17559e266" containerName="payload-pre-processing" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332054 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb32df2-192e-44f9-9eb0-dab17559e266" containerName="payload-pre-processing" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332066 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ae08bd5-de0f-44e9-ac94-8f6761648502" containerName="payload-processing" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332074 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ae08bd5-de0f-44e9-ac94-8f6761648502" containerName="payload-processing" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332087 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63232f3e-a2aa-4a72-a06c-4b4d9290d03c" containerName="payload-pre-processing" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332095 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="63232f3e-a2aa-4a72-a06c-4b4d9290d03c" containerName="payload-pre-processing" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332121 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc657f2-91a2-4b62-a387-ca98962e4bd5" containerName="main" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332131 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc657f2-91a2-4b62-a387-ca98962e4bd5" containerName="main" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332141 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="893ef306-3682-46f8-9824-7f79ee6ad39a" containerName="payload-processing" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332150 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="893ef306-3682-46f8-9824-7f79ee6ad39a" containerName="payload-processing" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332169 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d5f85c-1121-494f-879e-c0989cf91231" containerName="main" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332178 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d5f85c-1121-494f-879e-c0989cf91231" containerName="main" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332187 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58f04664-ad0c-49a4-a3e6-bc7d0d862ae4" containerName="revoke-keys" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332192 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="58f04664-ad0c-49a4-a3e6-bc7d0d862ae4" containerName="revoke-keys" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332275 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="893ef306-3682-46f8-9824-7f79ee6ad39a" containerName="payload-processing" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332283 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a899ee3-cc24-4b0c-865f-972feea82a70" containerName="payload-pre-processing" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332289 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ae08bd5-de0f-44e9-ac94-8f6761648502" containerName="payload-processing" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332294 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="63232f3e-a2aa-4a72-a06c-4b4d9290d03c" containerName="payload-pre-processing" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332300 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="58f04664-ad0c-49a4-a3e6-bc7d0d862ae4" containerName="revoke-keys" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332305 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb32df2-192e-44f9-9eb0-dab17559e266" containerName="payload-pre-processing" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332312 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffc657f2-91a2-4b62-a387-ca98962e4bd5" containerName="main" Sep 11 17:13:35.334037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.332317 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d5f85c-1121-494f-879e-c0989cf91231" containerName="main" Sep 11 17:13:35.335242 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.335225 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m" Sep 11 17:13:35.342802 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.342781 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-269ba8\"" Sep 11 17:13:35.363327 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.363295 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m"] Sep 11 17:13:35.407110 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.407073 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29"] Sep 11 17:13:35.410646 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.410623 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29" Sep 11 17:13:35.413549 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.413526 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-269ba8-dockercfg-hqt9k\"" Sep 11 17:13:35.423832 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.423809 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29"] Sep 11 17:13:35.424892 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.424865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3019d3c0-2f0b-40c8-b3ad-08afd0896cb1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29\" (UID: \"3019d3c0-2f0b-40c8-b3ad-08afd0896cb1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29" Sep 11 17:13:35.424977 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.424931 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppf9m\" (UniqueName: \"kubernetes.io/projected/13f74896-5d21-4a91-ab55-1d4d7c5bb2e1-kube-api-access-ppf9m\") pod \"payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m\" (UID: \"13f74896-5d21-4a91-ab55-1d4d7c5bb2e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m" Sep 11 17:13:35.425018 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.424993 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb5jg\" (UniqueName: \"kubernetes.io/projected/3019d3c0-2f0b-40c8-b3ad-08afd0896cb1-kube-api-access-cb5jg\") pod \"payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29\" (UID: \"3019d3c0-2f0b-40c8-b3ad-08afd0896cb1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29" Sep 11 17:13:35.425059 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.425036 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f74896-5d21-4a91-ab55-1d4d7c5bb2e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m\" (UID: \"13f74896-5d21-4a91-ab55-1d4d7c5bb2e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m" Sep 11 17:13:35.525678 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.525651 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3019d3c0-2f0b-40c8-b3ad-08afd0896cb1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29\" (UID: \"3019d3c0-2f0b-40c8-b3ad-08afd0896cb1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29" Sep 11 17:13:35.525872 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.525711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppf9m\" (UniqueName: \"kubernetes.io/projected/13f74896-5d21-4a91-ab55-1d4d7c5bb2e1-kube-api-access-ppf9m\") pod \"payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m\" (UID: \"13f74896-5d21-4a91-ab55-1d4d7c5bb2e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m" Sep 11 17:13:35.525872 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.525758 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb5jg\" (UniqueName: \"kubernetes.io/projected/3019d3c0-2f0b-40c8-b3ad-08afd0896cb1-kube-api-access-cb5jg\") pod \"payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29\" (UID: \"3019d3c0-2f0b-40c8-b3ad-08afd0896cb1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29" Sep 11 17:13:35.526006 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.525896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f74896-5d21-4a91-ab55-1d4d7c5bb2e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m\" (UID: \"13f74896-5d21-4a91-ab55-1d4d7c5bb2e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m" Sep 11 17:13:35.526551 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.526525 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3019d3c0-2f0b-40c8-b3ad-08afd0896cb1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29\" (UID: \"3019d3c0-2f0b-40c8-b3ad-08afd0896cb1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29" Sep 11 17:13:35.526821 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.526802 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f74896-5d21-4a91-ab55-1d4d7c5bb2e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m\" (UID: \"13f74896-5d21-4a91-ab55-1d4d7c5bb2e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m" Sep 11 17:13:35.535004 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.534984 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb5jg\" (UniqueName: \"kubernetes.io/projected/3019d3c0-2f0b-40c8-b3ad-08afd0896cb1-kube-api-access-cb5jg\") pod \"payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29\" (UID: \"3019d3c0-2f0b-40c8-b3ad-08afd0896cb1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29" Sep 11 17:13:35.535129 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.535107 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppf9m\" (UniqueName: \"kubernetes.io/projected/13f74896-5d21-4a91-ab55-1d4d7c5bb2e1-kube-api-access-ppf9m\") pod \"payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m\" (UID: \"13f74896-5d21-4a91-ab55-1d4d7c5bb2e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m" Sep 11 17:13:35.645670 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.645582 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m" Sep 11 17:13:35.720978 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.720937 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29" Sep 11 17:13:35.917463 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:35.917423 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m"] Sep 11 17:13:35.920027 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:13:35.919999 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f74896_5d21_4a91_ab55_1d4d7c5bb2e1.slice/crio-7dc4fb94715a4788dd0da64e14dedbfff9dbb1483b4434fc36441a58f9fb5e88 WatchSource:0}: Error finding container 7dc4fb94715a4788dd0da64e14dedbfff9dbb1483b4434fc36441a58f9fb5e88: Status 404 returned error can't find the container with id 7dc4fb94715a4788dd0da64e14dedbfff9dbb1483b4434fc36441a58f9fb5e88 Sep 11 17:13:36.006950 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:36.006912 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29"] Sep 11 17:13:36.009892 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:13:36.009865 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3019d3c0_2f0b_40c8_b3ad_08afd0896cb1.slice/crio-a78069ea6b46f24a76b2f8cb539021e52a70850a3a5fe6008e0901ba847d80a2 WatchSource:0}: Error finding container a78069ea6b46f24a76b2f8cb539021e52a70850a3a5fe6008e0901ba847d80a2: Status 404 returned error can't find the container with id a78069ea6b46f24a76b2f8cb539021e52a70850a3a5fe6008e0901ba847d80a2 Sep 11 17:13:36.603456 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:36.603419 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29" event={"ID":"3019d3c0-2f0b-40c8-b3ad-08afd0896cb1","Type":"ContainerStarted","Data":"a1231f51aeb7c4ab7ae5a5bf4b24c35ec4784a063e0e3eb35b03aee9bafa9412"} Sep 11 17:13:36.603958 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:36.603478 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29" event={"ID":"3019d3c0-2f0b-40c8-b3ad-08afd0896cb1","Type":"ContainerStarted","Data":"a78069ea6b46f24a76b2f8cb539021e52a70850a3a5fe6008e0901ba847d80a2"} Sep 11 17:13:36.603958 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:36.603612 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29" Sep 11 17:13:36.605098 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:36.605079 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m" event={"ID":"13f74896-5d21-4a91-ab55-1d4d7c5bb2e1","Type":"ContainerStarted","Data":"0ec729afd7a5eff5c6f36e9e714daca6235f1cd2c0d05f4d3d88bc3978db3336"} Sep 11 17:13:36.605190 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:36.605114 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m" event={"ID":"13f74896-5d21-4a91-ab55-1d4d7c5bb2e1","Type":"ContainerStarted","Data":"7dc4fb94715a4788dd0da64e14dedbfff9dbb1483b4434fc36441a58f9fb5e88"} Sep 11 17:13:36.605190 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:36.605140 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m" Sep 11 17:13:36.625592 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:36.625532 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29" podStartSLOduration=1.625487366 podStartE2EDuration="1.625487366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:13:36.624010401 +0000 UTC m=+1832.032112694" watchObservedRunningTime="2026-09-11 17:13:36.625487366 +0000 UTC m=+1832.033589629" Sep 11 17:13:36.645134 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:36.645079 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m" podStartSLOduration=1.64506156 podStartE2EDuration="1.64506156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:13:36.642620996 +0000 UTC m=+1832.050723290" watchObservedRunningTime="2026-09-11 17:13:36.64506156 +0000 UTC m=+1832.053163831" Sep 11 17:13:47.613081 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:47.613046 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29" Sep 11 17:13:47.613587 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:13:47.613128 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m" Sep 11 17:14:05.175065 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:05.175028 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh"] Sep 11 17:14:05.178699 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:05.178681 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh" Sep 11 17:14:05.181725 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:05.181702 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-750358\"" Sep 11 17:14:05.181839 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:05.181739 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-750358-dockercfg-7k7g9\"" Sep 11 17:14:05.189794 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:05.189773 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh"] Sep 11 17:14:05.314928 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:05.314889 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38cd6edf-249e-4f12-bd0b-1e82cc33d0ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh\" (UID: \"38cd6edf-249e-4f12-bd0b-1e82cc33d0ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh" Sep 11 17:14:05.315077 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:05.315028 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlk5l\" (UniqueName: \"kubernetes.io/projected/38cd6edf-249e-4f12-bd0b-1e82cc33d0ed-kube-api-access-nlk5l\") pod \"payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh\" (UID: \"38cd6edf-249e-4f12-bd0b-1e82cc33d0ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh" Sep 11 17:14:05.415681 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:05.415648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlk5l\" (UniqueName: \"kubernetes.io/projected/38cd6edf-249e-4f12-bd0b-1e82cc33d0ed-kube-api-access-nlk5l\") pod \"payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh\" (UID: \"38cd6edf-249e-4f12-bd0b-1e82cc33d0ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh" Sep 11 17:14:05.415884 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:05.415697 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38cd6edf-249e-4f12-bd0b-1e82cc33d0ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh\" (UID: \"38cd6edf-249e-4f12-bd0b-1e82cc33d0ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh" Sep 11 17:14:05.416353 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:05.416335 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38cd6edf-249e-4f12-bd0b-1e82cc33d0ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh\" (UID: \"38cd6edf-249e-4f12-bd0b-1e82cc33d0ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh" Sep 11 17:14:05.424280 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:05.424247 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlk5l\" (UniqueName: \"kubernetes.io/projected/38cd6edf-249e-4f12-bd0b-1e82cc33d0ed-kube-api-access-nlk5l\") pod \"payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh\" (UID: \"38cd6edf-249e-4f12-bd0b-1e82cc33d0ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh" Sep 11 17:14:05.489999 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:05.489963 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh" Sep 11 17:14:05.774268 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:05.774243 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh"] Sep 11 17:14:05.775888 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:14:05.775861 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38cd6edf_249e_4f12_bd0b_1e82cc33d0ed.slice/crio-a3e33c69f1b6941d63f80ed0284dc486971442b94f5e6986c2dee804b31b4af0 WatchSource:0}: Error finding container a3e33c69f1b6941d63f80ed0284dc486971442b94f5e6986c2dee804b31b4af0: Status 404 returned error can't find the container with id a3e33c69f1b6941d63f80ed0284dc486971442b94f5e6986c2dee804b31b4af0 Sep 11 17:14:06.765912 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:06.765878 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh" event={"ID":"38cd6edf-249e-4f12-bd0b-1e82cc33d0ed","Type":"ContainerStarted","Data":"ba981c3e4056490ed18002ff13050565c8ac0afb89bc4ace268038e3c8ab7084"} Sep 11 17:14:06.766340 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:06.765941 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh" Sep 11 17:14:06.766340 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:06.765962 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh" event={"ID":"38cd6edf-249e-4f12-bd0b-1e82cc33d0ed","Type":"ContainerStarted","Data":"a3e33c69f1b6941d63f80ed0284dc486971442b94f5e6986c2dee804b31b4af0"} Sep 11 17:14:06.788259 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:06.788203 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh" podStartSLOduration=1.788186957 podStartE2EDuration="1.788186957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:14:06.786741411 +0000 UTC m=+1862.194843701" watchObservedRunningTime="2026-09-11 17:14:06.788186957 +0000 UTC m=+1862.196289226" Sep 11 17:14:17.774073 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:17.774038 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh" Sep 11 17:14:31.849623 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:31.849540 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh_38cd6edf-249e-4f12-bd0b-1e82cc33d0ed/payload-processing/0.log" Sep 11 17:14:57.377192 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:57.377157 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29_3019d3c0-2f0b-40c8-b3ad-08afd0896cb1/payload-processing/0.log" Sep 11 17:14:57.619361 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:14:57.619315 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh_38cd6edf-249e-4f12-bd0b-1e82cc33d0ed/payload-processing/0.log" Sep 11 17:15:00.136556 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:00.136510 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-269ba8-29819115-xz28z"] Sep 11 17:15:00.140402 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:00.140380 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-750358-29819115-wc9hd"] Sep 11 17:15:00.140549 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:00.140491 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-269ba8-29819115-xz28z" Sep 11 17:15:00.143523 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:00.143478 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-327a3a-29819115-hhhkq"] Sep 11 17:15:00.143654 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:00.143580 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-750358-29819115-wc9hd" Sep 11 17:15:00.144096 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:00.144073 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pc4qb\"" Sep 11 17:15:00.146955 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:00.146934 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-327a3a-29819115-hhhkq" Sep 11 17:15:00.149092 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:00.148971 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-269ba8-29819115-xz28z"] Sep 11 17:15:00.153076 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:00.152459 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-750358-29819115-wc9hd"] Sep 11 17:15:00.154782 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:00.154765 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-269ba8-29819115-xz28z" Sep 11 17:15:00.159333 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:00.158668 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-327a3a-29819115-hhhkq"] Sep 11 17:15:00.165268 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:00.165247 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-750358-29819115-wc9hd" Sep 11 17:15:00.199391 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:00.199300 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-327a3a-29819115-hhhkq" Sep 11 17:15:00.447794 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:00.447769 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-269ba8-29819115-xz28z"] Sep 11 17:15:00.449537 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:15:00.449506 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ae029f_e63a_4c3c_af03_8993975e93b3.slice/crio-e6a810eab437399ddc687c9bcd83d258aa29cc09ea75a049ffcf3dd2f564e69d WatchSource:0}: Error finding container e6a810eab437399ddc687c9bcd83d258aa29cc09ea75a049ffcf3dd2f564e69d: Status 404 returned error can't find the container with id e6a810eab437399ddc687c9bcd83d258aa29cc09ea75a049ffcf3dd2f564e69d Sep 11 17:15:00.458146 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:00.458121 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-750358-29819115-wc9hd"] Sep 11 17:15:00.462931 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:15:00.462909 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f7a9d8_3262_43f3_8244_1fd0dffee7e6.slice/crio-0c7c1ce4b3371b76729e521d229ce56fdd38cdbdf6efafa20637f3931c58b335 WatchSource:0}: Error finding container 0c7c1ce4b3371b76729e521d229ce56fdd38cdbdf6efafa20637f3931c58b335: Status 404 returned error can't find the container with id 0c7c1ce4b3371b76729e521d229ce56fdd38cdbdf6efafa20637f3931c58b335 Sep 11 17:15:00.509875 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:00.506866 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-327a3a-29819115-hhhkq"] Sep 11 17:15:00.510991 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:15:00.510248 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcde8e3e_b010_44ea_816b_9764c44bdcab.slice/crio-79e81644f59021e189ab2d15b8973d6362f1e36d1f8ae73589b85a95f56a8ddd WatchSource:0}: Error finding container 79e81644f59021e189ab2d15b8973d6362f1e36d1f8ae73589b85a95f56a8ddd: Status 404 returned error can't find the container with id 79e81644f59021e189ab2d15b8973d6362f1e36d1f8ae73589b85a95f56a8ddd Sep 11 17:15:01.052289 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:01.052254 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-750358-29819115-wc9hd" event={"ID":"f4f7a9d8-3262-43f3-8244-1fd0dffee7e6","Type":"ContainerStarted","Data":"d8acc6eeba2e73c1c40b0b8b4e1cd323b0ef07e7ad557ad747806b2a67ae8e6d"} Sep 11 17:15:01.052471 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:01.052320 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-750358-29819115-wc9hd" event={"ID":"f4f7a9d8-3262-43f3-8244-1fd0dffee7e6","Type":"ContainerStarted","Data":"0c7c1ce4b3371b76729e521d229ce56fdd38cdbdf6efafa20637f3931c58b335"} Sep 11 17:15:01.053858 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:01.053828 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-269ba8-29819115-xz28z" event={"ID":"57ae029f-e63a-4c3c-af03-8993975e93b3","Type":"ContainerStarted","Data":"c8ca20aeff0835c5d7af3a8b7b81b5e14586df96faa64656a2aff5e944c4225f"} Sep 11 17:15:01.053992 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:01.053876 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-269ba8-29819115-xz28z" event={"ID":"57ae029f-e63a-4c3c-af03-8993975e93b3","Type":"ContainerStarted","Data":"e6a810eab437399ddc687c9bcd83d258aa29cc09ea75a049ffcf3dd2f564e69d"} Sep 11 17:15:01.055348 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:01.055328 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-327a3a-29819115-hhhkq" event={"ID":"bcde8e3e-b010-44ea-816b-9764c44bdcab","Type":"ContainerStarted","Data":"4dfbb81c4cf8da0c54ee8414443eeff15ea501f27fadf8047d23a7ed3cf0eccd"} Sep 11 17:15:01.055447 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:01.055364 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-327a3a-29819115-hhhkq" event={"ID":"bcde8e3e-b010-44ea-816b-9764c44bdcab","Type":"ContainerStarted","Data":"79e81644f59021e189ab2d15b8973d6362f1e36d1f8ae73589b85a95f56a8ddd"} Sep 11 17:15:01.074045 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:01.073973 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-750358-29819115-wc9hd" podStartSLOduration=1.073958549 podStartE2EDuration="1.073958549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:15:01.072051261 +0000 UTC m=+1916.480153529" watchObservedRunningTime="2026-09-11 17:15:01.073958549 +0000 UTC m=+1916.482060818" Sep 11 17:15:01.093553 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:01.093490 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-327a3a-29819115-hhhkq" podStartSLOduration=1.093474397 podStartE2EDuration="1.093474397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:15:01.091180844 +0000 UTC m=+1916.499283114" watchObservedRunningTime="2026-09-11 17:15:01.093474397 +0000 UTC m=+1916.501576667" Sep 11 17:15:01.110398 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:01.110349 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-269ba8-29819115-xz28z" podStartSLOduration=1.110334605 podStartE2EDuration="1.110334605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:15:01.108755527 +0000 UTC m=+1916.516857821" watchObservedRunningTime="2026-09-11 17:15:01.110334605 +0000 UTC m=+1916.518436904" Sep 11 17:15:02.061760 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:02.061726 2577 generic.go:356] "Generic (PLEG): container finished" podID="f4f7a9d8-3262-43f3-8244-1fd0dffee7e6" containerID="d8acc6eeba2e73c1c40b0b8b4e1cd323b0ef07e7ad557ad747806b2a67ae8e6d" exitCode=0 Sep 11 17:15:02.062244 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:02.061818 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-750358-29819115-wc9hd" event={"ID":"f4f7a9d8-3262-43f3-8244-1fd0dffee7e6","Type":"ContainerDied","Data":"d8acc6eeba2e73c1c40b0b8b4e1cd323b0ef07e7ad557ad747806b2a67ae8e6d"} Sep 11 17:15:02.063423 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:02.063402 2577 generic.go:356] "Generic (PLEG): container finished" podID="57ae029f-e63a-4c3c-af03-8993975e93b3" containerID="c8ca20aeff0835c5d7af3a8b7b81b5e14586df96faa64656a2aff5e944c4225f" exitCode=0 Sep 11 17:15:02.063566 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:02.063458 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-269ba8-29819115-xz28z" event={"ID":"57ae029f-e63a-4c3c-af03-8993975e93b3","Type":"ContainerDied","Data":"c8ca20aeff0835c5d7af3a8b7b81b5e14586df96faa64656a2aff5e944c4225f"} Sep 11 17:15:02.064944 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:02.064925 2577 generic.go:356] "Generic (PLEG): container finished" podID="bcde8e3e-b010-44ea-816b-9764c44bdcab" containerID="4dfbb81c4cf8da0c54ee8414443eeff15ea501f27fadf8047d23a7ed3cf0eccd" exitCode=0 Sep 11 17:15:02.065037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:02.064996 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-327a3a-29819115-hhhkq" event={"ID":"bcde8e3e-b010-44ea-816b-9764c44bdcab","Type":"ContainerDied","Data":"4dfbb81c4cf8da0c54ee8414443eeff15ea501f27fadf8047d23a7ed3cf0eccd"} Sep 11 17:15:03.226354 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:03.226322 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-269ba8-29819115-xz28z" Sep 11 17:15:03.273141 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:03.273114 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-327a3a-29819115-hhhkq" Sep 11 17:15:03.276448 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:03.276418 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-750358-29819115-wc9hd" Sep 11 17:15:04.078574 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:04.078482 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-750358-29819115-wc9hd" Sep 11 17:15:04.078574 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:04.078537 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-750358-29819115-wc9hd" event={"ID":"f4f7a9d8-3262-43f3-8244-1fd0dffee7e6","Type":"ContainerDied","Data":"0c7c1ce4b3371b76729e521d229ce56fdd38cdbdf6efafa20637f3931c58b335"} Sep 11 17:15:04.078813 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:04.078588 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c7c1ce4b3371b76729e521d229ce56fdd38cdbdf6efafa20637f3931c58b335" Sep 11 17:15:04.079948 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:04.079930 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-269ba8-29819115-xz28z" Sep 11 17:15:04.079948 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:04.079945 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-269ba8-29819115-xz28z" event={"ID":"57ae029f-e63a-4c3c-af03-8993975e93b3","Type":"ContainerDied","Data":"e6a810eab437399ddc687c9bcd83d258aa29cc09ea75a049ffcf3dd2f564e69d"} Sep 11 17:15:04.080150 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:04.079979 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6a810eab437399ddc687c9bcd83d258aa29cc09ea75a049ffcf3dd2f564e69d" Sep 11 17:15:04.081426 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:04.081402 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-327a3a-29819115-hhhkq" event={"ID":"bcde8e3e-b010-44ea-816b-9764c44bdcab","Type":"ContainerDied","Data":"79e81644f59021e189ab2d15b8973d6362f1e36d1f8ae73589b85a95f56a8ddd"} Sep 11 17:15:04.081585 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:04.081433 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-327a3a-29819115-hhhkq" Sep 11 17:15:04.081585 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:04.081443 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79e81644f59021e189ab2d15b8973d6362f1e36d1f8ae73589b85a95f56a8ddd" Sep 11 17:15:05.413904 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.413872 2577 scope.go:117] "RemoveContainer" containerID="61bf70e580aea25ebee883d56915b216a8fa2fe866f12ff7f088a00e62747dc1" Sep 11 17:15:05.505847 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.505788 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m"] Sep 11 17:15:05.506230 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.506214 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57ae029f-e63a-4c3c-af03-8993975e93b3" containerName="cleanup" Sep 11 17:15:05.506287 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.506232 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="57ae029f-e63a-4c3c-af03-8993975e93b3" containerName="cleanup" Sep 11 17:15:05.506287 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.506240 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4f7a9d8-3262-43f3-8244-1fd0dffee7e6" containerName="cleanup" Sep 11 17:15:05.506287 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.506247 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4f7a9d8-3262-43f3-8244-1fd0dffee7e6" containerName="cleanup" Sep 11 17:15:05.506287 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.506260 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcde8e3e-b010-44ea-816b-9764c44bdcab" containerName="cleanup" Sep 11 17:15:05.506287 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.506265 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcde8e3e-b010-44ea-816b-9764c44bdcab" containerName="cleanup" Sep 11 17:15:05.506447 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.506326 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="57ae029f-e63a-4c3c-af03-8993975e93b3" containerName="cleanup" Sep 11 17:15:05.506447 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.506335 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcde8e3e-b010-44ea-816b-9764c44bdcab" containerName="cleanup" Sep 11 17:15:05.506447 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.506341 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4f7a9d8-3262-43f3-8244-1fd0dffee7e6" containerName="cleanup" Sep 11 17:15:05.511013 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.510993 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m" Sep 11 17:15:05.513873 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.513847 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-891cc8-dockercfg-lvdsn\"" Sep 11 17:15:05.514040 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.514021 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-891cc8\"" Sep 11 17:15:05.519362 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.519334 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m"] Sep 11 17:15:05.579863 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.579829 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcm2g\" (UniqueName: \"kubernetes.io/projected/8c02a32d-f97f-4598-bc6a-643565c009cf-kube-api-access-lcm2g\") pod \"payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m\" (UID: \"8c02a32d-f97f-4598-bc6a-643565c009cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m" Sep 11 17:15:05.580047 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.579883 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c02a32d-f97f-4598-bc6a-643565c009cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m\" (UID: \"8c02a32d-f97f-4598-bc6a-643565c009cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m" Sep 11 17:15:05.681201 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.681101 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcm2g\" (UniqueName: \"kubernetes.io/projected/8c02a32d-f97f-4598-bc6a-643565c009cf-kube-api-access-lcm2g\") pod \"payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m\" (UID: \"8c02a32d-f97f-4598-bc6a-643565c009cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m" Sep 11 17:15:05.681201 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.681156 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c02a32d-f97f-4598-bc6a-643565c009cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m\" (UID: \"8c02a32d-f97f-4598-bc6a-643565c009cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m" Sep 11 17:15:05.681940 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.681920 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c02a32d-f97f-4598-bc6a-643565c009cf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m\" (UID: \"8c02a32d-f97f-4598-bc6a-643565c009cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m" Sep 11 17:15:05.690476 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.690439 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcm2g\" (UniqueName: \"kubernetes.io/projected/8c02a32d-f97f-4598-bc6a-643565c009cf-kube-api-access-lcm2g\") pod \"payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m\" (UID: \"8c02a32d-f97f-4598-bc6a-643565c009cf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m" Sep 11 17:15:05.824226 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:05.824184 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m" Sep 11 17:15:06.098221 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:06.098199 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m"] Sep 11 17:15:06.100187 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:15:06.100158 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c02a32d_f97f_4598_bc6a_643565c009cf.slice/crio-3eff406af152f7a5bf506df9fc8ad1517efd58d7911d34e192f2b10457146cdd WatchSource:0}: Error finding container 3eff406af152f7a5bf506df9fc8ad1517efd58d7911d34e192f2b10457146cdd: Status 404 returned error can't find the container with id 3eff406af152f7a5bf506df9fc8ad1517efd58d7911d34e192f2b10457146cdd Sep 11 17:15:07.098362 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:07.098325 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m" event={"ID":"8c02a32d-f97f-4598-bc6a-643565c009cf","Type":"ContainerStarted","Data":"59c3706c549a37f1ded1c2a6b8761a8dfa60e2d90b2e172816679bd1de90f44e"} Sep 11 17:15:07.098772 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:07.098387 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m" event={"ID":"8c02a32d-f97f-4598-bc6a-643565c009cf","Type":"ContainerStarted","Data":"3eff406af152f7a5bf506df9fc8ad1517efd58d7911d34e192f2b10457146cdd"} Sep 11 17:15:07.098772 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:07.098524 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m" Sep 11 17:15:07.117484 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:07.117440 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m" podStartSLOduration=2.117426803 podStartE2EDuration="2.117426803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:15:07.116358713 +0000 UTC m=+1922.524460982" watchObservedRunningTime="2026-09-11 17:15:07.117426803 +0000 UTC m=+1922.525529157" Sep 11 17:15:18.106213 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:18.106177 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m" Sep 11 17:15:31.944678 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:31.944634 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-891cc8-6jq2x"] Sep 11 17:15:31.949787 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:31.949762 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-891cc8-6jq2x" Sep 11 17:15:31.952554 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:31.952531 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pc4qb\"" Sep 11 17:15:31.952687 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:31.952533 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 17:15:31.956776 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:31.956753 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-891cc8-6jq2x"] Sep 11 17:15:32.024925 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:32.024885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19fc6025-6492-4cc0-9b91-e6cee0bb042e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-891cc8-6jq2x\" (UID: \"19fc6025-6492-4cc0-9b91-e6cee0bb042e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-891cc8-6jq2x" Sep 11 17:15:32.126263 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:32.126233 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19fc6025-6492-4cc0-9b91-e6cee0bb042e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-891cc8-6jq2x\" (UID: \"19fc6025-6492-4cc0-9b91-e6cee0bb042e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-891cc8-6jq2x" Sep 11 17:15:32.126875 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:32.126856 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19fc6025-6492-4cc0-9b91-e6cee0bb042e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-891cc8-6jq2x\" (UID: \"19fc6025-6492-4cc0-9b91-e6cee0bb042e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-891cc8-6jq2x" Sep 11 17:15:32.262332 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:32.262298 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-891cc8-6jq2x" Sep 11 17:15:32.536637 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:32.536611 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-891cc8-6jq2x"] Sep 11 17:15:32.538977 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:15:32.538952 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19fc6025_6492_4cc0_9b91_e6cee0bb042e.slice/crio-1fd8233c6b2220c6a781e194dcc7474d30c60121ae49d23863e14190318fafa7 WatchSource:0}: Error finding container 1fd8233c6b2220c6a781e194dcc7474d30c60121ae49d23863e14190318fafa7: Status 404 returned error can't find the container with id 1fd8233c6b2220c6a781e194dcc7474d30c60121ae49d23863e14190318fafa7 Sep 11 17:15:33.230900 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:33.230874 2577 generic.go:356] "Generic (PLEG): container finished" podID="19fc6025-6492-4cc0-9b91-e6cee0bb042e" containerID="c3728b89ca9ff18c35239acc0358ea547279fc277bf8dd7d1312e8bb05e94fcb" exitCode=0 Sep 11 17:15:33.231230 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:33.230953 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-891cc8-6jq2x" event={"ID":"19fc6025-6492-4cc0-9b91-e6cee0bb042e","Type":"ContainerDied","Data":"c3728b89ca9ff18c35239acc0358ea547279fc277bf8dd7d1312e8bb05e94fcb"} Sep 11 17:15:33.231230 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:33.231003 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-891cc8-6jq2x" event={"ID":"19fc6025-6492-4cc0-9b91-e6cee0bb042e","Type":"ContainerStarted","Data":"1fd8233c6b2220c6a781e194dcc7474d30c60121ae49d23863e14190318fafa7"} Sep 11 17:15:34.379177 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:34.379151 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-891cc8-6jq2x" Sep 11 17:15:34.446573 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:34.446476 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19fc6025-6492-4cc0-9b91-e6cee0bb042e-maas-api-service-ca\") pod \"19fc6025-6492-4cc0-9b91-e6cee0bb042e\" (UID: \"19fc6025-6492-4cc0-9b91-e6cee0bb042e\") " Sep 11 17:15:34.446827 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:34.446806 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19fc6025-6492-4cc0-9b91-e6cee0bb042e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "19fc6025-6492-4cc0-9b91-e6cee0bb042e" (UID: "19fc6025-6492-4cc0-9b91-e6cee0bb042e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:15:34.547576 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:34.547474 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19fc6025-6492-4cc0-9b91-e6cee0bb042e-maas-api-service-ca\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:15:35.243197 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:35.243169 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-891cc8-6jq2x" Sep 11 17:15:35.243370 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:35.243206 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-891cc8-6jq2x" event={"ID":"19fc6025-6492-4cc0-9b91-e6cee0bb042e","Type":"ContainerDied","Data":"1fd8233c6b2220c6a781e194dcc7474d30c60121ae49d23863e14190318fafa7"} Sep 11 17:15:35.243370 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:35.243254 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fd8233c6b2220c6a781e194dcc7474d30c60121ae49d23863e14190318fafa7" Sep 11 17:15:43.043541 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:43.043490 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m"] Sep 11 17:15:43.043974 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:43.043803 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m" podUID="8c02a32d-f97f-4598-bc6a-643565c009cf" containerName="payload-processing" containerID="cri-o://59c3706c549a37f1ded1c2a6b8761a8dfa60e2d90b2e172816679bd1de90f44e" gracePeriod=30 Sep 11 17:15:43.208549 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:43.208514 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-891cc8-6jq2x"] Sep 11 17:15:43.213954 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:43.213925 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-891cc8-6jq2x"] Sep 11 17:15:43.293479 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:43.293451 2577 generic.go:356] "Generic (PLEG): container finished" podID="8c02a32d-f97f-4598-bc6a-643565c009cf" containerID="59c3706c549a37f1ded1c2a6b8761a8dfa60e2d90b2e172816679bd1de90f44e" exitCode=0 Sep 11 17:15:43.293663 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:43.293541 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m" event={"ID":"8c02a32d-f97f-4598-bc6a-643565c009cf","Type":"ContainerDied","Data":"59c3706c549a37f1ded1c2a6b8761a8dfa60e2d90b2e172816679bd1de90f44e"} Sep 11 17:15:43.313311 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:43.313271 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m" Sep 11 17:15:43.426795 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:43.426758 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcm2g\" (UniqueName: \"kubernetes.io/projected/8c02a32d-f97f-4598-bc6a-643565c009cf-kube-api-access-lcm2g\") pod \"8c02a32d-f97f-4598-bc6a-643565c009cf\" (UID: \"8c02a32d-f97f-4598-bc6a-643565c009cf\") " Sep 11 17:15:43.426989 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:43.426833 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c02a32d-f97f-4598-bc6a-643565c009cf-plugins-config-volume\") pod \"8c02a32d-f97f-4598-bc6a-643565c009cf\" (UID: \"8c02a32d-f97f-4598-bc6a-643565c009cf\") " Sep 11 17:15:43.427266 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:43.427240 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c02a32d-f97f-4598-bc6a-643565c009cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8c02a32d-f97f-4598-bc6a-643565c009cf" (UID: "8c02a32d-f97f-4598-bc6a-643565c009cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:15:43.428978 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:43.428953 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c02a32d-f97f-4598-bc6a-643565c009cf-kube-api-access-lcm2g" (OuterVolumeSpecName: "kube-api-access-lcm2g") pod "8c02a32d-f97f-4598-bc6a-643565c009cf" (UID: "8c02a32d-f97f-4598-bc6a-643565c009cf"). InnerVolumeSpecName "kube-api-access-lcm2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:15:43.527873 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:43.527832 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcm2g\" (UniqueName: \"kubernetes.io/projected/8c02a32d-f97f-4598-bc6a-643565c009cf-kube-api-access-lcm2g\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:15:43.527873 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:43.527873 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c02a32d-f97f-4598-bc6a-643565c009cf-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:15:44.299476 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:44.299444 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m" Sep 11 17:15:44.299917 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:44.299479 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m" event={"ID":"8c02a32d-f97f-4598-bc6a-643565c009cf","Type":"ContainerDied","Data":"3eff406af152f7a5bf506df9fc8ad1517efd58d7911d34e192f2b10457146cdd"} Sep 11 17:15:44.299917 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:44.299557 2577 scope.go:117] "RemoveContainer" containerID="59c3706c549a37f1ded1c2a6b8761a8dfa60e2d90b2e172816679bd1de90f44e" Sep 11 17:15:44.322371 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:44.322342 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m"] Sep 11 17:15:44.327122 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:44.327099 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-891cc8-d9bc7765f-kk59m"] Sep 11 17:15:45.094980 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:45.094949 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19fc6025-6492-4cc0-9b91-e6cee0bb042e" path="/var/lib/kubelet/pods/19fc6025-6492-4cc0-9b91-e6cee0bb042e/volumes" Sep 11 17:15:45.095329 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:45.095316 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c02a32d-f97f-4598-bc6a-643565c009cf" path="/var/lib/kubelet/pods/8c02a32d-f97f-4598-bc6a-643565c009cf/volumes" Sep 11 17:15:54.300048 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:54.300013 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5"] Sep 11 17:15:54.300454 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:54.300401 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19fc6025-6492-4cc0-9b91-e6cee0bb042e" containerName="revoke-keys" Sep 11 17:15:54.300454 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:54.300413 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="19fc6025-6492-4cc0-9b91-e6cee0bb042e" containerName="revoke-keys" Sep 11 17:15:54.300454 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:54.300434 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c02a32d-f97f-4598-bc6a-643565c009cf" containerName="payload-processing" Sep 11 17:15:54.300454 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:54.300440 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c02a32d-f97f-4598-bc6a-643565c009cf" containerName="payload-processing" Sep 11 17:15:54.300625 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:54.300514 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c02a32d-f97f-4598-bc6a-643565c009cf" containerName="payload-processing" Sep 11 17:15:54.300625 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:54.300525 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="19fc6025-6492-4cc0-9b91-e6cee0bb042e" containerName="revoke-keys" Sep 11 17:15:54.304740 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:54.304720 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5" Sep 11 17:15:54.307798 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:54.307778 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 17:15:54.308038 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:54.308023 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pc4qb\"" Sep 11 17:15:54.317410 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:54.317378 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5"] Sep 11 17:15:54.433615 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:54.433580 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5\" (UID: \"fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5" Sep 11 17:15:54.534886 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:54.534849 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5\" (UID: \"fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5" Sep 11 17:15:54.535459 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:54.535440 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5\" (UID: \"fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5" Sep 11 17:15:54.616851 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:54.616768 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5" Sep 11 17:15:54.889839 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:54.889809 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5"] Sep 11 17:15:54.891296 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:15:54.891269 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8ad3e7_4ac2_4afb_b298_9d3ea1715dfa.slice/crio-f0a33a3252e14f15dfff904ce273c416c83432b328457ae1295b9a4ef5cd3325 WatchSource:0}: Error finding container f0a33a3252e14f15dfff904ce273c416c83432b328457ae1295b9a4ef5cd3325: Status 404 returned error can't find the container with id f0a33a3252e14f15dfff904ce273c416c83432b328457ae1295b9a4ef5cd3325 Sep 11 17:15:55.359657 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:55.359619 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5" event={"ID":"fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa","Type":"ContainerStarted","Data":"b608ce9e950682f48aa5236e4bc38cb5fa365989faf7bf03304650bdd6ff88c4"} Sep 11 17:15:55.360053 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:55.359708 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5" event={"ID":"fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa","Type":"ContainerStarted","Data":"f0a33a3252e14f15dfff904ce273c416c83432b328457ae1295b9a4ef5cd3325"} Sep 11 17:15:55.381151 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:55.381103 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5" podStartSLOduration=1.381088362 podStartE2EDuration="1.381088362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:15:55.379607832 +0000 UTC m=+1970.787710097" watchObservedRunningTime="2026-09-11 17:15:55.381088362 +0000 UTC m=+1970.789190630" Sep 11 17:15:56.367061 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:56.367018 2577 generic.go:356] "Generic (PLEG): container finished" podID="fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa" containerID="b608ce9e950682f48aa5236e4bc38cb5fa365989faf7bf03304650bdd6ff88c4" exitCode=0 Sep 11 17:15:56.367470 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:56.367105 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5" event={"ID":"fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa","Type":"ContainerDied","Data":"b608ce9e950682f48aa5236e4bc38cb5fa365989faf7bf03304650bdd6ff88c4"} Sep 11 17:15:57.502976 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:57.502951 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5" Sep 11 17:15:57.664141 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:57.664057 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa-maas-api-service-ca\") pod \"fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa\" (UID: \"fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa\") " Sep 11 17:15:57.664427 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:57.664397 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa" (UID: "fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:15:57.765699 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:57.765666 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa-maas-api-service-ca\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:15:58.378674 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:58.378644 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5" Sep 11 17:15:58.378843 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:58.378647 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5" event={"ID":"fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa","Type":"ContainerDied","Data":"f0a33a3252e14f15dfff904ce273c416c83432b328457ae1295b9a4ef5cd3325"} Sep 11 17:15:58.378843 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:15:58.378769 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0a33a3252e14f15dfff904ce273c416c83432b328457ae1295b9a4ef5cd3325" Sep 11 17:16:05.436312 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:05.436281 2577 scope.go:117] "RemoveContainer" containerID="344314ee4311f7a1b6e2ea05f9849051fd95a9dbd9b1109e074f9f741b233c7e" Sep 11 17:16:09.925615 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:09.925583 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-269ba8-29819115-xz28z"] Sep 11 17:16:09.928338 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:09.928317 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-269ba8-29819115-xz28z"] Sep 11 17:16:09.961783 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:09.961751 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29"] Sep 11 17:16:09.962022 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:09.961995 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29" podUID="3019d3c0-2f0b-40c8-b3ad-08afd0896cb1" containerName="payload-processing" containerID="cri-o://a1231f51aeb7c4ab7ae5a5bf4b24c35ec4784a063e0e3eb35b03aee9bafa9412" gracePeriod=30 Sep 11 17:16:09.968522 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:09.968463 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m"] Sep 11 17:16:09.969065 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:09.968715 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m" podUID="13f74896-5d21-4a91-ab55-1d4d7c5bb2e1" containerName="payload-pre-processing" containerID="cri-o://0ec729afd7a5eff5c6f36e9e714daca6235f1cd2c0d05f4d3d88bc3978db3336" gracePeriod=30 Sep 11 17:16:10.148569 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.148477 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5"] Sep 11 17:16:10.151426 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.151402 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-269ba8-5jrp5"] Sep 11 17:16:10.226628 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.226600 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29" Sep 11 17:16:10.258883 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.258853 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m" Sep 11 17:16:10.283980 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.283949 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb5jg\" (UniqueName: \"kubernetes.io/projected/3019d3c0-2f0b-40c8-b3ad-08afd0896cb1-kube-api-access-cb5jg\") pod \"3019d3c0-2f0b-40c8-b3ad-08afd0896cb1\" (UID: \"3019d3c0-2f0b-40c8-b3ad-08afd0896cb1\") " Sep 11 17:16:10.287333 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.284003 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3019d3c0-2f0b-40c8-b3ad-08afd0896cb1-plugins-config-volume\") pod \"3019d3c0-2f0b-40c8-b3ad-08afd0896cb1\" (UID: \"3019d3c0-2f0b-40c8-b3ad-08afd0896cb1\") " Sep 11 17:16:10.287333 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.284044 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f74896-5d21-4a91-ab55-1d4d7c5bb2e1-plugins-config-volume\") pod \"13f74896-5d21-4a91-ab55-1d4d7c5bb2e1\" (UID: \"13f74896-5d21-4a91-ab55-1d4d7c5bb2e1\") " Sep 11 17:16:10.287914 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.287885 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppf9m\" (UniqueName: \"kubernetes.io/projected/13f74896-5d21-4a91-ab55-1d4d7c5bb2e1-kube-api-access-ppf9m\") pod \"13f74896-5d21-4a91-ab55-1d4d7c5bb2e1\" (UID: \"13f74896-5d21-4a91-ab55-1d4d7c5bb2e1\") " Sep 11 17:16:10.288880 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.288842 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3019d3c0-2f0b-40c8-b3ad-08afd0896cb1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3019d3c0-2f0b-40c8-b3ad-08afd0896cb1" (UID: "3019d3c0-2f0b-40c8-b3ad-08afd0896cb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:16:10.289363 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.289254 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f74896-5d21-4a91-ab55-1d4d7c5bb2e1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13f74896-5d21-4a91-ab55-1d4d7c5bb2e1" (UID: "13f74896-5d21-4a91-ab55-1d4d7c5bb2e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:16:10.290962 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.290941 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3019d3c0-2f0b-40c8-b3ad-08afd0896cb1-kube-api-access-cb5jg" (OuterVolumeSpecName: "kube-api-access-cb5jg") pod "3019d3c0-2f0b-40c8-b3ad-08afd0896cb1" (UID: "3019d3c0-2f0b-40c8-b3ad-08afd0896cb1"). InnerVolumeSpecName "kube-api-access-cb5jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:16:10.291071 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.291043 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f74896-5d21-4a91-ab55-1d4d7c5bb2e1-kube-api-access-ppf9m" (OuterVolumeSpecName: "kube-api-access-ppf9m") pod "13f74896-5d21-4a91-ab55-1d4d7c5bb2e1" (UID: "13f74896-5d21-4a91-ab55-1d4d7c5bb2e1"). InnerVolumeSpecName "kube-api-access-ppf9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:16:10.389752 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.389712 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppf9m\" (UniqueName: \"kubernetes.io/projected/13f74896-5d21-4a91-ab55-1d4d7c5bb2e1-kube-api-access-ppf9m\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:16:10.389752 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.389744 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb5jg\" (UniqueName: \"kubernetes.io/projected/3019d3c0-2f0b-40c8-b3ad-08afd0896cb1-kube-api-access-cb5jg\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:16:10.389752 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.389754 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3019d3c0-2f0b-40c8-b3ad-08afd0896cb1-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:16:10.389752 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.389762 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f74896-5d21-4a91-ab55-1d4d7c5bb2e1-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:16:10.442814 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.442720 2577 generic.go:356] "Generic (PLEG): container finished" podID="13f74896-5d21-4a91-ab55-1d4d7c5bb2e1" containerID="0ec729afd7a5eff5c6f36e9e714daca6235f1cd2c0d05f4d3d88bc3978db3336" exitCode=0 Sep 11 17:16:10.442814 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.442768 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m" Sep 11 17:16:10.443073 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.442805 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m" event={"ID":"13f74896-5d21-4a91-ab55-1d4d7c5bb2e1","Type":"ContainerDied","Data":"0ec729afd7a5eff5c6f36e9e714daca6235f1cd2c0d05f4d3d88bc3978db3336"} Sep 11 17:16:10.443073 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.442862 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m" event={"ID":"13f74896-5d21-4a91-ab55-1d4d7c5bb2e1","Type":"ContainerDied","Data":"7dc4fb94715a4788dd0da64e14dedbfff9dbb1483b4434fc36441a58f9fb5e88"} Sep 11 17:16:10.443073 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.442892 2577 scope.go:117] "RemoveContainer" containerID="0ec729afd7a5eff5c6f36e9e714daca6235f1cd2c0d05f4d3d88bc3978db3336" Sep 11 17:16:10.444262 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.444243 2577 generic.go:356] "Generic (PLEG): container finished" podID="3019d3c0-2f0b-40c8-b3ad-08afd0896cb1" containerID="a1231f51aeb7c4ab7ae5a5bf4b24c35ec4784a063e0e3eb35b03aee9bafa9412" exitCode=0 Sep 11 17:16:10.444382 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.444279 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29" Sep 11 17:16:10.444382 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.444320 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29" event={"ID":"3019d3c0-2f0b-40c8-b3ad-08afd0896cb1","Type":"ContainerDied","Data":"a1231f51aeb7c4ab7ae5a5bf4b24c35ec4784a063e0e3eb35b03aee9bafa9412"} Sep 11 17:16:10.444382 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.444363 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29" event={"ID":"3019d3c0-2f0b-40c8-b3ad-08afd0896cb1","Type":"ContainerDied","Data":"a78069ea6b46f24a76b2f8cb539021e52a70850a3a5fe6008e0901ba847d80a2"} Sep 11 17:16:10.452141 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.452126 2577 scope.go:117] "RemoveContainer" containerID="0ec729afd7a5eff5c6f36e9e714daca6235f1cd2c0d05f4d3d88bc3978db3336" Sep 11 17:16:10.452414 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:16:10.452398 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec729afd7a5eff5c6f36e9e714daca6235f1cd2c0d05f4d3d88bc3978db3336\": container with ID starting with 0ec729afd7a5eff5c6f36e9e714daca6235f1cd2c0d05f4d3d88bc3978db3336 not found: ID does not exist" containerID="0ec729afd7a5eff5c6f36e9e714daca6235f1cd2c0d05f4d3d88bc3978db3336" Sep 11 17:16:10.452479 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.452422 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec729afd7a5eff5c6f36e9e714daca6235f1cd2c0d05f4d3d88bc3978db3336"} err="failed to get container status \"0ec729afd7a5eff5c6f36e9e714daca6235f1cd2c0d05f4d3d88bc3978db3336\": rpc error: code = NotFound desc = could not find container \"0ec729afd7a5eff5c6f36e9e714daca6235f1cd2c0d05f4d3d88bc3978db3336\": container with ID starting with 0ec729afd7a5eff5c6f36e9e714daca6235f1cd2c0d05f4d3d88bc3978db3336 not found: ID does not exist" Sep 11 17:16:10.452479 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.452438 2577 scope.go:117] "RemoveContainer" containerID="a1231f51aeb7c4ab7ae5a5bf4b24c35ec4784a063e0e3eb35b03aee9bafa9412" Sep 11 17:16:10.460847 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.460830 2577 scope.go:117] "RemoveContainer" containerID="a1231f51aeb7c4ab7ae5a5bf4b24c35ec4784a063e0e3eb35b03aee9bafa9412" Sep 11 17:16:10.461109 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:16:10.461090 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1231f51aeb7c4ab7ae5a5bf4b24c35ec4784a063e0e3eb35b03aee9bafa9412\": container with ID starting with a1231f51aeb7c4ab7ae5a5bf4b24c35ec4784a063e0e3eb35b03aee9bafa9412 not found: ID does not exist" containerID="a1231f51aeb7c4ab7ae5a5bf4b24c35ec4784a063e0e3eb35b03aee9bafa9412" Sep 11 17:16:10.461173 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.461121 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1231f51aeb7c4ab7ae5a5bf4b24c35ec4784a063e0e3eb35b03aee9bafa9412"} err="failed to get container status \"a1231f51aeb7c4ab7ae5a5bf4b24c35ec4784a063e0e3eb35b03aee9bafa9412\": rpc error: code = NotFound desc = could not find container \"a1231f51aeb7c4ab7ae5a5bf4b24c35ec4784a063e0e3eb35b03aee9bafa9412\": container with ID starting with a1231f51aeb7c4ab7ae5a5bf4b24c35ec4784a063e0e3eb35b03aee9bafa9412 not found: ID does not exist" Sep 11 17:16:10.470619 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.470592 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29"] Sep 11 17:16:10.477299 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.477276 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-269ba8-698c4bbcbf-76g29"] Sep 11 17:16:10.488422 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.488395 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m"] Sep 11 17:16:10.494303 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:10.494282 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-269ba8-67dfcc44c8-x2b6m"] Sep 11 17:16:11.094110 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:11.094080 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13f74896-5d21-4a91-ab55-1d4d7c5bb2e1" path="/var/lib/kubelet/pods/13f74896-5d21-4a91-ab55-1d4d7c5bb2e1/volumes" Sep 11 17:16:11.094472 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:11.094440 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3019d3c0-2f0b-40c8-b3ad-08afd0896cb1" path="/var/lib/kubelet/pods/3019d3c0-2f0b-40c8-b3ad-08afd0896cb1/volumes" Sep 11 17:16:11.094710 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:11.094697 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57ae029f-e63a-4c3c-af03-8993975e93b3" path="/var/lib/kubelet/pods/57ae029f-e63a-4c3c-af03-8993975e93b3/volumes" Sep 11 17:16:11.094951 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:11.094941 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa" path="/var/lib/kubelet/pods/fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa/volumes" Sep 11 17:16:59.712589 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:59.712552 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-750358-qfvl9"] Sep 11 17:16:59.713037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:59.712930 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa" containerName="revoke-keys" Sep 11 17:16:59.713037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:59.712942 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa" containerName="revoke-keys" Sep 11 17:16:59.713037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:59.712951 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f74896-5d21-4a91-ab55-1d4d7c5bb2e1" containerName="payload-pre-processing" Sep 11 17:16:59.713037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:59.712957 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f74896-5d21-4a91-ab55-1d4d7c5bb2e1" containerName="payload-pre-processing" Sep 11 17:16:59.713037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:59.712965 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3019d3c0-2f0b-40c8-b3ad-08afd0896cb1" containerName="payload-processing" Sep 11 17:16:59.713037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:59.712970 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3019d3c0-2f0b-40c8-b3ad-08afd0896cb1" containerName="payload-processing" Sep 11 17:16:59.713037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:59.713027 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f74896-5d21-4a91-ab55-1d4d7c5bb2e1" containerName="payload-pre-processing" Sep 11 17:16:59.713037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:59.713037 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe8ad3e7-4ac2-4afb-b298-9d3ea1715dfa" containerName="revoke-keys" Sep 11 17:16:59.713318 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:59.713046 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3019d3c0-2f0b-40c8-b3ad-08afd0896cb1" containerName="payload-processing" Sep 11 17:16:59.716119 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:59.716102 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-750358-qfvl9" Sep 11 17:16:59.718995 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:59.718975 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pc4qb\"" Sep 11 17:16:59.719312 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:59.719295 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 17:16:59.728574 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:59.728539 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-750358-qfvl9"] Sep 11 17:16:59.738225 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:59.738197 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec4dda2-c73b-4d40-8ef1-b7005c868bdd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-750358-qfvl9\" (UID: \"7ec4dda2-c73b-4d40-8ef1-b7005c868bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-750358-qfvl9" Sep 11 17:16:59.839072 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:59.839032 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec4dda2-c73b-4d40-8ef1-b7005c868bdd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-750358-qfvl9\" (UID: \"7ec4dda2-c73b-4d40-8ef1-b7005c868bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-750358-qfvl9" Sep 11 17:16:59.839719 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:16:59.839700 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec4dda2-c73b-4d40-8ef1-b7005c868bdd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-750358-qfvl9\" (UID: \"7ec4dda2-c73b-4d40-8ef1-b7005c868bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-750358-qfvl9" Sep 11 17:17:00.027148 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:00.027113 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-750358-qfvl9" Sep 11 17:17:00.308012 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:00.307981 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-750358-qfvl9"] Sep 11 17:17:00.309966 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:17:00.309925 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec4dda2_c73b_4d40_8ef1_b7005c868bdd.slice/crio-6953fee0ba35264a75ed5ef8d1f15d0108acf43db49878e8c5fc1e8c83bc7b85 WatchSource:0}: Error finding container 6953fee0ba35264a75ed5ef8d1f15d0108acf43db49878e8c5fc1e8c83bc7b85: Status 404 returned error can't find the container with id 6953fee0ba35264a75ed5ef8d1f15d0108acf43db49878e8c5fc1e8c83bc7b85 Sep 11 17:17:00.698598 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:00.698553 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-750358-qfvl9" event={"ID":"7ec4dda2-c73b-4d40-8ef1-b7005c868bdd","Type":"ContainerStarted","Data":"eb0e64e4291f40a1ca6cea06bb9d125ac868f6185a80b35c4da171058d7f9067"} Sep 11 17:17:00.698780 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:00.698618 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-750358-qfvl9" event={"ID":"7ec4dda2-c73b-4d40-8ef1-b7005c868bdd","Type":"ContainerStarted","Data":"6953fee0ba35264a75ed5ef8d1f15d0108acf43db49878e8c5fc1e8c83bc7b85"} Sep 11 17:17:01.705563 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:01.705526 2577 generic.go:356] "Generic (PLEG): container finished" podID="7ec4dda2-c73b-4d40-8ef1-b7005c868bdd" containerID="eb0e64e4291f40a1ca6cea06bb9d125ac868f6185a80b35c4da171058d7f9067" exitCode=0 Sep 11 17:17:01.705966 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:01.705560 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-750358-qfvl9" event={"ID":"7ec4dda2-c73b-4d40-8ef1-b7005c868bdd","Type":"ContainerDied","Data":"eb0e64e4291f40a1ca6cea06bb9d125ac868f6185a80b35c4da171058d7f9067"} Sep 11 17:17:02.868108 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:02.868079 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-750358-qfvl9" Sep 11 17:17:02.969587 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:02.969449 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec4dda2-c73b-4d40-8ef1-b7005c868bdd-maas-api-service-ca\") pod \"7ec4dda2-c73b-4d40-8ef1-b7005c868bdd\" (UID: \"7ec4dda2-c73b-4d40-8ef1-b7005c868bdd\") " Sep 11 17:17:02.969853 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:02.969829 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec4dda2-c73b-4d40-8ef1-b7005c868bdd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ec4dda2-c73b-4d40-8ef1-b7005c868bdd" (UID: "7ec4dda2-c73b-4d40-8ef1-b7005c868bdd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:17:03.070611 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:03.070571 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec4dda2-c73b-4d40-8ef1-b7005c868bdd-maas-api-service-ca\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:17:03.717648 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:03.717619 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-750358-qfvl9" Sep 11 17:17:03.717854 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:03.717665 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-750358-qfvl9" event={"ID":"7ec4dda2-c73b-4d40-8ef1-b7005c868bdd","Type":"ContainerDied","Data":"6953fee0ba35264a75ed5ef8d1f15d0108acf43db49878e8c5fc1e8c83bc7b85"} Sep 11 17:17:03.717854 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:03.717711 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6953fee0ba35264a75ed5ef8d1f15d0108acf43db49878e8c5fc1e8c83bc7b85" Sep 11 17:17:10.274853 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.274813 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-750358-29819115-wc9hd"] Sep 11 17:17:10.280247 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.280211 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-750358-29819115-wc9hd"] Sep 11 17:17:10.304468 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.304429 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh"] Sep 11 17:17:10.304754 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.304719 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh" podUID="38cd6edf-249e-4f12-bd0b-1e82cc33d0ed" containerName="payload-processing" containerID="cri-o://ba981c3e4056490ed18002ff13050565c8ac0afb89bc4ace268038e3c8ab7084" gracePeriod=30 Sep 11 17:17:10.565336 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.565304 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-750358-qfvl9"] Sep 11 17:17:10.568087 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.568061 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-750358-qfvl9"] Sep 11 17:17:10.571448 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.571423 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh" Sep 11 17:17:10.635646 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.635608 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlk5l\" (UniqueName: \"kubernetes.io/projected/38cd6edf-249e-4f12-bd0b-1e82cc33d0ed-kube-api-access-nlk5l\") pod \"38cd6edf-249e-4f12-bd0b-1e82cc33d0ed\" (UID: \"38cd6edf-249e-4f12-bd0b-1e82cc33d0ed\") " Sep 11 17:17:10.635981 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.635684 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38cd6edf-249e-4f12-bd0b-1e82cc33d0ed-plugins-config-volume\") pod \"38cd6edf-249e-4f12-bd0b-1e82cc33d0ed\" (UID: \"38cd6edf-249e-4f12-bd0b-1e82cc33d0ed\") " Sep 11 17:17:10.636588 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.636556 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38cd6edf-249e-4f12-bd0b-1e82cc33d0ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "38cd6edf-249e-4f12-bd0b-1e82cc33d0ed" (UID: "38cd6edf-249e-4f12-bd0b-1e82cc33d0ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:17:10.638570 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.638540 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38cd6edf-249e-4f12-bd0b-1e82cc33d0ed-kube-api-access-nlk5l" (OuterVolumeSpecName: "kube-api-access-nlk5l") pod "38cd6edf-249e-4f12-bd0b-1e82cc33d0ed" (UID: "38cd6edf-249e-4f12-bd0b-1e82cc33d0ed"). InnerVolumeSpecName "kube-api-access-nlk5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:17:10.737541 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.737475 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlk5l\" (UniqueName: \"kubernetes.io/projected/38cd6edf-249e-4f12-bd0b-1e82cc33d0ed-kube-api-access-nlk5l\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:17:10.737541 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.737540 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38cd6edf-249e-4f12-bd0b-1e82cc33d0ed-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:17:10.753480 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.753440 2577 generic.go:356] "Generic (PLEG): container finished" podID="38cd6edf-249e-4f12-bd0b-1e82cc33d0ed" containerID="ba981c3e4056490ed18002ff13050565c8ac0afb89bc4ace268038e3c8ab7084" exitCode=0 Sep 11 17:17:10.753681 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.753488 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh" Sep 11 17:17:10.753681 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.753531 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh" event={"ID":"38cd6edf-249e-4f12-bd0b-1e82cc33d0ed","Type":"ContainerDied","Data":"ba981c3e4056490ed18002ff13050565c8ac0afb89bc4ace268038e3c8ab7084"} Sep 11 17:17:10.753681 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.753608 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh" event={"ID":"38cd6edf-249e-4f12-bd0b-1e82cc33d0ed","Type":"ContainerDied","Data":"a3e33c69f1b6941d63f80ed0284dc486971442b94f5e6986c2dee804b31b4af0"} Sep 11 17:17:10.753681 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.753640 2577 scope.go:117] "RemoveContainer" containerID="ba981c3e4056490ed18002ff13050565c8ac0afb89bc4ace268038e3c8ab7084" Sep 11 17:17:10.763231 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.763204 2577 scope.go:117] "RemoveContainer" containerID="ba981c3e4056490ed18002ff13050565c8ac0afb89bc4ace268038e3c8ab7084" Sep 11 17:17:10.763527 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:17:10.763505 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba981c3e4056490ed18002ff13050565c8ac0afb89bc4ace268038e3c8ab7084\": container with ID starting with ba981c3e4056490ed18002ff13050565c8ac0afb89bc4ace268038e3c8ab7084 not found: ID does not exist" containerID="ba981c3e4056490ed18002ff13050565c8ac0afb89bc4ace268038e3c8ab7084" Sep 11 17:17:10.763596 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.763536 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba981c3e4056490ed18002ff13050565c8ac0afb89bc4ace268038e3c8ab7084"} err="failed to get container status \"ba981c3e4056490ed18002ff13050565c8ac0afb89bc4ace268038e3c8ab7084\": rpc error: code = NotFound desc = could not find container \"ba981c3e4056490ed18002ff13050565c8ac0afb89bc4ace268038e3c8ab7084\": container with ID starting with ba981c3e4056490ed18002ff13050565c8ac0afb89bc4ace268038e3c8ab7084 not found: ID does not exist" Sep 11 17:17:10.776701 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.776666 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh"] Sep 11 17:17:10.783085 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:10.783057 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-750358-7595cf8555-2nnnh"] Sep 11 17:17:11.094158 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:11.094121 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38cd6edf-249e-4f12-bd0b-1e82cc33d0ed" path="/var/lib/kubelet/pods/38cd6edf-249e-4f12-bd0b-1e82cc33d0ed/volumes" Sep 11 17:17:11.094534 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:11.094492 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ec4dda2-c73b-4d40-8ef1-b7005c868bdd" path="/var/lib/kubelet/pods/7ec4dda2-c73b-4d40-8ef1-b7005c868bdd/volumes" Sep 11 17:17:11.094805 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:11.094793 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4f7a9d8-3262-43f3-8244-1fd0dffee7e6" path="/var/lib/kubelet/pods/f4f7a9d8-3262-43f3-8244-1fd0dffee7e6/volumes" Sep 11 17:17:29.486408 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.486248 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m"] Sep 11 17:17:29.486975 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.486482 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m" podUID="e7356d5c-3dd3-4a49-96f6-675eb840fbfe" containerName="payload-processing" containerID="cri-o://f157c6d41193ed09a956270ae850c2993c945ebd9ce79da6453426af5d8abe5b" gracePeriod=30 Sep 11 17:17:29.497004 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.496881 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5"] Sep 11 17:17:29.497175 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.497121 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5" podUID="eb27089f-c888-4453-8bd4-591406ef170e" containerName="payload-pre-processing" containerID="cri-o://661ea18016f0b27c46b500cb909485f7be3ef67f96ada47b07c7dd61d2b9c3e3" gracePeriod=30 Sep 11 17:17:29.510301 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.510070 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m" podUID="e7356d5c-3dd3-4a49-96f6-675eb840fbfe" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.67:9004: connect: connection refused" Sep 11 17:17:29.510301 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.510271 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5" podUID="eb27089f-c888-4453-8bd4-591406ef170e" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.66:9004: connect: connection refused" Sep 11 17:17:29.771747 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.771722 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5" Sep 11 17:17:29.774396 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.774375 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m" Sep 11 17:17:29.855059 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.855024 2577 generic.go:356] "Generic (PLEG): container finished" podID="eb27089f-c888-4453-8bd4-591406ef170e" containerID="661ea18016f0b27c46b500cb909485f7be3ef67f96ada47b07c7dd61d2b9c3e3" exitCode=0 Sep 11 17:17:29.855231 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.855071 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5" Sep 11 17:17:29.855231 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.855108 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5" event={"ID":"eb27089f-c888-4453-8bd4-591406ef170e","Type":"ContainerDied","Data":"661ea18016f0b27c46b500cb909485f7be3ef67f96ada47b07c7dd61d2b9c3e3"} Sep 11 17:17:29.855231 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.855163 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5" event={"ID":"eb27089f-c888-4453-8bd4-591406ef170e","Type":"ContainerDied","Data":"285bc629482d8e8eb9102a2e75e8cba87581f55528b5202c986afe7296794770"} Sep 11 17:17:29.855231 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.855191 2577 scope.go:117] "RemoveContainer" containerID="661ea18016f0b27c46b500cb909485f7be3ef67f96ada47b07c7dd61d2b9c3e3" Sep 11 17:17:29.856568 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.856546 2577 generic.go:356] "Generic (PLEG): container finished" podID="e7356d5c-3dd3-4a49-96f6-675eb840fbfe" containerID="f157c6d41193ed09a956270ae850c2993c945ebd9ce79da6453426af5d8abe5b" exitCode=0 Sep 11 17:17:29.856689 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.856597 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m" Sep 11 17:17:29.856689 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.856632 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m" event={"ID":"e7356d5c-3dd3-4a49-96f6-675eb840fbfe","Type":"ContainerDied","Data":"f157c6d41193ed09a956270ae850c2993c945ebd9ce79da6453426af5d8abe5b"} Sep 11 17:17:29.856799 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.856688 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m" event={"ID":"e7356d5c-3dd3-4a49-96f6-675eb840fbfe","Type":"ContainerDied","Data":"cd8b8233743cb5e4e7b269880e033d9508580e9e00db011609e489334ed9170e"} Sep 11 17:17:29.863839 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.863822 2577 scope.go:117] "RemoveContainer" containerID="661ea18016f0b27c46b500cb909485f7be3ef67f96ada47b07c7dd61d2b9c3e3" Sep 11 17:17:29.864116 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:17:29.864099 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661ea18016f0b27c46b500cb909485f7be3ef67f96ada47b07c7dd61d2b9c3e3\": container with ID starting with 661ea18016f0b27c46b500cb909485f7be3ef67f96ada47b07c7dd61d2b9c3e3 not found: ID does not exist" containerID="661ea18016f0b27c46b500cb909485f7be3ef67f96ada47b07c7dd61d2b9c3e3" Sep 11 17:17:29.864158 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.864124 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661ea18016f0b27c46b500cb909485f7be3ef67f96ada47b07c7dd61d2b9c3e3"} err="failed to get container status \"661ea18016f0b27c46b500cb909485f7be3ef67f96ada47b07c7dd61d2b9c3e3\": rpc error: code = NotFound desc = could not find container \"661ea18016f0b27c46b500cb909485f7be3ef67f96ada47b07c7dd61d2b9c3e3\": container with ID starting with 661ea18016f0b27c46b500cb909485f7be3ef67f96ada47b07c7dd61d2b9c3e3 not found: ID does not exist" Sep 11 17:17:29.864158 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.864138 2577 scope.go:117] "RemoveContainer" containerID="f157c6d41193ed09a956270ae850c2993c945ebd9ce79da6453426af5d8abe5b" Sep 11 17:17:29.871963 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.871947 2577 scope.go:117] "RemoveContainer" containerID="f157c6d41193ed09a956270ae850c2993c945ebd9ce79da6453426af5d8abe5b" Sep 11 17:17:29.872248 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:17:29.872228 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f157c6d41193ed09a956270ae850c2993c945ebd9ce79da6453426af5d8abe5b\": container with ID starting with f157c6d41193ed09a956270ae850c2993c945ebd9ce79da6453426af5d8abe5b not found: ID does not exist" containerID="f157c6d41193ed09a956270ae850c2993c945ebd9ce79da6453426af5d8abe5b" Sep 11 17:17:29.872296 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.872255 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f157c6d41193ed09a956270ae850c2993c945ebd9ce79da6453426af5d8abe5b"} err="failed to get container status \"f157c6d41193ed09a956270ae850c2993c945ebd9ce79da6453426af5d8abe5b\": rpc error: code = NotFound desc = could not find container \"f157c6d41193ed09a956270ae850c2993c945ebd9ce79da6453426af5d8abe5b\": container with ID starting with f157c6d41193ed09a956270ae850c2993c945ebd9ce79da6453426af5d8abe5b not found: ID does not exist" Sep 11 17:17:29.913429 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.913400 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7356d5c-3dd3-4a49-96f6-675eb840fbfe-plugins-config-volume\") pod \"e7356d5c-3dd3-4a49-96f6-675eb840fbfe\" (UID: \"e7356d5c-3dd3-4a49-96f6-675eb840fbfe\") " Sep 11 17:17:29.913608 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.913442 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jh2x\" (UniqueName: \"kubernetes.io/projected/eb27089f-c888-4453-8bd4-591406ef170e-kube-api-access-5jh2x\") pod \"eb27089f-c888-4453-8bd4-591406ef170e\" (UID: \"eb27089f-c888-4453-8bd4-591406ef170e\") " Sep 11 17:17:29.913608 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.913461 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb27089f-c888-4453-8bd4-591406ef170e-plugins-config-volume\") pod \"eb27089f-c888-4453-8bd4-591406ef170e\" (UID: \"eb27089f-c888-4453-8bd4-591406ef170e\") " Sep 11 17:17:29.913608 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.913518 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmmgl\" (UniqueName: \"kubernetes.io/projected/e7356d5c-3dd3-4a49-96f6-675eb840fbfe-kube-api-access-hmmgl\") pod \"e7356d5c-3dd3-4a49-96f6-675eb840fbfe\" (UID: \"e7356d5c-3dd3-4a49-96f6-675eb840fbfe\") " Sep 11 17:17:29.913871 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.913848 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb27089f-c888-4453-8bd4-591406ef170e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb27089f-c888-4453-8bd4-591406ef170e" (UID: "eb27089f-c888-4453-8bd4-591406ef170e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:17:29.913936 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.913863 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7356d5c-3dd3-4a49-96f6-675eb840fbfe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e7356d5c-3dd3-4a49-96f6-675eb840fbfe" (UID: "e7356d5c-3dd3-4a49-96f6-675eb840fbfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:17:29.915591 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.915562 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb27089f-c888-4453-8bd4-591406ef170e-kube-api-access-5jh2x" (OuterVolumeSpecName: "kube-api-access-5jh2x") pod "eb27089f-c888-4453-8bd4-591406ef170e" (UID: "eb27089f-c888-4453-8bd4-591406ef170e"). InnerVolumeSpecName "kube-api-access-5jh2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:17:29.915675 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:29.915608 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7356d5c-3dd3-4a49-96f6-675eb840fbfe-kube-api-access-hmmgl" (OuterVolumeSpecName: "kube-api-access-hmmgl") pod "e7356d5c-3dd3-4a49-96f6-675eb840fbfe" (UID: "e7356d5c-3dd3-4a49-96f6-675eb840fbfe"). InnerVolumeSpecName "kube-api-access-hmmgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:17:30.014941 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:30.014851 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7356d5c-3dd3-4a49-96f6-675eb840fbfe-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:17:30.014941 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:30.014881 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jh2x\" (UniqueName: \"kubernetes.io/projected/eb27089f-c888-4453-8bd4-591406ef170e-kube-api-access-5jh2x\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:17:30.014941 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:30.014891 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb27089f-c888-4453-8bd4-591406ef170e-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:17:30.014941 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:30.014899 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmmgl\" (UniqueName: \"kubernetes.io/projected/e7356d5c-3dd3-4a49-96f6-675eb840fbfe-kube-api-access-hmmgl\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:17:30.177803 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:30.177772 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5"] Sep 11 17:17:30.182092 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:30.182065 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2fae12-dcbcf697d-c2qt5"] Sep 11 17:17:30.200566 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:30.200529 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m"] Sep 11 17:17:30.202128 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:30.202102 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2fae12-6645c8484f-bzd8m"] Sep 11 17:17:31.093673 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:31.093638 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7356d5c-3dd3-4a49-96f6-675eb840fbfe" path="/var/lib/kubelet/pods/e7356d5c-3dd3-4a49-96f6-675eb840fbfe/volumes" Sep 11 17:17:31.094047 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:31.094010 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb27089f-c888-4453-8bd4-591406ef170e" path="/var/lib/kubelet/pods/eb27089f-c888-4453-8bd4-591406ef170e/volumes" Sep 11 17:17:47.622967 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.622926 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-327a3a-29819115-hhhkq"] Sep 11 17:17:47.625011 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.624986 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-327a3a-29819115-hhhkq"] Sep 11 17:17:47.650947 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.650911 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d"] Sep 11 17:17:47.651193 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.651163 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d" podUID="199e0b37-0bca-40f7-ac6a-31370fe0ec47" containerName="payload-processing" containerID="cri-o://3b42e4844e3f4012a338f2bbfc0a4fca4bdfbed9dfd8c1c85464937ca89701f3" gracePeriod=30 Sep 11 17:17:47.661318 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.660875 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj"] Sep 11 17:17:47.661467 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.661436 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj" podUID="2c19eeac-7303-41ce-a036-f426073c85a7" containerName="payload-pre-processing" containerID="cri-o://f3f50f2c16b4070a67d22d9bda8d62058fbdfc40b04ea7feb35b3715f8d66361" gracePeriod=30 Sep 11 17:17:47.919144 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.919113 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d" Sep 11 17:17:47.948177 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.948153 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj" Sep 11 17:17:47.952362 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.952330 2577 generic.go:356] "Generic (PLEG): container finished" podID="199e0b37-0bca-40f7-ac6a-31370fe0ec47" containerID="3b42e4844e3f4012a338f2bbfc0a4fca4bdfbed9dfd8c1c85464937ca89701f3" exitCode=0 Sep 11 17:17:47.952487 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.952370 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d" Sep 11 17:17:47.952487 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.952405 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d" event={"ID":"199e0b37-0bca-40f7-ac6a-31370fe0ec47","Type":"ContainerDied","Data":"3b42e4844e3f4012a338f2bbfc0a4fca4bdfbed9dfd8c1c85464937ca89701f3"} Sep 11 17:17:47.952487 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.952459 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d" event={"ID":"199e0b37-0bca-40f7-ac6a-31370fe0ec47","Type":"ContainerDied","Data":"24935a95c4d93460cf807af5888ac9f097a12920ff0273977de8b5b5d53bf9d6"} Sep 11 17:17:47.952487 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.952483 2577 scope.go:117] "RemoveContainer" containerID="3b42e4844e3f4012a338f2bbfc0a4fca4bdfbed9dfd8c1c85464937ca89701f3" Sep 11 17:17:47.953942 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.953919 2577 generic.go:356] "Generic (PLEG): container finished" podID="2c19eeac-7303-41ce-a036-f426073c85a7" containerID="f3f50f2c16b4070a67d22d9bda8d62058fbdfc40b04ea7feb35b3715f8d66361" exitCode=0 Sep 11 17:17:47.954044 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.953950 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj" Sep 11 17:17:47.954044 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.954004 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj" event={"ID":"2c19eeac-7303-41ce-a036-f426073c85a7","Type":"ContainerDied","Data":"f3f50f2c16b4070a67d22d9bda8d62058fbdfc40b04ea7feb35b3715f8d66361"} Sep 11 17:17:47.954145 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.954048 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj" event={"ID":"2c19eeac-7303-41ce-a036-f426073c85a7","Type":"ContainerDied","Data":"a2f37e948d34e13e0cfdc3059fd14d00a4b365db39005b07fd3f20fa452e8467"} Sep 11 17:17:47.962697 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.962682 2577 scope.go:117] "RemoveContainer" containerID="3b42e4844e3f4012a338f2bbfc0a4fca4bdfbed9dfd8c1c85464937ca89701f3" Sep 11 17:17:47.962995 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:17:47.962972 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b42e4844e3f4012a338f2bbfc0a4fca4bdfbed9dfd8c1c85464937ca89701f3\": container with ID starting with 3b42e4844e3f4012a338f2bbfc0a4fca4bdfbed9dfd8c1c85464937ca89701f3 not found: ID does not exist" containerID="3b42e4844e3f4012a338f2bbfc0a4fca4bdfbed9dfd8c1c85464937ca89701f3" Sep 11 17:17:47.963105 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.963001 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b42e4844e3f4012a338f2bbfc0a4fca4bdfbed9dfd8c1c85464937ca89701f3"} err="failed to get container status \"3b42e4844e3f4012a338f2bbfc0a4fca4bdfbed9dfd8c1c85464937ca89701f3\": rpc error: code = NotFound desc = could not find container \"3b42e4844e3f4012a338f2bbfc0a4fca4bdfbed9dfd8c1c85464937ca89701f3\": container with ID starting with 3b42e4844e3f4012a338f2bbfc0a4fca4bdfbed9dfd8c1c85464937ca89701f3 not found: ID does not exist" Sep 11 17:17:47.963105 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.963024 2577 scope.go:117] "RemoveContainer" containerID="f3f50f2c16b4070a67d22d9bda8d62058fbdfc40b04ea7feb35b3715f8d66361" Sep 11 17:17:47.974078 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.974048 2577 scope.go:117] "RemoveContainer" containerID="f3f50f2c16b4070a67d22d9bda8d62058fbdfc40b04ea7feb35b3715f8d66361" Sep 11 17:17:47.974598 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:17:47.974473 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f50f2c16b4070a67d22d9bda8d62058fbdfc40b04ea7feb35b3715f8d66361\": container with ID starting with f3f50f2c16b4070a67d22d9bda8d62058fbdfc40b04ea7feb35b3715f8d66361 not found: ID does not exist" containerID="f3f50f2c16b4070a67d22d9bda8d62058fbdfc40b04ea7feb35b3715f8d66361" Sep 11 17:17:47.974701 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.974609 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f50f2c16b4070a67d22d9bda8d62058fbdfc40b04ea7feb35b3715f8d66361"} err="failed to get container status \"f3f50f2c16b4070a67d22d9bda8d62058fbdfc40b04ea7feb35b3715f8d66361\": rpc error: code = NotFound desc = could not find container \"f3f50f2c16b4070a67d22d9bda8d62058fbdfc40b04ea7feb35b3715f8d66361\": container with ID starting with f3f50f2c16b4070a67d22d9bda8d62058fbdfc40b04ea7feb35b3715f8d66361 not found: ID does not exist" Sep 11 17:17:47.978681 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.978642 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c19eeac-7303-41ce-a036-f426073c85a7-plugins-config-volume\") pod \"2c19eeac-7303-41ce-a036-f426073c85a7\" (UID: \"2c19eeac-7303-41ce-a036-f426073c85a7\") " Sep 11 17:17:47.978793 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.978695 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199e0b37-0bca-40f7-ac6a-31370fe0ec47-plugins-config-volume\") pod \"199e0b37-0bca-40f7-ac6a-31370fe0ec47\" (UID: \"199e0b37-0bca-40f7-ac6a-31370fe0ec47\") " Sep 11 17:17:47.978793 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.978765 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfzkr\" (UniqueName: \"kubernetes.io/projected/2c19eeac-7303-41ce-a036-f426073c85a7-kube-api-access-nfzkr\") pod \"2c19eeac-7303-41ce-a036-f426073c85a7\" (UID: \"2c19eeac-7303-41ce-a036-f426073c85a7\") " Sep 11 17:17:47.978972 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.978808 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bkc6\" (UniqueName: \"kubernetes.io/projected/199e0b37-0bca-40f7-ac6a-31370fe0ec47-kube-api-access-7bkc6\") pod \"199e0b37-0bca-40f7-ac6a-31370fe0ec47\" (UID: \"199e0b37-0bca-40f7-ac6a-31370fe0ec47\") " Sep 11 17:17:47.979446 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.979422 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/199e0b37-0bca-40f7-ac6a-31370fe0ec47-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "199e0b37-0bca-40f7-ac6a-31370fe0ec47" (UID: "199e0b37-0bca-40f7-ac6a-31370fe0ec47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:17:47.981132 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.980100 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c19eeac-7303-41ce-a036-f426073c85a7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c19eeac-7303-41ce-a036-f426073c85a7" (UID: "2c19eeac-7303-41ce-a036-f426073c85a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:17:47.981522 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.981476 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199e0b37-0bca-40f7-ac6a-31370fe0ec47-kube-api-access-7bkc6" (OuterVolumeSpecName: "kube-api-access-7bkc6") pod "199e0b37-0bca-40f7-ac6a-31370fe0ec47" (UID: "199e0b37-0bca-40f7-ac6a-31370fe0ec47"). InnerVolumeSpecName "kube-api-access-7bkc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:17:47.981621 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:47.981534 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c19eeac-7303-41ce-a036-f426073c85a7-kube-api-access-nfzkr" (OuterVolumeSpecName: "kube-api-access-nfzkr") pod "2c19eeac-7303-41ce-a036-f426073c85a7" (UID: "2c19eeac-7303-41ce-a036-f426073c85a7"). InnerVolumeSpecName "kube-api-access-nfzkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:17:48.080224 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:48.080182 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfzkr\" (UniqueName: \"kubernetes.io/projected/2c19eeac-7303-41ce-a036-f426073c85a7-kube-api-access-nfzkr\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:17:48.080224 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:48.080221 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bkc6\" (UniqueName: \"kubernetes.io/projected/199e0b37-0bca-40f7-ac6a-31370fe0ec47-kube-api-access-7bkc6\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:17:48.080441 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:48.080234 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c19eeac-7303-41ce-a036-f426073c85a7-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:17:48.080441 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:48.080246 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199e0b37-0bca-40f7-ac6a-31370fe0ec47-plugins-config-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:17:48.302534 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:48.302486 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d"] Sep 11 17:17:48.313983 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:48.313952 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-327a3a-798b88466c-7tq4d"] Sep 11 17:17:48.333913 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:48.333883 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj"] Sep 11 17:17:48.338583 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:48.338555 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-327a3a-758897fc94-6ndmj"] Sep 11 17:17:49.093582 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:49.093548 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="199e0b37-0bca-40f7-ac6a-31370fe0ec47" path="/var/lib/kubelet/pods/199e0b37-0bca-40f7-ac6a-31370fe0ec47/volumes" Sep 11 17:17:49.093960 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:49.093907 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c19eeac-7303-41ce-a036-f426073c85a7" path="/var/lib/kubelet/pods/2c19eeac-7303-41ce-a036-f426073c85a7/volumes" Sep 11 17:17:49.094152 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:17:49.094139 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcde8e3e-b010-44ea-816b-9764c44bdcab" path="/var/lib/kubelet/pods/bcde8e3e-b010-44ea-816b-9764c44bdcab/volumes" Sep 11 17:18:05.169219 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:05.169188 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 17:18:05.174526 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:05.174487 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 17:18:59.836704 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.836475 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-76848564f6-dzfsn"] Sep 11 17:18:59.838007 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.837979 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ec4dda2-c73b-4d40-8ef1-b7005c868bdd" containerName="revoke-keys" Sep 11 17:18:59.838178 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.838167 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ec4dda2-c73b-4d40-8ef1-b7005c868bdd" containerName="revoke-keys" Sep 11 17:18:59.838266 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.838256 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7356d5c-3dd3-4a49-96f6-675eb840fbfe" containerName="payload-processing" Sep 11 17:18:59.838353 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.838345 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7356d5c-3dd3-4a49-96f6-675eb840fbfe" containerName="payload-processing" Sep 11 17:18:59.838696 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.838679 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199e0b37-0bca-40f7-ac6a-31370fe0ec47" containerName="payload-processing" Sep 11 17:18:59.838812 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.838802 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="199e0b37-0bca-40f7-ac6a-31370fe0ec47" containerName="payload-processing" Sep 11 17:18:59.838901 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.838891 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c19eeac-7303-41ce-a036-f426073c85a7" containerName="payload-pre-processing" Sep 11 17:18:59.838972 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.838964 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c19eeac-7303-41ce-a036-f426073c85a7" containerName="payload-pre-processing" Sep 11 17:18:59.839061 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.839052 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38cd6edf-249e-4f12-bd0b-1e82cc33d0ed" containerName="payload-processing" Sep 11 17:18:59.839136 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.839128 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="38cd6edf-249e-4f12-bd0b-1e82cc33d0ed" containerName="payload-processing" Sep 11 17:18:59.839248 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.839239 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb27089f-c888-4453-8bd4-591406ef170e" containerName="payload-pre-processing" Sep 11 17:18:59.839319 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.839311 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb27089f-c888-4453-8bd4-591406ef170e" containerName="payload-pre-processing" Sep 11 17:18:59.839585 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.839570 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c19eeac-7303-41ce-a036-f426073c85a7" containerName="payload-pre-processing" Sep 11 17:18:59.839735 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.839723 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb27089f-c888-4453-8bd4-591406ef170e" containerName="payload-pre-processing" Sep 11 17:18:59.839816 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.839806 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7356d5c-3dd3-4a49-96f6-675eb840fbfe" containerName="payload-processing" Sep 11 17:18:59.839887 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.839878 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="38cd6edf-249e-4f12-bd0b-1e82cc33d0ed" containerName="payload-processing" Sep 11 17:18:59.840020 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.840011 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="199e0b37-0bca-40f7-ac6a-31370fe0ec47" containerName="payload-processing" Sep 11 17:18:59.840089 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.840081 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ec4dda2-c73b-4d40-8ef1-b7005c868bdd" containerName="revoke-keys" Sep 11 17:18:59.844150 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.844129 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76848564f6-dzfsn" Sep 11 17:18:59.847458 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.847437 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 11 17:18:59.848857 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.848842 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-67lb8\"" Sep 11 17:18:59.848927 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.848846 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 11 17:18:59.849016 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.848998 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 11 17:18:59.855938 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.855913 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76848564f6-dzfsn"] Sep 11 17:18:59.926704 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.926665 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9cqt\" (UniqueName: \"kubernetes.io/projected/98b40838-a556-4d5d-ba8a-5e1b6cc63e24-kube-api-access-h9cqt\") pod \"maas-controller-76848564f6-dzfsn\" (UID: \"98b40838-a556-4d5d-ba8a-5e1b6cc63e24\") " pod="opendatahub/maas-controller-76848564f6-dzfsn" Sep 11 17:18:59.926887 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.926712 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98b40838-a556-4d5d-ba8a-5e1b6cc63e24-metrics-certs\") pod \"maas-controller-76848564f6-dzfsn\" (UID: \"98b40838-a556-4d5d-ba8a-5e1b6cc63e24\") " pod="opendatahub/maas-controller-76848564f6-dzfsn" Sep 11 17:18:59.926887 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:18:59.926861 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98b40838-a556-4d5d-ba8a-5e1b6cc63e24-webhook-cert\") pod \"maas-controller-76848564f6-dzfsn\" (UID: \"98b40838-a556-4d5d-ba8a-5e1b6cc63e24\") " pod="opendatahub/maas-controller-76848564f6-dzfsn" Sep 11 17:19:00.027601 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:00.027566 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98b40838-a556-4d5d-ba8a-5e1b6cc63e24-webhook-cert\") pod \"maas-controller-76848564f6-dzfsn\" (UID: \"98b40838-a556-4d5d-ba8a-5e1b6cc63e24\") " pod="opendatahub/maas-controller-76848564f6-dzfsn" Sep 11 17:19:00.027781 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:00.027610 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9cqt\" (UniqueName: \"kubernetes.io/projected/98b40838-a556-4d5d-ba8a-5e1b6cc63e24-kube-api-access-h9cqt\") pod \"maas-controller-76848564f6-dzfsn\" (UID: \"98b40838-a556-4d5d-ba8a-5e1b6cc63e24\") " pod="opendatahub/maas-controller-76848564f6-dzfsn" Sep 11 17:19:00.027781 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:00.027638 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98b40838-a556-4d5d-ba8a-5e1b6cc63e24-metrics-certs\") pod \"maas-controller-76848564f6-dzfsn\" (UID: \"98b40838-a556-4d5d-ba8a-5e1b6cc63e24\") " pod="opendatahub/maas-controller-76848564f6-dzfsn" Sep 11 17:19:00.030347 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:00.030314 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98b40838-a556-4d5d-ba8a-5e1b6cc63e24-metrics-certs\") pod \"maas-controller-76848564f6-dzfsn\" (UID: \"98b40838-a556-4d5d-ba8a-5e1b6cc63e24\") " pod="opendatahub/maas-controller-76848564f6-dzfsn" Sep 11 17:19:00.030466 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:00.030314 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98b40838-a556-4d5d-ba8a-5e1b6cc63e24-webhook-cert\") pod \"maas-controller-76848564f6-dzfsn\" (UID: \"98b40838-a556-4d5d-ba8a-5e1b6cc63e24\") " pod="opendatahub/maas-controller-76848564f6-dzfsn" Sep 11 17:19:00.036204 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:00.036176 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9cqt\" (UniqueName: \"kubernetes.io/projected/98b40838-a556-4d5d-ba8a-5e1b6cc63e24-kube-api-access-h9cqt\") pod \"maas-controller-76848564f6-dzfsn\" (UID: \"98b40838-a556-4d5d-ba8a-5e1b6cc63e24\") " pod="opendatahub/maas-controller-76848564f6-dzfsn" Sep 11 17:19:00.154958 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:00.154870 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76848564f6-dzfsn" Sep 11 17:19:00.431980 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:00.431954 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76848564f6-dzfsn"] Sep 11 17:19:00.433881 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:19:00.433856 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b40838_a556_4d5d_ba8a_5e1b6cc63e24.slice/crio-ea9942afba732c54390fc772e5ea67b214b4f988f76135c4b2753a51d66afd30 WatchSource:0}: Error finding container ea9942afba732c54390fc772e5ea67b214b4f988f76135c4b2753a51d66afd30: Status 404 returned error can't find the container with id ea9942afba732c54390fc772e5ea67b214b4f988f76135c4b2753a51d66afd30 Sep 11 17:19:00.435247 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:00.435225 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 17:19:01.313298 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:01.313260 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76848564f6-dzfsn" event={"ID":"98b40838-a556-4d5d-ba8a-5e1b6cc63e24","Type":"ContainerStarted","Data":"5d64ceefd987764de5a157ca74ba465ff6ab57f6819babc83dd3914cb48f72de"} Sep 11 17:19:01.313711 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:01.313316 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76848564f6-dzfsn" event={"ID":"98b40838-a556-4d5d-ba8a-5e1b6cc63e24","Type":"ContainerStarted","Data":"ea9942afba732c54390fc772e5ea67b214b4f988f76135c4b2753a51d66afd30"} Sep 11 17:19:01.313711 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:01.313352 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-76848564f6-dzfsn" Sep 11 17:19:01.342749 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:01.342698 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-76848564f6-dzfsn" podStartSLOduration=1.849643018 podStartE2EDuration="2.342682197s" totalImagesPullingTime="493.039179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:18:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:01.340434239 +0000 UTC m=+2156.748536528" watchObservedRunningTime="2026-09-11 17:19:01.342682197 +0000 UTC m=+2156.750784466" Sep 11 17:19:12.322444 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:12.322415 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-76848564f6-dzfsn" Sep 11 17:19:21.668073 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:21.668026 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76848564f6-dzfsn"] Sep 11 17:19:21.669037 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:21.668251 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-76848564f6-dzfsn" podUID="98b40838-a556-4d5d-ba8a-5e1b6cc63e24" containerName="manager" containerID="cri-o://5d64ceefd987764de5a157ca74ba465ff6ab57f6819babc83dd3914cb48f72de" gracePeriod=30 Sep 11 17:19:21.918483 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:21.918419 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76848564f6-dzfsn" Sep 11 17:19:21.924543 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:21.924517 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98b40838-a556-4d5d-ba8a-5e1b6cc63e24-webhook-cert\") pod \"98b40838-a556-4d5d-ba8a-5e1b6cc63e24\" (UID: \"98b40838-a556-4d5d-ba8a-5e1b6cc63e24\") " Sep 11 17:19:21.924683 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:21.924577 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9cqt\" (UniqueName: \"kubernetes.io/projected/98b40838-a556-4d5d-ba8a-5e1b6cc63e24-kube-api-access-h9cqt\") pod \"98b40838-a556-4d5d-ba8a-5e1b6cc63e24\" (UID: \"98b40838-a556-4d5d-ba8a-5e1b6cc63e24\") " Sep 11 17:19:21.924683 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:21.924630 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98b40838-a556-4d5d-ba8a-5e1b6cc63e24-metrics-certs\") pod \"98b40838-a556-4d5d-ba8a-5e1b6cc63e24\" (UID: \"98b40838-a556-4d5d-ba8a-5e1b6cc63e24\") " Sep 11 17:19:21.926541 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:21.926509 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b40838-a556-4d5d-ba8a-5e1b6cc63e24-kube-api-access-h9cqt" (OuterVolumeSpecName: "kube-api-access-h9cqt") pod "98b40838-a556-4d5d-ba8a-5e1b6cc63e24" (UID: "98b40838-a556-4d5d-ba8a-5e1b6cc63e24"). InnerVolumeSpecName "kube-api-access-h9cqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:19:21.926664 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:21.926619 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b40838-a556-4d5d-ba8a-5e1b6cc63e24-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "98b40838-a556-4d5d-ba8a-5e1b6cc63e24" (UID: "98b40838-a556-4d5d-ba8a-5e1b6cc63e24"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 17:19:21.926734 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:21.926674 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b40838-a556-4d5d-ba8a-5e1b6cc63e24-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "98b40838-a556-4d5d-ba8a-5e1b6cc63e24" (UID: "98b40838-a556-4d5d-ba8a-5e1b6cc63e24"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 17:19:22.025704 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:22.025669 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98b40838-a556-4d5d-ba8a-5e1b6cc63e24-webhook-cert\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:19:22.025704 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:22.025701 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9cqt\" (UniqueName: \"kubernetes.io/projected/98b40838-a556-4d5d-ba8a-5e1b6cc63e24-kube-api-access-h9cqt\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:19:22.025704 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:22.025711 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98b40838-a556-4d5d-ba8a-5e1b6cc63e24-metrics-certs\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:19:22.425065 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:22.425029 2577 generic.go:356] "Generic (PLEG): container finished" podID="98b40838-a556-4d5d-ba8a-5e1b6cc63e24" containerID="5d64ceefd987764de5a157ca74ba465ff6ab57f6819babc83dd3914cb48f72de" exitCode=0 Sep 11 17:19:22.425276 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:22.425073 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76848564f6-dzfsn" event={"ID":"98b40838-a556-4d5d-ba8a-5e1b6cc63e24","Type":"ContainerDied","Data":"5d64ceefd987764de5a157ca74ba465ff6ab57f6819babc83dd3914cb48f72de"} Sep 11 17:19:22.425276 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:22.425082 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76848564f6-dzfsn" Sep 11 17:19:22.425276 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:22.425125 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76848564f6-dzfsn" event={"ID":"98b40838-a556-4d5d-ba8a-5e1b6cc63e24","Type":"ContainerDied","Data":"ea9942afba732c54390fc772e5ea67b214b4f988f76135c4b2753a51d66afd30"} Sep 11 17:19:22.425276 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:22.425153 2577 scope.go:117] "RemoveContainer" containerID="5d64ceefd987764de5a157ca74ba465ff6ab57f6819babc83dd3914cb48f72de" Sep 11 17:19:22.442374 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:22.442349 2577 scope.go:117] "RemoveContainer" containerID="5d64ceefd987764de5a157ca74ba465ff6ab57f6819babc83dd3914cb48f72de" Sep 11 17:19:22.442756 ip-10-0-143-165 kubenswrapper[2577]: E0911 17:19:22.442733 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d64ceefd987764de5a157ca74ba465ff6ab57f6819babc83dd3914cb48f72de\": container with ID starting with 5d64ceefd987764de5a157ca74ba465ff6ab57f6819babc83dd3914cb48f72de not found: ID does not exist" containerID="5d64ceefd987764de5a157ca74ba465ff6ab57f6819babc83dd3914cb48f72de" Sep 11 17:19:22.442816 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:22.442765 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d64ceefd987764de5a157ca74ba465ff6ab57f6819babc83dd3914cb48f72de"} err="failed to get container status \"5d64ceefd987764de5a157ca74ba465ff6ab57f6819babc83dd3914cb48f72de\": rpc error: code = NotFound desc = could not find container \"5d64ceefd987764de5a157ca74ba465ff6ab57f6819babc83dd3914cb48f72de\": container with ID starting with 5d64ceefd987764de5a157ca74ba465ff6ab57f6819babc83dd3914cb48f72de not found: ID does not exist" Sep 11 17:19:22.473958 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:22.473932 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76848564f6-dzfsn"] Sep 11 17:19:22.480335 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:22.480311 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-76848564f6-dzfsn"] Sep 11 17:19:23.093378 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.093347 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98b40838-a556-4d5d-ba8a-5e1b6cc63e24" path="/var/lib/kubelet/pods/98b40838-a556-4d5d-ba8a-5e1b6cc63e24/volumes" Sep 11 17:19:23.293901 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.293865 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-76848564f6-dkpcd"] Sep 11 17:19:23.294215 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.294203 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98b40838-a556-4d5d-ba8a-5e1b6cc63e24" containerName="manager" Sep 11 17:19:23.294264 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.294217 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="98b40838-a556-4d5d-ba8a-5e1b6cc63e24" containerName="manager" Sep 11 17:19:23.294299 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.294279 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="98b40838-a556-4d5d-ba8a-5e1b6cc63e24" containerName="manager" Sep 11 17:19:23.298568 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.298547 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76848564f6-dkpcd" Sep 11 17:19:23.305061 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.304861 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 11 17:19:23.305061 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.304899 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-67lb8\"" Sep 11 17:19:23.305725 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.305709 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 11 17:19:23.305810 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.305773 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 11 17:19:23.312151 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.312129 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76848564f6-dkpcd"] Sep 11 17:19:23.336711 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.336683 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpfl9\" (UniqueName: \"kubernetes.io/projected/3cb61b31-96d0-43a2-85d3-56b86209ae9e-kube-api-access-lpfl9\") pod \"maas-controller-76848564f6-dkpcd\" (UID: \"3cb61b31-96d0-43a2-85d3-56b86209ae9e\") " pod="opendatahub/maas-controller-76848564f6-dkpcd" Sep 11 17:19:23.336870 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.336752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3cb61b31-96d0-43a2-85d3-56b86209ae9e-webhook-cert\") pod \"maas-controller-76848564f6-dkpcd\" (UID: \"3cb61b31-96d0-43a2-85d3-56b86209ae9e\") " pod="opendatahub/maas-controller-76848564f6-dkpcd" Sep 11 17:19:23.336870 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.336793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cb61b31-96d0-43a2-85d3-56b86209ae9e-metrics-certs\") pod \"maas-controller-76848564f6-dkpcd\" (UID: \"3cb61b31-96d0-43a2-85d3-56b86209ae9e\") " pod="opendatahub/maas-controller-76848564f6-dkpcd" Sep 11 17:19:23.437648 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.437566 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpfl9\" (UniqueName: \"kubernetes.io/projected/3cb61b31-96d0-43a2-85d3-56b86209ae9e-kube-api-access-lpfl9\") pod \"maas-controller-76848564f6-dkpcd\" (UID: \"3cb61b31-96d0-43a2-85d3-56b86209ae9e\") " pod="opendatahub/maas-controller-76848564f6-dkpcd" Sep 11 17:19:23.437804 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.437652 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3cb61b31-96d0-43a2-85d3-56b86209ae9e-webhook-cert\") pod \"maas-controller-76848564f6-dkpcd\" (UID: \"3cb61b31-96d0-43a2-85d3-56b86209ae9e\") " pod="opendatahub/maas-controller-76848564f6-dkpcd" Sep 11 17:19:23.437804 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.437678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cb61b31-96d0-43a2-85d3-56b86209ae9e-metrics-certs\") pod \"maas-controller-76848564f6-dkpcd\" (UID: \"3cb61b31-96d0-43a2-85d3-56b86209ae9e\") " pod="opendatahub/maas-controller-76848564f6-dkpcd" Sep 11 17:19:23.440324 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.440288 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cb61b31-96d0-43a2-85d3-56b86209ae9e-metrics-certs\") pod \"maas-controller-76848564f6-dkpcd\" (UID: \"3cb61b31-96d0-43a2-85d3-56b86209ae9e\") " pod="opendatahub/maas-controller-76848564f6-dkpcd" Sep 11 17:19:23.440454 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.440288 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3cb61b31-96d0-43a2-85d3-56b86209ae9e-webhook-cert\") pod \"maas-controller-76848564f6-dkpcd\" (UID: \"3cb61b31-96d0-43a2-85d3-56b86209ae9e\") " pod="opendatahub/maas-controller-76848564f6-dkpcd" Sep 11 17:19:23.464742 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.464714 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpfl9\" (UniqueName: \"kubernetes.io/projected/3cb61b31-96d0-43a2-85d3-56b86209ae9e-kube-api-access-lpfl9\") pod \"maas-controller-76848564f6-dkpcd\" (UID: \"3cb61b31-96d0-43a2-85d3-56b86209ae9e\") " pod="opendatahub/maas-controller-76848564f6-dkpcd" Sep 11 17:19:23.608803 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.608771 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76848564f6-dkpcd" Sep 11 17:19:23.881737 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:23.881708 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76848564f6-dkpcd"] Sep 11 17:19:23.883372 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:19:23.883345 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb61b31_96d0_43a2_85d3_56b86209ae9e.slice/crio-2521a1ef33b7d69a0502b0e11d49c2a486f57a40a0c22aa3d4eb65e47ef2789d WatchSource:0}: Error finding container 2521a1ef33b7d69a0502b0e11d49c2a486f57a40a0c22aa3d4eb65e47ef2789d: Status 404 returned error can't find the container with id 2521a1ef33b7d69a0502b0e11d49c2a486f57a40a0c22aa3d4eb65e47ef2789d Sep 11 17:19:24.437671 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:24.437632 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76848564f6-dkpcd" event={"ID":"3cb61b31-96d0-43a2-85d3-56b86209ae9e","Type":"ContainerStarted","Data":"7c760db97654e85415427df2e862a2f00a99582547a0a57a4e3149efe3391828"} Sep 11 17:19:24.438076 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:24.437696 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-76848564f6-dkpcd" Sep 11 17:19:24.438076 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:24.437717 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76848564f6-dkpcd" event={"ID":"3cb61b31-96d0-43a2-85d3-56b86209ae9e","Type":"ContainerStarted","Data":"2521a1ef33b7d69a0502b0e11d49c2a486f57a40a0c22aa3d4eb65e47ef2789d"} Sep 11 17:19:24.461243 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:24.461146 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-76848564f6-dkpcd" podStartSLOduration=1.151990899 podStartE2EDuration="1.46112472s" totalImagesPullingTime="309.133821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:24.459151452 +0000 UTC m=+2179.867253746" watchObservedRunningTime="2026-09-11 17:19:24.46112472 +0000 UTC m=+2179.869226991" Sep 11 17:19:35.445988 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:19:35.445956 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-76848564f6-dkpcd" Sep 11 17:20:31.296389 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:20:31.296286 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8"] Sep 11 17:20:31.296871 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:20:31.296483 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8" podUID="f499bf6d-64ff-4d7b-9872-eae31c9338ae" containerName="manager" containerID="cri-o://3f0d9501101405af525f8d56317218db9630f65b572108a0cb0befb2be15fbad" gracePeriod=10 Sep 11 17:20:33.781686 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:20:33.781649 2577 generic.go:356] "Generic (PLEG): container finished" podID="f499bf6d-64ff-4d7b-9872-eae31c9338ae" containerID="3f0d9501101405af525f8d56317218db9630f65b572108a0cb0befb2be15fbad" exitCode=0 Sep 11 17:20:33.782065 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:20:33.781724 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8" event={"ID":"f499bf6d-64ff-4d7b-9872-eae31c9338ae","Type":"ContainerDied","Data":"3f0d9501101405af525f8d56317218db9630f65b572108a0cb0befb2be15fbad"} Sep 11 17:20:33.947944 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:20:33.947917 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8" Sep 11 17:20:34.079002 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:20:34.078931 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f499bf6d-64ff-4d7b-9872-eae31c9338ae-extensions-socket-volume\") pod \"f499bf6d-64ff-4d7b-9872-eae31c9338ae\" (UID: \"f499bf6d-64ff-4d7b-9872-eae31c9338ae\") " Sep 11 17:20:34.079002 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:20:34.078986 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8w8t8\" (UniqueName: \"kubernetes.io/projected/f499bf6d-64ff-4d7b-9872-eae31c9338ae-kube-api-access-8w8t8\") pod \"f499bf6d-64ff-4d7b-9872-eae31c9338ae\" (UID: \"f499bf6d-64ff-4d7b-9872-eae31c9338ae\") " Sep 11 17:20:34.079355 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:20:34.079328 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f499bf6d-64ff-4d7b-9872-eae31c9338ae-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f499bf6d-64ff-4d7b-9872-eae31c9338ae" (UID: "f499bf6d-64ff-4d7b-9872-eae31c9338ae"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:20:34.081266 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:20:34.081242 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f499bf6d-64ff-4d7b-9872-eae31c9338ae-kube-api-access-8w8t8" (OuterVolumeSpecName: "kube-api-access-8w8t8") pod "f499bf6d-64ff-4d7b-9872-eae31c9338ae" (UID: "f499bf6d-64ff-4d7b-9872-eae31c9338ae"). InnerVolumeSpecName "kube-api-access-8w8t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:20:34.180586 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:20:34.180550 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f499bf6d-64ff-4d7b-9872-eae31c9338ae-extensions-socket-volume\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:20:34.180586 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:20:34.180579 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w8t8\" (UniqueName: \"kubernetes.io/projected/f499bf6d-64ff-4d7b-9872-eae31c9338ae-kube-api-access-8w8t8\") on node \"ip-10-0-143-165.ec2.internal\" DevicePath \"\"" Sep 11 17:20:34.787677 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:20:34.787639 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8" Sep 11 17:20:34.788094 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:20:34.787680 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8" event={"ID":"f499bf6d-64ff-4d7b-9872-eae31c9338ae","Type":"ContainerDied","Data":"a598b9f4c207afca6b85a362d048dbd0abc006de2b191920bbc68287e7c2f87d"} Sep 11 17:20:34.788094 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:20:34.787744 2577 scope.go:117] "RemoveContainer" containerID="3f0d9501101405af525f8d56317218db9630f65b572108a0cb0befb2be15fbad" Sep 11 17:20:34.810923 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:20:34.810891 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8"] Sep 11 17:20:34.816700 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:20:34.816678 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7mdx8"] Sep 11 17:20:35.094812 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:20:35.094735 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f499bf6d-64ff-4d7b-9872-eae31c9338ae" path="/var/lib/kubelet/pods/f499bf6d-64ff-4d7b-9872-eae31c9338ae/volumes" Sep 11 17:21:05.525520 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:05.525465 2577 scope.go:117] "RemoveContainer" containerID="c8ca20aeff0835c5d7af3a8b7b81b5e14586df96faa64656a2aff5e944c4225f" Sep 11 17:21:05.534608 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:05.534588 2577 scope.go:117] "RemoveContainer" containerID="4dfbb81c4cf8da0c54ee8414443eeff15ea501f27fadf8047d23a7ed3cf0eccd" Sep 11 17:21:05.542308 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:05.542289 2577 scope.go:117] "RemoveContainer" containerID="d8acc6eeba2e73c1c40b0b8b4e1cd323b0ef07e7ad557ad747806b2a67ae8e6d" Sep 11 17:21:31.964447 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:31.964410 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fpsl7"] Sep 11 17:21:31.964954 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:31.964938 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f499bf6d-64ff-4d7b-9872-eae31c9338ae" containerName="manager" Sep 11 17:21:31.965001 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:31.964957 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f499bf6d-64ff-4d7b-9872-eae31c9338ae" containerName="manager" Sep 11 17:21:31.965043 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:31.965033 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f499bf6d-64ff-4d7b-9872-eae31c9338ae" containerName="manager" Sep 11 17:21:31.968154 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:31.968134 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fpsl7" Sep 11 17:21:31.970986 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:31.970959 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ngvwn\"" Sep 11 17:21:31.980259 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:31.980233 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fpsl7"] Sep 11 17:21:32.075844 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:32.075811 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzlrv\" (UniqueName: \"kubernetes.io/projected/1f26c76a-187b-423c-8aaa-2df743f60f91-kube-api-access-qzlrv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fpsl7\" (UID: \"1f26c76a-187b-423c-8aaa-2df743f60f91\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fpsl7" Sep 11 17:21:32.075844 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:32.075849 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f26c76a-187b-423c-8aaa-2df743f60f91-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fpsl7\" (UID: \"1f26c76a-187b-423c-8aaa-2df743f60f91\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fpsl7" Sep 11 17:21:32.176803 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:32.176765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzlrv\" (UniqueName: \"kubernetes.io/projected/1f26c76a-187b-423c-8aaa-2df743f60f91-kube-api-access-qzlrv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fpsl7\" (UID: \"1f26c76a-187b-423c-8aaa-2df743f60f91\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fpsl7" Sep 11 17:21:32.176803 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:32.176803 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f26c76a-187b-423c-8aaa-2df743f60f91-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fpsl7\" (UID: \"1f26c76a-187b-423c-8aaa-2df743f60f91\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fpsl7" Sep 11 17:21:32.177174 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:32.177151 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f26c76a-187b-423c-8aaa-2df743f60f91-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fpsl7\" (UID: \"1f26c76a-187b-423c-8aaa-2df743f60f91\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fpsl7" Sep 11 17:21:32.186068 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:32.186036 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzlrv\" (UniqueName: \"kubernetes.io/projected/1f26c76a-187b-423c-8aaa-2df743f60f91-kube-api-access-qzlrv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fpsl7\" (UID: \"1f26c76a-187b-423c-8aaa-2df743f60f91\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fpsl7" Sep 11 17:21:32.279324 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:32.279285 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fpsl7" Sep 11 17:21:32.563340 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:32.563316 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fpsl7"] Sep 11 17:21:32.565911 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:21:32.565873 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f26c76a_187b_423c_8aaa_2df743f60f91.slice/crio-d6aa20676876f128acb0edb96a79645b76eab6efd024df076f5738c599bbed40 WatchSource:0}: Error finding container d6aa20676876f128acb0edb96a79645b76eab6efd024df076f5738c599bbed40: Status 404 returned error can't find the container with id d6aa20676876f128acb0edb96a79645b76eab6efd024df076f5738c599bbed40 Sep 11 17:21:33.074285 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:33.074248 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fpsl7" event={"ID":"1f26c76a-187b-423c-8aaa-2df743f60f91","Type":"ContainerStarted","Data":"fc04c340f1f85d2dfacb9ad2b5cf0eeeef66afc79fd53c96d773eab2cbea490d"} Sep 11 17:21:33.074720 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:33.074309 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fpsl7" event={"ID":"1f26c76a-187b-423c-8aaa-2df743f60f91","Type":"ContainerStarted","Data":"d6aa20676876f128acb0edb96a79645b76eab6efd024df076f5738c599bbed40"} Sep 11 17:21:33.074720 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:33.074427 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fpsl7" Sep 11 17:21:33.097646 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:33.097596 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fpsl7" podStartSLOduration=2.09758015 podStartE2EDuration="2.09758015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:33.095528694 +0000 UTC m=+2308.503630971" watchObservedRunningTime="2026-09-11 17:21:33.09758015 +0000 UTC m=+2308.505682418" Sep 11 17:21:44.082517 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:21:44.082476 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fpsl7" Sep 11 17:22:05.573274 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:22:05.573196 2577 scope.go:117] "RemoveContainer" containerID="b608ce9e950682f48aa5236e4bc38cb5fa365989faf7bf03304650bdd6ff88c4" Sep 11 17:22:05.581603 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:22:05.581584 2577 scope.go:117] "RemoveContainer" containerID="c3728b89ca9ff18c35239acc0358ea547279fc277bf8dd7d1312e8bb05e94fcb" Sep 11 17:23:05.198187 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:23:05.198161 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 17:23:05.204241 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:23:05.204216 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 17:23:05.606550 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:23:05.606521 2577 scope.go:117] "RemoveContainer" containerID="eb0e64e4291f40a1ca6cea06bb9d125ac868f6185a80b35c4da171058d7f9067" Sep 11 17:24:21.010513 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:21.010470 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-kjbdq_0ec9a480-9bb1-463f-83c9-d5518ab0dcb3/manager/0.log" Sep 11 17:24:21.150828 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:21.150788 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-76848564f6-dkpcd_3cb61b31-96d0-43a2-85d3-56b86209ae9e/manager/0.log" Sep 11 17:24:21.797677 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:21.797647 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d67cb4f96-s9cms_c210ed9a-fa06-4b38-a9b1-537f9b44aa19/copy-manifests/0.log" Sep 11 17:24:21.804258 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:21.804231 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d67cb4f96-s9cms_c210ed9a-fa06-4b38-a9b1-537f9b44aa19/manager/0.log" Sep 11 17:24:22.420825 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:22.420793 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819100-6g6h5_ac48fc33-c994-4f55-99cd-9cd5d96352a0/cleanup/0.log" Sep 11 17:24:23.651124 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:23.651086 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-m4fx4_1a4356e3-3699-4b1e-9486-37d2fe270bda/manager/0.log" Sep 11 17:24:24.028016 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:24.027982 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-fpsl7_1f26c76a-187b-423c-8aaa-2df743f60f91/manager/0.log" Sep 11 17:24:24.812467 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:24.812442 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vv6td_c36344a8-30b6-4b52-b868-3a30bfa914e9/discovery/0.log" Sep 11 17:24:25.062263 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:25.062239 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75dc676c8d-z2798_fcee08b9-9855-498b-88da-c1f8591a19f3/kube-auth-proxy/0.log" Sep 11 17:24:25.316780 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:25.316756 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-5zjwh_4c091003-cbc1-4289-959f-0779b52e49f5/payload-pre-processing/0.log" Sep 11 17:24:25.566461 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:25.566434 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7669dcd8d-whgmr_4cc211e5-b500-49d5-84be-13c2bfa7f7bc/router/0.log" Sep 11 17:24:25.937126 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:25.937093 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-t2nwr_373b1503-48c1-4514-8742-91212afde6d3/main/0.log" Sep 11 17:24:26.189687 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:26.189607 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-xwmw8_94b2c580-995b-42f1-9d43-097a77f12226/main/0.log" Sep 11 17:24:26.442976 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:26.442902 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsw7wt_9b4884b3-83d6-45ae-85c8-eed414110527/main/0.log" Sep 11 17:24:35.217357 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:35.217326 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kx9qk_9c31fb9a-1977-49b8-8396-c462b1d9f47f/global-pull-secret-syncer/0.log" Sep 11 17:24:35.373661 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:35.373625 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vrc4d_cd648125-8d5e-455b-95bf-5834c01e4db6/konnectivity-agent/0.log" Sep 11 17:24:35.441274 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:35.441242 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-165.ec2.internal_a3752174baf855d01c4a6df678c4937e/haproxy/0.log" Sep 11 17:24:39.690791 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:39.690757 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-m4fx4_1a4356e3-3699-4b1e-9486-37d2fe270bda/manager/0.log" Sep 11 17:24:39.780652 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:39.780621 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-fpsl7_1f26c76a-187b-423c-8aaa-2df743f60f91/manager/0.log" Sep 11 17:24:41.594017 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:41.593978 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-rsrzh_fde3ddc0-fc6a-42c7-aab9-866ca084bd59/cluster-monitoring-operator/0.log" Sep 11 17:24:41.833848 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:41.833819 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jfk88_96a37056-4433-4acf-863e-6f1725d53ee5/node-exporter/0.log" Sep 11 17:24:41.853787 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:41.853715 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jfk88_96a37056-4433-4acf-863e-6f1725d53ee5/kube-rbac-proxy/0.log" Sep 11 17:24:41.907736 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:41.907709 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jfk88_96a37056-4433-4acf-863e-6f1725d53ee5/init-textfile/0.log" Sep 11 17:24:44.890803 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:44.890777 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/1.log" Sep 11 17:24:44.895370 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:44.895349 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tlxcd_13cca68e-a749-4f4e-bbdf-21de50e37c6c/console-operator/2.log" Sep 11 17:24:45.822103 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:45.822070 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb"] Sep 11 17:24:45.825510 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:45.825477 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" Sep 11 17:24:45.828068 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:45.828043 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4wcq9\"/\"openshift-service-ca.crt\"" Sep 11 17:24:45.829369 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:45.829352 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4wcq9\"/\"kube-root-ca.crt\"" Sep 11 17:24:45.829460 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:45.829358 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4wcq9\"/\"default-dockercfg-g5zvq\"" Sep 11 17:24:45.833479 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:45.833453 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb"] Sep 11 17:24:45.926682 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:45.926643 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d-lib-modules\") pod \"perf-node-gather-daemonset-jsqtb\" (UID: \"3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d\") " pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" Sep 11 17:24:45.926682 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:45.926689 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d-proc\") pod \"perf-node-gather-daemonset-jsqtb\" (UID: \"3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d\") " pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" Sep 11 17:24:45.927115 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:45.926707 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddnv5\" (UniqueName: \"kubernetes.io/projected/3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d-kube-api-access-ddnv5\") pod \"perf-node-gather-daemonset-jsqtb\" (UID: \"3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d\") " pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" Sep 11 17:24:45.927115 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:45.926725 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d-sys\") pod \"perf-node-gather-daemonset-jsqtb\" (UID: \"3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d\") " pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" Sep 11 17:24:45.927115 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:45.926816 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d-podres\") pod \"perf-node-gather-daemonset-jsqtb\" (UID: \"3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d\") " pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" Sep 11 17:24:46.028309 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:46.028272 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d-proc\") pod \"perf-node-gather-daemonset-jsqtb\" (UID: \"3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d\") " pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" Sep 11 17:24:46.028309 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:46.028309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddnv5\" (UniqueName: \"kubernetes.io/projected/3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d-kube-api-access-ddnv5\") pod \"perf-node-gather-daemonset-jsqtb\" (UID: \"3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d\") " pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" Sep 11 17:24:46.028551 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:46.028328 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d-sys\") pod \"perf-node-gather-daemonset-jsqtb\" (UID: \"3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d\") " pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" Sep 11 17:24:46.028551 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:46.028371 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d-podres\") pod \"perf-node-gather-daemonset-jsqtb\" (UID: \"3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d\") " pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" Sep 11 17:24:46.028551 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:46.028400 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d-proc\") pod \"perf-node-gather-daemonset-jsqtb\" (UID: \"3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d\") " pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" Sep 11 17:24:46.028551 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:46.028477 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d-sys\") pod \"perf-node-gather-daemonset-jsqtb\" (UID: \"3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d\") " pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" Sep 11 17:24:46.028551 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:46.028483 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d-lib-modules\") pod \"perf-node-gather-daemonset-jsqtb\" (UID: \"3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d\") " pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" Sep 11 17:24:46.028722 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:46.028566 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d-podres\") pod \"perf-node-gather-daemonset-jsqtb\" (UID: \"3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d\") " pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" Sep 11 17:24:46.028722 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:46.028589 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d-lib-modules\") pod \"perf-node-gather-daemonset-jsqtb\" (UID: \"3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d\") " pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" Sep 11 17:24:46.036694 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:46.036669 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddnv5\" (UniqueName: \"kubernetes.io/projected/3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d-kube-api-access-ddnv5\") pod \"perf-node-gather-daemonset-jsqtb\" (UID: \"3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d\") " pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" Sep 11 17:24:46.136718 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:46.136628 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" Sep 11 17:24:46.244424 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:46.244394 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-zk4rf_a575fca1-0433-4012-b079-5baa6ca5a060/volume-data-source-validator/0.log" Sep 11 17:24:46.406928 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:46.406895 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb"] Sep 11 17:24:46.408909 ip-10-0-143-165 kubenswrapper[2577]: W0911 17:24:46.408855 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3eb9d980_bbd4_4f8e_a998_46e1cb8d4d7d.slice/crio-3e6e8eaa557c04040d30b94eaf79ea34720d925b1c300c8ec87ccbb0e501cb53 WatchSource:0}: Error finding container 3e6e8eaa557c04040d30b94eaf79ea34720d925b1c300c8ec87ccbb0e501cb53: Status 404 returned error can't find the container with id 3e6e8eaa557c04040d30b94eaf79ea34720d925b1c300c8ec87ccbb0e501cb53 Sep 11 17:24:46.997774 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:46.997735 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" event={"ID":"3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d","Type":"ContainerStarted","Data":"aff90eb836c0ad6bf8192b3fbca247f347a51902ba54d9b28b7ea2bd46537c8d"} Sep 11 17:24:46.998242 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:46.997815 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" event={"ID":"3eb9d980-bbd4-4f8e-a998-46e1cb8d4d7d","Type":"ContainerStarted","Data":"3e6e8eaa557c04040d30b94eaf79ea34720d925b1c300c8ec87ccbb0e501cb53"} Sep 11 17:24:46.998242 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:46.997835 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" Sep 11 17:24:47.015686 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:47.015631 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" podStartSLOduration=2.015615852 podStartE2EDuration="2.015615852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:24:47.014751437 +0000 UTC m=+2502.422853758" watchObservedRunningTime="2026-09-11 17:24:47.015615852 +0000 UTC m=+2502.423718121" Sep 11 17:24:47.508543 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:47.508516 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c4dn7_727f81c8-b4b4-4b6e-9570-5739b9698950/dns/0.log" Sep 11 17:24:47.528670 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:47.528645 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c4dn7_727f81c8-b4b4-4b6e-9570-5739b9698950/kube-rbac-proxy/0.log" Sep 11 17:24:47.647481 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:47.647447 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4ptxv_c11370ff-491d-4494-861c-090e05beda2c/dns-node-resolver/0.log" Sep 11 17:24:48.352836 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:48.352803 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r4sbm_ef68ac95-3add-420a-85d4-31cef2cddd8a/node-ca/0.log" Sep 11 17:24:49.840963 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:49.840928 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vv6td_c36344a8-30b6-4b52-b868-3a30bfa914e9/discovery/0.log" Sep 11 17:24:49.886469 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:49.886437 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75dc676c8d-z2798_fcee08b9-9855-498b-88da-c1f8591a19f3/kube-auth-proxy/0.log" Sep 11 17:24:49.952605 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:49.952569 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-5zjwh_4c091003-cbc1-4289-959f-0779b52e49f5/payload-pre-processing/0.log" Sep 11 17:24:50.007109 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:50.007079 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7669dcd8d-whgmr_4cc211e5-b500-49d5-84be-13c2bfa7f7bc/router/0.log" Sep 11 17:24:50.962509 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:50.962469 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w5wz5_42fbe40d-e07f-429e-bfab-01b77d189a2a/serve-healthcheck-canary/0.log" Sep 11 17:24:51.491004 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:51.490975 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-d9hsz_f76030c8-35a4-4b21-8a5d-ac0ffa450ce6/insights-operator/0.log" Sep 11 17:24:51.491542 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:51.491528 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-d9hsz_f76030c8-35a4-4b21-8a5d-ac0ffa450ce6/insights-operator/1.log" Sep 11 17:24:51.573083 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:51.573052 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fgkvg_31d2da56-fa00-4468-b597-209e9817349b/kube-rbac-proxy/0.log" Sep 11 17:24:51.593998 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:51.593970 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fgkvg_31d2da56-fa00-4468-b597-209e9817349b/exporter/0.log" Sep 11 17:24:51.613829 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:51.613804 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fgkvg_31d2da56-fa00-4468-b597-209e9817349b/extractor/0.log" Sep 11 17:24:53.012128 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:53.012101 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4wcq9/perf-node-gather-daemonset-jsqtb" Sep 11 17:24:54.060119 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:54.060086 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-kjbdq_0ec9a480-9bb1-463f-83c9-d5518ab0dcb3/manager/0.log" Sep 11 17:24:54.110255 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:54.110223 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-76848564f6-dkpcd_3cb61b31-96d0-43a2-85d3-56b86209ae9e/manager/0.log" Sep 11 17:24:54.501120 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:54.501091 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d67cb4f96-s9cms_c210ed9a-fa06-4b38-a9b1-537f9b44aa19/manager/0.log" Sep 11 17:24:54.539678 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:54.539653 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d67cb4f96-s9cms_c210ed9a-fa06-4b38-a9b1-537f9b44aa19/copy-manifests/0.log" Sep 11 17:24:56.671126 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:56.671093 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vv6td_c36344a8-30b6-4b52-b868-3a30bfa914e9/discovery/0.log" Sep 11 17:24:56.697833 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:56.697799 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75dc676c8d-z2798_fcee08b9-9855-498b-88da-c1f8591a19f3/kube-auth-proxy/0.log" Sep 11 17:24:56.741853 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:56.741818 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-5zjwh_4c091003-cbc1-4289-959f-0779b52e49f5/payload-pre-processing/0.log" Sep 11 17:24:56.789582 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:56.789561 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7669dcd8d-whgmr_4cc211e5-b500-49d5-84be-13c2bfa7f7bc/router/0.log" Sep 11 17:24:58.733940 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:58.733902 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vv6td_c36344a8-30b6-4b52-b868-3a30bfa914e9/discovery/0.log" Sep 11 17:24:58.782334 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:58.782304 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75dc676c8d-z2798_fcee08b9-9855-498b-88da-c1f8591a19f3/kube-auth-proxy/0.log" Sep 11 17:24:58.833323 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:58.833296 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-5zjwh_4c091003-cbc1-4289-959f-0779b52e49f5/payload-pre-processing/0.log" Sep 11 17:24:58.875046 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:24:58.875000 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7669dcd8d-whgmr_4cc211e5-b500-49d5-84be-13c2bfa7f7bc/router/0.log" Sep 11 17:25:01.761614 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:25:01.761567 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-fc9sn_ed61a19b-2fa7-466b-9772-3b7a1f4a5d25/kube-storage-version-migrator-operator/1.log" Sep 11 17:25:01.762389 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:25:01.762356 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-fc9sn_ed61a19b-2fa7-466b-9772-3b7a1f4a5d25/kube-storage-version-migrator-operator/0.log" Sep 11 17:25:02.973768 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:25:02.973739 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k7dfv_9052d32a-f4da-4ace-9da5-6a63d6e7e0c2/kube-multus-additional-cni-plugins/0.log" Sep 11 17:25:03.000970 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:25:03.000945 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k7dfv_9052d32a-f4da-4ace-9da5-6a63d6e7e0c2/egress-router-binary-copy/0.log" Sep 11 17:25:03.023390 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:25:03.023363 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k7dfv_9052d32a-f4da-4ace-9da5-6a63d6e7e0c2/cni-plugins/0.log" Sep 11 17:25:03.044162 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:25:03.044137 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k7dfv_9052d32a-f4da-4ace-9da5-6a63d6e7e0c2/bond-cni-plugin/0.log" Sep 11 17:25:03.065064 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:25:03.065031 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k7dfv_9052d32a-f4da-4ace-9da5-6a63d6e7e0c2/routeoverride-cni/0.log" Sep 11 17:25:03.085466 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:25:03.085438 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k7dfv_9052d32a-f4da-4ace-9da5-6a63d6e7e0c2/whereabouts-cni-bincopy/0.log" Sep 11 17:25:03.104812 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:25:03.104789 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k7dfv_9052d32a-f4da-4ace-9da5-6a63d6e7e0c2/whereabouts-cni/0.log" Sep 11 17:25:03.425657 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:25:03.425620 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zgzsg_a269739e-353a-4b14-ae8c-05de63045f05/kube-multus/0.log" Sep 11 17:25:03.451059 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:25:03.451011 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-68tln_f1193c55-865c-4fa7-aabb-9ebbdadd11d2/network-metrics-daemon/0.log" Sep 11 17:25:03.467856 ip-10-0-143-165 kubenswrapper[2577]: I0911 17:25:03.467816 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-68tln_f1193c55-865c-4fa7-aabb-9ebbdadd11d2/kube-rbac-proxy/0.log"